ITPub博客
首页 博客 专栏 学院 社区 IT新媒体评选
发博

首页 > 数据库 > Oracle > 【RAC】因系统时间设置不当，造成RAC一节点集群资源及数据库关闭

【RAC】因系统时间设置不当，造成RAC一节点集群资源及数据库关闭

原创 Oracle 作者：xysoul_云龙 时间：2016-02-19 15:31:41 0 删除 编辑


        下午接到一个同事电话，说一体机（RAC)第二个节点数据库连不上了，让我帮忙看看。我便登上系统，在第一个节点查看信息，如下

点击(此处)折叠或打开

  1. [grid@pwjkdb01 ~]$ crs_stat -t
  2. Name Type Target State Host
  3. ------------------------------------------------------------
  4. ora....PWJK.dg ora....up.type ONLINE ONLINE pwjkdb01
  5. ora.DBFS_DG.dg ora....up.type ONLINE ONLINE pwjkdb01
  6. ora....ER.lsnr ora....er.type ONLINE ONLINE pwjkdb01
  7. ora....N1.lsnr ora....er.type ONLINE OFFLINE
  8. ora....PWJK.dg ora....up.type ONLINE ONLINE pwjkdb01
  9. ora.asm ora.asm.type ONLINE ONLINE pwjkdb01
  10. ora.cvu ora.cvu.type ONLINE OFFLINE
  11. ora.gsd ora.gsd.type OFFLINE OFFLINE
  12. ora....network ora....rk.type ONLINE ONLINE pwjkdb01
  13. ora.oc4j ora.oc4j.type ONLINE OFFLINE
  14. ora.ons ora.ons.type ONLINE ONLINE pwjkdb01
  15. ora.pwdata.db ora....se.type ONLINE ONLINE pwjkdb01
  16. ora....rv1.svc ora....ce.type ONLINE ONLINE pwjkdb01
  17. ora....rv2.svc ora....ce.type ONLINE ONLINE pwjkdb01
  18. ora....SM1.asm application ONLINE ONLINE pwjkdb01
  19. ora....01.lsnr application ONLINE ONLINE pwjkdb01
  20. ora....b01.gsd application OFFLINE OFFLINE
  21. ora....b01.ons application ONLINE ONLINE pwjkdb01
  22. ora....b01.vip ora....t1.type ONLINE ONLINE pwjkdb01
  23. ora....b02.vip ora....t1.type ONLINE OFFLINE
  24. ora.scan1.vip ora....ip.type ONLINE OFFLINE
登录第二个节点，查看部分信息


点击(此处)折叠或打开

  1. [root@pwjkdb02 ~]# ps -ef |grep pmon
  2. root 6679 1 0 2015 ? 00:01:42 /usr/bin/perl -w /opt/oracle.cellos/compmon/exadata_mon_hw_asr.pl -server
  3. root 63055 62491 0 16:45 pts/1 00:00:00 grep pmon

由于业务系统关系，我查看了下系统时间、运行时间，便尝试启动第二个节点集群资源


点击(此处)折叠或打开

  1. [root@pwjkdb02 bin]# ./crsctl start crs
  2. CRS-4640: Oracle High Availability Services is already active
  3. CRS-4000: Command Start failed, or completed with errors.
  4. [root@pwjkdb02 bin]# ./crsctl start cluster
  5. CRS-2672: Attempting to start 'ora.asm' on 'pwjkdb02'
  6. CRS-2676: Start of 'ora.asm' on 'pwjkdb02' succeeded
  7. CRS-2672: Attempting to start 'ora.crsd' on 'pwjkdb02'
  8. CRS-2676: Start of 'ora.crsd' on 'pwjkdb02' succeeded
  9. [root@pwjkdb02 bin]#

节点二启动正常，如下

点击(此处)折叠或打开

  1. [grid@pwjkdb02 ~]$ crs_stat -t
  2. Name Type Target State Host
  3. ------------------------------------------------------------
  4. ora....PWJK.dg ora....up.type ONLINE ONLINE pwjkdb01
  5. ora.DBFS_DG.dg ora....up.type ONLINE ONLINE pwjkdb01
  6. ora....ER.lsnr ora....er.type ONLINE ONLINE pwjkdb01
  7. ora....N1.lsnr ora....er.type ONLINE ONLINE pwjkdb02
  8. ora....PWJK.dg ora....up.type ONLINE ONLINE pwjkdb01
  9. ora.asm ora.asm.type ONLINE ONLINE pwjkdb01
  10. ora.cvu ora.cvu.type ONLINE ONLINE pwjkdb02
  11. ora.gsd ora.gsd.type OFFLINE OFFLINE
  12. ora....network ora....rk.type ONLINE ONLINE pwjkdb01
  13. ora.oc4j ora.oc4j.type ONLINE ONLINE pwjkdb02
  14. ora.ons ora.ons.type ONLINE ONLINE pwjkdb01
  15. ora.pwdata.db ora....se.type ONLINE ONLINE pwjkdb01
  16. ora....rv1.svc ora....ce.type ONLINE ONLINE pwjkdb01
  17. ora....rv2.svc ora....ce.type ONLINE ONLINE pwjkdb01
  18. ora....SM1.asm application ONLINE ONLINE pwjkdb01
  19. ora....01.lsnr application ONLINE ONLINE pwjkdb01
  20. ora....b01.gsd application OFFLINE OFFLINE
  21. ora....b01.ons application ONLINE ONLINE pwjkdb01
  22. ora....b01.vip ora....t1.type ONLINE ONLINE pwjkdb01
  23. ora....SM2.asm application ONLINE ONLINE pwjkdb02
  24. ora....02.lsnr application ONLINE ONLINE pwjkdb02
  25. ora....b02.gsd application OFFLINE OFFLINE
  26. ora....b02.ons application ONLINE ONLINE pwjkdb02
  27. ora....b02.vip ora....t1.type ONLINE ONLINE pwjkdb02
  28. ora.scan1.vip ora....ip.type ONLINE ONLINE pwjkdb02

启动后，查看部分日志

数据库日志：

点击(此处)折叠或打开

  1. tail -100f alertpwjkdb02.log

  3. 4016-01-02 16:26:00.736
  4. [/u01/app/11.2.0.3/grid/bin/oraagent.bin(48051)]CRS-5011:Check of resource "pwdata" failed: details at "(:CLSN00007:)" in "/u01/app/11.2.0.3/grid/log/pwjkdb02/agent/crsd/oraagent_oracle/oraagent_oracle.log"
  5. 4016-01-02 16:26:01.262
  6. [crsd(9981)]CRS-2765:Resource 'ora.pwdata.db' has failed on server 'pwjkdb02'.
  7. 4016-01-02 16:26:01.329
  8. [crsd(9981)]CRS-2765:Resource 'ora.pwdata.pwdatasrv2.svc' has failed on server 'pwjkdb02'.
  9. 4016-01-02 16:26:01.329
  10. [crsd(9981)]CRS-2771:Maximum restart attempts reached for resource 'ora.pwdata.pwdatasrv2.svc'; will not restart.
  11. 4016-01-02 16:26:01.510
  12. [/u01/app/11.2.0.3/grid/bin/oraagent.bin(8988)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0.3/grid/log/pwjkdb02/agent/ohasd/oraagent_grid/oraagent_grid.log"
  13. 4016-01-02 16:26:01.614
  14. [ohasd(6722)]CRS-2765:Resource 'ora.asm' has failed on server 'pwjkdb02'.
  15. 4016-01-02 16:26:01.663
  16. [/u01/app/11.2.0.3/grid/bin/oraagent.bin(8988)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0.3/grid/log/pwjkdb02/agent/ohasd/oraagent_grid/oraagent_grid.log"
  17. ……………………………………
ASM实例警告日志，如下所示，报错为系统当前系统时间问题


点击(此处)折叠或打开

  1. tail -600f alert_+ASM2.log |more

  2. Warning: VKTM detected a time drift.
    Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.
    Tue Jan 12 00:39:25 2016
    Warning: VKTM detected a time drift.
    Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.
    Sat Jan 02 16:26:00 4016
    Warning: VKTM detected a time drift.
    Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.
    Sat Jan 02 16:26:00 4016
    Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_pmon_9913.trc:
    ORA-01513: invalid current time returned by operating system
    PMON (ospid: 9913): terminating the instance due to error 1513
    Sat Jan 02 16:26:01 4016
    System state dump requested by (instance=2, osid=9913 (PMON)), summary=[abnormal instance termination].
    System State dumped to trace file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_diag_9923.trc
    Dumping diagnostic data in directory=[cdmp_19740622152201], requested by (instance=2, osid=9913 (PMON)), summary=[abnormal instance 
    termination].
    Sat Jan 02 16:26:01 4016
    ORA-1092 : opitsk aborting process
    Sat Jan 02 16:26:01 4016
    License high water mark = 24
    Instance terminated by PMON, pid = 9913
    USER (ospid: 49268): terminating the instance

查看该节点操作系统操作历史记录,

点击(此处)折叠或打开

  1. vi .bash_profile

  3. su - oracle
  4. #1454306838
  5. sar 1 5
  6. #1454315175
  7. date
  8. #1454315199
  9. date 010216264016.00
  10. #64565627161
  11. date
  12. #64565627177
  13. date 0102162716.00
  14. #1451723220
  15. date
  16. #1451723223
  17. date
  18. #1451723225
  19. date
  20. #1451723227
  21. date
  22. #1451723233
  23. date
  24. #1451723250
  25. date
  26. #1451723302
  27. date
  28. #1451723310
  29. date
  30. #1451723315
  31. date
  32. #1451723323
  33. su - oracle
  34. #1451723446
  35. date
  36. #1451723534
  37. date 0201163016.00
  38. #1454315401
  39. date
  40. #1454315505
  41. date 0201163416.00
  42. #1454315642
  43. date
  44. #1454315647
  45. date
  46. #1454315648

通过以上我们可以找到一条记录为：date 010216264016.00，再通过警告日志及查看其它集群日志，可以确认，由于更改操作系统时间造成RAC节点二集群关闭，经过电话沟通，该同事发现系统时间慢5分钟，直接在操作系统上更改（请注意，更改操作系统时间需谨慎，尤其数据库系统运行状态，以免影响业务应用），由于命令不熟，将时间改为4016年，带来以上问题。

任何操作都有风险性，在做操作时，我们应该提前做好规划、操作方案以及应急预案及风险性评估，切不可想当然对在线系统做任何更改。

来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/29487349/viewspace-1991263/ ，如需转载，请注明出处，否则将追究法律责任。

0
0
上一篇： 【DG】怎么使用Data Pump备份物理备用数据库
下一篇： 【HINT】Hint的常见错误使用方式(nologging)
请登录后发表评论 登录
全部评论
xysoul_云龙
专家姓名：杨云龙
主要从事Oracle数据库 DBA相关工作，其他操作系统、中间件等也有涉及，热衷分享。岁月老将至，如有幸在江湖中留点踪迹，也是欣慰。 【文盲筱烨】好读书爱运动的IT技术爱好者 微博：文盲筱烨 微信公众号：筱烨视点

注册时间：2014-02-15

  • 博文量
    412
  • 访问量
    1104532

最新文章

支持我们 作者招募 用户协议 FAQ Contact Us

北京盛拓优讯信息技术有限公司. 版权所有  京ICP备16024965号-8 北京市公安局海淀分局网监中心备案编号：11010802021510 niuxiaotong@pcpop.com 17352615567 未成年人举报专区

广播电视节目制作经营许可证(京) 字第1234号 中国互联网协会会员

违法和不良信息举报电话:18101376593