ITPub博客

首页 > Linux操作系统 > Linux操作系统 > 11.2 RAC自动启动报错诊断(二)

11.2 RAC自动启动报错诊断(二)

原创 Linux操作系统 作者:yangtingkun 时间:2011-06-02 23:58:28 0 删除 编辑

一个客户的11.2 RAC for Linux X86-64环境停电后出现了故障,RAC环境无法自动启动。

这一篇描述新的问题。

11.2 RAC自动启动报错诊断:http://yangtingkun.itpub.net/post/468/518656

 

前两天提到了诊断问题发现错误出现在硬件或存储层面,大量的IO问题导致了CLUSTER和数据库频繁出现错误,虽然客户通过重建OCRVOT的方式来解决了RAC无法启动的问题,但是问题的根源并没有解决。

果然在运行了一天之后,两个节点又频繁出现了错误,告警日志信息为:

Wed Jun 01 03:06:53 2011
WARNING: Write Failed. group:5 disk:0 AU:102 offset:49152 size:16384
Errors in file /oracle/diag/rdbms/spsp/SPSP1/trace/SPSP1_ckpt_7745.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 0 logical extent 0 of file 256 in group 5 on disk 0 allocation unit 102
Errors in file /oracle/diag/rdbms/spsp/SPSP1/trace/SPSP1_ckpt_7745.trc:
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+DG_REDO/spsp/controlfile/current.256.751041959'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
Errors in file /oracle/diag/rdbms/spsp/SPSP1/trace/SPSP1_ckpt_7745.trc:
ORA-00221: error on write to control file
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+DG_REDO/spsp/controlfile/current.256.751041959'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
CKPT (ospid: 7745): terminating the instance due to error 221
Wed Jun 01 03:06:53 2011
System state dump requested by (instance=1, sid=7745 (CKPT)), summary=[abnormal instance termination].
System State dumped to trace file /oracle/diag/rdbms/spsp/SPSP1/trace/SPSP1_diag_7709.trc
Wed Jun 01 03:06:54 2011
ORA-1092 : opitsk aborting process
Wed Jun 01 03:06:55 2011
ORA-1092 : opitsk aborting process
Wed Jun 01 03:06:55 2011
ORA-1092 : opitsk aborting process
Wed Jun 01 03:06:56 2011
ORA-1092 : opitsk aborting process
Wed Jun 01 03:06:56 2011
License high water mark = 108
Instance terminated by CKPT, pid = 7745
USER (ospid: 29004): terminating the instance
Instance terminated by USER, pid = 29004
.
.
.
WARNING: Read Failed. group:5 disk:0 AU:106 offset:49152 size:16384
WARNING: failed to read mirror side 1 of virtual extent 4 logical extent 0 of file 256 in group [5.939042496] from disk REDO1 allocation unit 106 reason error; if possible,will try another mirror side
WARNING: Read Failed. group:5 disk:0 AU:106 offset:49152 size:16384
WARNING: Read Failed. group:5 disk:0 AU:102 offset:16384 size:16384
WARNING: group 5 dismounted: failed to read virtual extent 0 of file 256
Errors in file /oracle/diag/rdbms/spsp/SPSP1/trace/SPSP1_lmon_14128.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 4 logical extent 0 of file 256 in group 5 on disk 0 allocation unit 106
Errors in file /oracle/diag/rdbms/spsp/SPSP1/trace/SPSP1_ckpt_14152.trc:
ORA-00202: control file: '+DG_REDO/spsp/controlfile/current.256.751041959'
ORA-15079: ASM file is closed
Errors in file /oracle/diag/rdbms/spsp/SPSP1/trace/SPSP1_lmon_14128.trc:
ORA-00202: control file: '+DG_REDO/spsp/controlfile/current.256.751041959'
ORA-15079: ASM file is closed
SUCCESS: diskgroup DG_REDO was dismounted
SUCCESS: diskgroup DG_REDO was dismounted
Wed Jun 01 06:15:43 2011
Errors in file /oracle/diag/rdbms/spsp/SPSP1/trace/SPSP1_lgwr_14150.trc:
ORA-00345: redo log write error block 19795 count 1364
ORA-00312: online log 2 thread 1: '+DG_REDO/spsp/onlinelog/group_2.258.751041965'
ORA-15078: ASM diskgroup was forcibly dismounted
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /oracle/diag/rdbms/spsp/SPSP1/trace/SPSP1_ckpt_14152.trc:
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+DG_REDO/spsp/controlfile/current.256.751041959'
ORA-15078: ASM diskgroup was forcibly dismounted
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /oracle/diag/rdbms/spsp/SPSP1/trace/SPSP1_ckpt_14152.trc:
ORA-00221: error on write to control file
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+DG_REDO/spsp/controlfile/current.256.751041959'
ORA-15078: ASM diskgroup was forcibly dismounted
ORA-15078: ASM diskgroup was forcibly dismounted
CKPT (ospid: 14152): terminating the instance due to error 221
System state dump requested by (instance=1, sid=14152 (CKPT)), summary=[abnormal instance termination].
System State dumped to trace file /oracle/diag/rdbms/spsp/SPSP1/trace/SPSP1_diag_14118.trc
Wed Jun 01 06:15:49 2011
Instance terminated by CKPT, pid = 14152

显然IO错误的问题仍然存在,而且这次错误发生在控制文件和联机日志文件上,直接导致数据库崩溃。

检查ASM实例错误:

Wed Jun 01 03:06:53 2011
WARNING: Write Failed. group:1 disk:0 AU:1 offset:1044480 size:4096
WARNING: disk 0.3909302363 (ARCH1) not responding to heart beat
ERROR: too many offline disks in PST (grp 1)
WARNING: Write Failed. group:2 disk:2 AU:1 offset:1044480 size:4096
WARNING: Write Failed. group:2 disk:1 AU:1 offset:1044480 size:4096
WARNING: Write Failed. group:2 disk:0 AU:1 offset:1044480 size:4096
WARNING: disk 0.3909302364 (VD1) not responding to heart beat
WARNING: disk 1.3909302365 (VD2) not responding to heart beat
ERROR: too many offline disks in PST (grp 2)
WARNING: disk 2.3909302366 (VD3) not responding to heart beat
ERROR: too many offline disks in PST (grp 2)
WARNING: Write Failed. group:3 disk:0 AU:1 offset:1044480 size:4096
WARNING: disk 0.3909302367 (DATA1) not responding to heart beat
ERROR: too many offline disks in PST (grp 3)
WARNING: Write Failed. group:4 disk:0 AU:1 offset:1044480 size:4096
WARNING: disk 0.3909302369 (FLSH1) not responding to heart beat
ERROR: too many offline disks in PST (grp 4)
WARNING: Write Failed. group:5 disk:0 AU:1 offset:1044480 size:4096
WARNING: disk 0.3909302370 (REDO1) not responding to heart beat
ERROR: too many offline disks in PST (grp 5)
Wed Jun 01 03:06:53 2011
NOTE: process 28970 initiating offline of disk 0.3909302363 (ARCH1) with mask 0x7e in group 1
Wed Jun 01 03:06:53 2011
NOTE: process 28972 initiating offline of disk 0.3909302364 (VD1) with mask 0x7e in group 2
NOTE: process 28972 initiating offline of disk 1.3909302365 (VD2) with mask 0x7e in group 2
NOTE: process 28972 initiating offline of disk 2.3909302366 (VD3) with mask 0x7e in group 2
NOTE: checking PST: grp = 2
Wed Jun 01 03:06:53 2011
NOTE: process 28974 initiating offline of disk 0.3909302367 (DATA1) with mask 0x7e in group 3
Wed Jun 01 03:06:53 2011
NOTE: process 28976 initiating offline of disk 0.3909302369 (FLSH1) with mask 0x7e in group 4
WARNING: Disk ARCH1 in mode 0x7f is now being offlined
WARNING: Disk ARCH1 in mode 0x7f is now being taken offline
GMON checking disk modes for group 2 at 21 for pid 35, osid 28972
NOTE: initiating PST update: grp = 1, dsk = 0/0xe903385b, mode = 0x15
ERROR: no PST quorum in group: required 2, found 0
NOTE: checking PST for grp 2 done.
WARNING: Disk VD1 in mode 0x7f is now being taken offline
WARNING: Disk VD2 in mode 0x7f is now being taken offline
WARNING: Disk VD3 in mode 0x7f is now being taken offline
NOTE: initiating PST update: grp = 2, dsk = 0/0xe903385c, mode = 0x15
WARNING: Disk DATA1 in mode 0x7f is now being offlinedWed Jun 01 03:06:53 2011
NOTE: initiating PST update: grp = 2, dsk = 1/0xe903385d, mode = 0x15
.
.
.
Wed Jun 01 03:06:53 2011
NOTE: halting all I/Os to diskgroup 2 (DG_CRS)
Wed Jun 01 03:06:53 2011
NOTE: LGWR doing non-clean dismount of group 2 (DG_CRS)
NOTE: LGWR sync ABA=7.56 last written ABA 7.56
GMON updating disk modes for group 1 at 23 for pid 33, osid 28970
ERROR: no PST quorum in group: required 1, found 0
WARNING: Disk ARCH1 in mode 0x7f offline aborted
Wed Jun 01 03:06:53 2011
kjbdomdet send to inst 2
detach from dom 2, sending detach message to inst 2
WARNING: Offline for disk ARCH1 in mode 0x7f failed.
NOTE: cache dismounting (not clean) group 1/0xA323C8A3 (DG_ARCH)
GMON updating disk modes for group 4 at 24 for pid 38, osid 28976
ERROR: no PST quorum in group: required 1, found 0
GMON updating disk modes for group 5 at 25 for pid 39, osid 28978
ERROR: no PST quorum in group: required 1, found 0
WARNING: Disk FLSH1 in mode 0x7f offline aborted
WARNING: Offline for disk FLSH1 in mode 0x7f failed.
NOTE: cache dismounting (not clean) group 4/0xA343C8A6 (DG_FLSH)
WARNING: Disk REDO1 in mode 0x7f offline aborted
WARNING: Offline for disk REDO1 in mode 0x7f failed.
NOTE: cache dismounting (not clean) group 5/0xA343C8A7 (DG_REDO)
GMON updating disk modes for group 3 at 26 for pid 36, osid 28974
WARNING: Read Failed. group:3 disk:1 AU:1 offset:4096 size:4096
WARNING: Read Failed. group:3 disk:1 AU:1 offset:0 size:4096
ERROR: no PST quorum in group: required 1, found 0
.
.
.
WARNING: dirty detached from domain 5
NOTE: cache dismounted group 5/0xA343C8A7 (DG_REDO)
SQL> alter diskgroup DG_REDO dismount force /* ASM SERVER */
Wed Jun 01 03:06:55 2011
ERROR: ORA-15130 in COD recovery for diskgroup 1/0xa323c8a3 (DG_ARCH)
NOTE: cache deleting context for group DG_ARCH 1/0xa323c8a3
ERROR: ORA-15130 thrown in RBAL for group number 1
Errors in file /oracle/gridbase/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_7195.trc:
ORA-15130: diskgroup "DG_ARCH" is being dismounted
ERROR: ORA-15130 in COD recovery for diskgroup 2/0xa333c8a4 (DG_CRS)
NOTE: cache deleting context for group DG_CRS 2/0xa333c8a4
ERROR: ORA-15130 thrown in RBAL for group number 2
Errors in file /oracle/gridbase/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_7195.trc:
ORA-15130: diskgroup "DG_CRS" is being dismounted
ERROR: ORA-15130 in COD recovery for diskgroup 3/0xa333c8a5 (DG_DATA)
NOTE: cache deleting context for group DG_DATA 3/0xa333c8a5
ERROR: ORA-15130 thrown in RBAL for group number 3
Errors in file /oracle/gridbase/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_7195.trc:
ORA-15130: diskgroup "DG_DATA" is being dismounted
ERROR: ORA-15130 in COD recovery for diskgroup 4/0xa343c8a6 (DG_FLSH)
NOTE: cache deleting context for group DG_FLSH 4/0xa343c8a6
ERROR: ORA-15130 thrown in RBAL for group number 4
Errors in file /oracle/gridbase/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_7195.trc:
ORA-15130: diskgroup "DG_FLSH" is being dismounted
ERROR: ORA-15130 in COD recovery for diskgroup 5/0xa343c8a7 (DG_REDO)
NOTE: cache deleting context for group DG_REDO 5/0xa343c8a7
ERROR: ORA-15130 thrown in RBAL for group number 5
Errors in file /oracle/gridbase/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_7195.trc:
ORA-15130: diskgroup "DG_REDO" is being dismounted
WARNING: Read Failed. group:1 disk:0 AU:1 offset:4096 size:4096
WARNING: Read Failed. group:1 disk:0 AU:1 offset:0 size:4096
ERROR: no PST quorum in group: required 1, found 0
ERROR: Could not heartbeat PST for grp 1. Force dismounting the disk group.
WARNING: Read Failed. group:2 disk:2 AU:1 offset:4096 size:4096
WARNING: Read Failed. group:2 disk:2 AU:1 offset:0 size:4096
WARNING: Read Failed. group:2 disk:1 AU:1 offset:4096 size:4096
WARNING: Read Failed. group:2 disk:1 AU:1 offset:0 size:4096
WARNING: Read Failed. group:2 disk:0 AU:1 offset:4096 size:4096
WARNING: Read Failed. group:2 disk:0 AU:1 offset:0 size:4096
ERROR: no PST quorum in group: required 2, found 0
ERROR: Could not heartbeat PST for grp 2. Force dismounting the disk group.
WARNING: Read Failed. group:3 disk:1 AU:1 offset:4096 size:4096
.
.
.
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 0 logical extent 0 of file 255 in group 2 on disk 0 allocation unit 60
Errors in file /oracle/gridbase/diag/asm/+asm/+ASM1/trace/+ASM1_ora_7224.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 2 of virtual extent 0 logical extent 1 of file 255 in group 2 on disk 1 allocation unit 60
Errors in file /oracle/gridbase/diag/asm/+asm/+ASM1/trace/+ASM1_ora_7224.trc:
ORA-15081: failed to submit an I/O operation to a disk
WARNING: Disk REDO1 in mode 0x7f is now being offlined

同样ASM也报出大量的错误,检查CLUSTER错误信息:

2011-06-01 03:06:51.427
[cssd(6997)]CRS-1649:An I/O error occured for voting file: ORCL:VD3; details at (:CSSNM00060:) in /oracle/product/11g/grid/log/oracle-01/cssd/ocssd.log.
2011-06-01 03:06:51.428
[cssd(6997)]CRS-1649:An I/O error occured for voting file: ORCL:VD1; details at (:CSSNM00060:) in /oracle/product/11g/grid/log/oracle-01/cssd/ocssd.log.
2011-06-01 03:06:51.428
[cssd(6997)]CRS-1649:An I/O error occured for voting file: ORCL:VD2; details at (:CSSNM00060:) in /oracle/product/11g/grid/log/oracle-01/cssd/ocssd.log.
2011-06-01 03:06:54.202
[crsd(7214)]CRS-2765:Resource 'ora.spsp.db' has failed on server 'oracle-02'.
2011-06-01 03:06:54.316
[/oracle/product/11g/grid/bin/oraagent.bin(7517)]CRS-5011:Check of resource "SPSP" failed: details at "(:CLSN00007:)" in "/oracle/product/11g/grid/log/oracle-01/agent/crsd/oraagent_oracle/oraagent_oracle.log"
2011-06-01 03:06:54.318
[crsd(7214)]CRS-2765:Resource 'ora.spsp.db' has failed on server 'oracle-01'.
2011-06-01 03:06:55.437
[cssd(6997)]CRS-1649:An I/O error occured for voting file: ORCL:VD3; details at (:CSSNM00060:) in /oracle/product/11g/grid/log/oracle-01/cssd/ocssd.log.
2011-06-01 03:06:55.437
[cssd(6997)]CRS-1649:An I/O error occured for voting file: ORCL:VD2; details at (:CSSNM00060:) in /oracle/product/11g/grid/log/oracle-01/cssd/ocssd.log.
2011-06-01 03:06:55.437
[cssd(6997)]CRS-1649:An I/O error occured for voting file: ORCL:VD1; details at (:CSSNM00060:) in /oracle/product/11g/grid/log/oracle-01/cssd/ocssd.log.
2011-06-01 03:07:00.866
[crsd(7214)]CRS-1006:The OCR location is inaccessible. Details in /oracle/product/11g/grid/log/oracle-01/crsd/crsd.log.
2011-06-01 03:07:02.028
[/oracle/product/11g/grid/bin/oraagent.bin(28990)]CRS-5822:Agent '/oracle/product/11g/grid/bin/oraagent_oracle' disconnected from server. Details at (:CRSAGF00117:) {0:8:3} in /oracle/product/11g/grid/log/oracle-01/agent/crsd/oraagent_oracle/oraagent_oracle.log.
2011-06-01 03:07:02.028
[/oracle/product/11g/grid/bin/scriptagent.bin(7515)]CRS-5822:Agent '/oracle/product/11g/grid/bin/scriptagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:5:6} in /oracle/product/11g/grid/log/oracle-01/agent/crsd/scriptagent_grid/scriptagent_grid.log.
2011-06-01 03:07:02.029
[/oracle/product/11g/grid/bin/orarootagent.bin(7322)]CRS-5822:Agent '/oracle/product/11g/grid/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:2:7} in /oracle/product/11g/grid/log/oracle-01/agent/crsd/orarootagent_root/orarootagent_root.log.
2011-06-01 03:07:02.029
[/oracle/product/11g/grid/bin/oraagent.bin(32669)]CRS-5822:Agent '/oracle/product/11g/grid/bin/oraagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:7:8} in /oracle/product/11g/grid/log/oracle-01/agent/crsd/oraagent_grid/oraagent_grid.log.
2011-06-01 03:07:02.540
[ohasd(12733)]CRS-2765:Resource 'ora.crsd' has failed on server 'oracle-01'.
2011-06-01 03:07:03.891
[crsd(29009)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /oracle/product/11g/grid/log/oracle-01/crsd/crsd.log.
2011-06-01 03:07:04.572
[ohasd(12733)]CRS-2765:Resource 'ora.crsd' has failed on server 'oracle-01'.
2011-06-01 03:07:05.914
[crsd(29022)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /oracle/product/11g/grid/log/oracle-01/crsd/crsd.log.
.
.
.
2011-06-01 03:07:22.847
[ohasd(12733)]CRS-2765:Resource 'ora.crsd' has failed on server 'oracle-01'.
2011-06-01 03:07:22.848
[ohasd(12733)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
2011-06-01 03:08:30.626
[cssd(6997)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file ORCL:VD1 will be considered not functional in 99880 milliseconds
2011-06-01 03:08:30.626
[cssd(6997)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file ORCL:VD2 will be considered not functional in 99930 milliseconds
2011-06-01 03:08:30.626
.
.
.
[cssd(6997)]CRS-1613:No I/O has completed after 90% of the maximum interval. Voting file ORCL:VD3 will be considered not functional in 19780 milliseconds
2011-06-01 03:10:10.826
[cssd(6997)]CRS-1604:CSSD voting file is offline: ORCL:VD1; details at (:CSSNM00058:) in /oracle/product/11g/grid/log/oracle-01/cssd/ocssd.log.
2011-06-01 03:10:10.826
[cssd(6997)]CRS-1604:CSSD voting file is offline: ORCL:VD2; details at (:CSSNM00058:) in /oracle/product/11g/grid/log/oracle-01/cssd/ocssd.log.
2011-06-01 03:10:10.827
[cssd(6997)]CRS-1604:CSSD voting file is offline: ORCL:VD3; details at (:CSSNM00058:) in /oracle/product/11g/grid/log/oracle-01/cssd/ocssd.log.
2011-06-01 03:10:10.827
[cssd(6997)]CRS-1606:The number of voting files available, 0, is less than the minimum number of voting files required, 2, resulting in CSSD termination to ensure data integrity; details at (:CSSNM00018:) in /oracle/product/11g/grid/log/oracle-01/cssd/ocssd.log
2011-06-01 03:10:10.827
[cssd(6997)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /oracle/product/11g/grid/log/oracle-01/cssd/ocssd.log
2011-06-01 03:10:10.877
[cssd(6997)]CRS-1652:Starting clean up of CRSD resources.
2011-06-01 03:10:11.059
[cssd(6997)]CRS-1653:The clean up of the CRSD resources failed.

可以看到,原来导致CLUSTER频繁重启的IO问题并没有解决,重建OCRVOT只是治标不治本的临时解决方案,根本的问题不解决,错误还会频繁出现。

检查一下对应时刻操作系统的messages信息:

Jun 1 03:06:32 Oracle-01 avahi-daemon[12772]: Registering new address record for 172.31.3.9 on eth0.
Jun 1 03:06:32 Oracle-01 avahi-daemon[12772]: Registering HINFO record with values 'X86_64'/'LINUX'.
Jun 1 03:06:51 Oracle-01 kernel: sd 6:0:0:2: Device not ready: <6>: Current: sense key: Not Ready
Jun 1 03:06:51 Oracle-01 kernel: Add. Sense: Logical unit not accessible, asymmetric access state transition
Jun 1 03:06:51 Oracle-01 kernel:
Jun 1 03:06:51 Oracle-01 kernel: end_request: I/O error, dev sdc, sector 1402565051
Jun 1 03:06:51 Oracle-01 kernel: sd 6:0:0:1: Device not ready: <6>: Current: sense key: Not Ready
Jun 1 03:06:51 Oracle-01 kernel: Add. Sense: Logical unit not accessible, asymmetric access state transition
Jun 1 03:06:51 Oracle-01 kernel:
Jun 1 03:06:51 Oracle-01 kernel: end_request: I/O error, dev sdb, sector 1600550111
Jun 1 03:06:51 Oracle-01 kernel: sd 6:0:0:2: Device not ready: <6>: Current: sense key: Not Ready
Jun 1 03:06:51 Oracle-01 kernel: Add. Sense: Logical unit not accessible, asymmetric access state transition
Jun 1 03:06:51 Oracle-01 kernel:
Jun 1 03:06:51 Oracle-01 kernel: end_request: I/O error, dev sdc, sector 1400556926
Jun 1 03:06:52 Oracle-01 avahi-daemon[12772]: Withdrawing address record for fe80::225:90ff:fe31:8abc on eth2.
Jun 1 03:06:52 Oracle-01 avahi-daemon[12772]: Withdrawing address record for 169.254.85.173 on eth2.
.
.
.
Jun 1 03:06:52 Oracle-01 avahi-daemon[12772]: Registering HINFO record with values 'X86_64'/'LINUX'.
Jun 1 03:06:53 Oracle-01 kernel: sd 6:0:0:1: Device not ready: <6>: Current: sense key: Not Ready
Jun 1 03:06:53 Oracle-01 kernel: Add. Sense: Logical unit not accessible, asymmetric access state transition
Jun 1 03:06:53 Oracle-01 kernel:
Jun 1 03:06:53 Oracle-01 kernel: end_request: I/O error, dev sdb, sector 4151
Jun 1 03:06:53 Oracle-01 kernel: sd 6:0:0:2: Device not ready: <6>: Current: sense key: Not Ready
Jun 1 03:06:53 Oracle-01 kernel: Add. Sense: Logical unit not accessible, asymmetric access state transition
Jun 1 03:06:53 Oracle-01 kernel:
Jun 1 03:06:53 Oracle-01 kernel: end_request: I/O error, dev sdc, sector 1402044833
Jun 1 03:06:53 Oracle-01 kernel: sd 6:0:0:2: Device not ready: <6>: Current: sense key: Not Ready
.
.
.
Jun 1 03:06:53 Oracle-01 kernel:
Jun 1 03:06:53 Oracle-01 kernel: end_request: I/O error, dev sdb, sector 1602038018
Jun 1 03:06:53 Oracle-01 kernel: sd 6:0:0:2: Device not ready: <6>: Current: sense key: Not Ready
Jun 1 03:06:53 Oracle-01 kernel: Add. Sense: Logical unit not accessible, asymmetric access state transition
Jun 1 03:06:53 Oracle-01 kernel:
Jun 1 03:06:53 Oracle-01 kernel: end_request: I/O error, dev sdc, sector 400020548
Jun 1 03:06:53 Oracle-01 kernel: sd 6:0:0:1: Device not ready: <6>: Current: sense key: Not Ready
Jun 1 03:06:53 Oracle-01 kernel: Add. Sense: Logical unit not accessible, asymmetric access state transition
Jun 1 03:06:53 Oracle-01 kernel:
Jun 1 03:06:53 Oracle-01 kernel: end_request: I/O error, dev sdb, sector 1602242922
Jun 1 03:06:55 Oracle-01 kernel: sd 6:0:0:1: Device not ready: <6>: Current: sense key: Not Ready

在数据库实例崩溃的时刻,操作系统上同样报错了I/O错误,显然问题已经不再是数据库和CLUSTER层面的,需要在操作系统和存储层面确定并解决问题。

 

来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/4227/viewspace-697035/,如需转载,请注明出处,否则将追究法律责任。

请登录后发表评论 登录
全部评论
暂无介绍

注册时间:2007-12-29

  • 博文量
    1955
  • 访问量
    10523062