CLUSTER启动报错无法访问物理设备
测试环境的RAC数据库的一个节点在意外死机后，发现CLUSTER无法启动。
由于停电，导致测试环境的RAC数据库的一个节点非正常关闭。重启系统后，其他节点正常，但是这个节点的CLUSTER无法启动，错误信息如下：
# /etc/init.d/init.crs start
Startup will be queued to init within 30 seconds.
# ps -ef|grep ora
oracle 1506 1504 0 16:01:56 pts/1 0:00 -sh
root 9360 9203 0 16:38:20 pts/2 0:00 grep ora
oracle 1510 1506 0 16:02:06 pts/1 0:00 bash
oracle 9087 9085 0 16:37:29 pts/2 0:00 -sh
oracle 9091 9087 0 16:37:33 pts/2 0:00 bash
# ls -l /tmp
total 80
-rw-r--r-- 1 oracle oinstall 155 Nov 26 16:38 crsctl.9237
-rw-r--r-- 1 oracle oinstall 155 Nov 26 16:38 crsctl.9240
-rw-r--r-- 1 oracle oinstall 155 Nov 26 16:38 crsctl.9245
-rw-rw-r-- 1 root sys 1028 Nov 26 15:57 ps_data
drwx------ 2 root root 182 Nov 26 15:57 smc898
# more /tmp/crsctl.9237
OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [No such device or address] [6]
检查系统发现，原来存储共享磁盘在当前服务器上映射的裸设备名称改变了：
# ls -l /dev/rac/ocr
lrwxrwxrwx 1 root other 18 Mar 14 2007 /dev/rac/ocr -> /dev/rdsk/c2t0d2s1
# dd if=/dev/rac/ocr f=/data/ocr.bak
dd: /dev/rac/ocr: open: No such device or address
为了避免RAC不同节点的映射名称不一样，采用操作系统链接的方式避免这个问题。Cluster启动的ocr盘的链接指向/dev/rdsk/c2t0d2s1，但是通过dd访问，发现这个裸设备并不存在。
检查系统挂载的裸设备：
# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c1t0d0
/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cfd99114,0
1. c2t1d0
/pci@8,700000/QLGC,qla@3/sd@1,0
2. c2t1d1
/pci@8,700000/QLGC,qla@3/sd@1,1
3. c2t1d2
/pci@8,700000/QLGC,qla@3/sd@1,2
4. c2t1d3
/pci@8,700000/QLGC,qla@3/sd@1,3
5. c2t1d4
/pci@8,700000/QLGC,qla@3/sd@1,4
6. c2t1d5
/pci@8,700000/QLGC,qla@3/sd@1,5
7. c2t1d6
/pci@8,700000/QLGC,qla@3/sd@1,6
8. c2t1d7
/pci@8,700000/QLGC,qla@3/sd@1,7
9. c2t1d8
/pci@8,700000/QLGC,qla@3/sd@1,8
10. c2t1d9
/pci@8,700000/QLGC,qla@3/sd@1,9
11. c2t1d10
/pci@8,700000/QLGC,qla@3/sd@1,a
12. c2t1d11
/pci@8,700000/QLGC,qla@3/sd@1,b
13. c2t1d12
/pci@8,700000/QLGC,qla@3/sd@1,c
14. c2t1d13
/pci@8,700000/QLGC,qla@3/sd@1,d
Specify disk (enter its number): 3
selecting c2t1d2
[disk formatted]
FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
repair - repair a defective sector
label - write label to the disk
analyze - surface analysis
defect - defect list management
backup - search for backup labels
verify - read and display labels
save - save new disk/partition definitions
inquiry - show vendor, product and revision
volname - set 8-character volume name
!
quit
format> p
PARTITION MENU:
0 - change `0' partition
1 - change `1' partition
2 - change `2' partition
3 - change `3' partition
4 - change `4' partition
5 - change `5' partition
6 - change `6' partition
7 - change `7' partition
select - select a predefined table
modify - modify a predefined partition table
name - name the current table
print - display the current table
label - write partition map and label to the disk
!
quit
partition> p
Current partition table (original):
Total disk cylinders available: 32766 + 2 (reserved cylinders)
Part Tag Flag Cylinders Size Blocks
0 root wm 0 - 5461 1.00GB (5462/0/0) 2097408
1 swap wu 5462 - 10923 1.00GB (5462/0/0) 2097408
2 backup wu 0 - 32765 6.00GB (32766/0/0) 12582144
3 unassigned wm 10924 - 16385 1.00GB (5462/0/0) 2097408
4 unassigned wm 16386 - 21847 1.00GB (5462/0/0) 2097408
5 unassigned wm 21848 - 27309 1.00GB (5462/0/0) 2097408
6 usr wm 27310 - 32765 1023.00MB (5456/0/0) 2095104
7 unassigned wm 0 0 (0/0/0) 0
partition> quit
FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
repair - repair a defective sector
label - write label to the disk
analyze - surface analysis
defect - defect list management
backup - search for backup labels
verify - read and display labels
save - save new disk/partition definitions
inquiry - show vendor, product and revision
volname - set 8-character volume name
!
quit
format> quit
# ls -l /dev/rdsk/c2t1d2s1
lrwxrwxrwx 1 root root 50 Aug 20 15:29 /dev/rdsk/c2t1d2s1 -> ../../devices/pci@8,700000/QLGC,qla@3/sd@1,2:b,raw
原来的裸设备c2t0d2s1，现在已经变成c2t1d2s1，下面需要修改/dev/rac/ocr和/dev/rac/vot链接指向新的裸设备地址：
# cd /dev/rac
# ls -l
total 4
lrwxrwxrwx 1 root other 18 Mar 14 2007 ocr -> /dev/rdsk/c2t0d2s1
lrwxrwxrwx 1 root other 18 Mar 14 2007 vot -> /dev/rdsk/c2t0d2s3
# rm ocr vot
# ln -fs /dev/rdsk/c2t1d2s1 ocr
# ln -fs /dev/rdsk/c2t1d2s3 vot
# ls -l
total 4
lrwxrwxrwx 1 root other 18 Nov 26 16:52 ocr -> /dev/rdsk/c2t1d2s1
lrwxrwxrwx 1 root other 18 Nov 26 16:52 vot -> /dev/rdsk/c2t1d2s3
然后尝试重新启动CLUSTER环境：
# /etc/init.d/init.crs start
Startup will be queued to init within 30 seconds.
# ps -ef|grep ora
oracle 1506 1504 0 16:01:56 pts/1 0:00 -sh
root 9677 9203 0 16:53:22 pts/2 0:00 grep ora
oracle 1510 1506 0 16:02:06 pts/1 0:00 bash
oracle 9087 9085 0 16:37:29 pts/2 0:00 -sh
oracle 9091 9087 0 16:37:33 pts/2 0:00 bash
# cd /tmp
# ls
crsctl.9237 crsctl.9240 crsctl.9245 ps_data smc898
# more crsctl.9237
OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [Permission denied] [13]
显然映射到不同名称的裸设备的授权消失，需要重新授权：
# chown oracle:oinstall /dev/rdsk/c2t1*
# /etc/init.d/init.crs start
Startup will be queued to init within 30 seconds.
对裸设备授权后，再次尝试重启CLUSTER，等待一段时间后，再次检查发现CLUSTER已经正常启动：
# ps -ef|grep ora
oracle 1506 1504 0 16:01:56 pts/1 0:00 -sh
oracle 10877 1 0 16:56:50 ? 0:00 asm_ckpt_+ASM1
oracle 9798 363 0 16:56:17 ? 0:01 /data/oracle/product/10.2/crs/bin/evmd.bin
oracle 9941 9780 0 16:56:20 ? 0:00 sh -c /bin/sh -c 'ulimit -c unlimited; cd /data/oracle/product/10.2/crs/log/rac
oracle 1510 1506 0 16:02:06 pts/1 0:00 bash
oracle 11230 366 0 16:57:02 ? 0:00 /bin/sh /data/oracle/product/10.2/database/bin/racgwrap start
oracle 363 1 0 15:57:18 ? 0:00 sh -c sh -c 'ulimit -c unlimited; cd /data/oracle/product/10.2/crs/log/racnode1
root 366 1 0 15:57:18 ? 0:04 /data/oracle/product/10.2/crs/bin/crsd.bin reboot
oracle 11429 1 1 16:57:10 ? 0:00 ora_pmon_testrac1
oracle 11266 11236 0 16:57:04 ? 0:00 /data/oracle/product/10.2/database/bin/racgeut -e _USR_ORA_DEBUG=0 -e ORACLE_SI
oracle 9087 9085 0 16:37:29 pts/2 0:00 -sh
oracle 10830 1 0 16:56:49 ? 0:00 asm_diag_+ASM1
oracle 11533 1 1 16:57:14 ? 0:01 ora_lms1_testrac1
oracle 11234 11230 0 16:57:03 ? 0:00 /data/oracle/product/10.2/database/bin/racgmain start
oracle 10201 9798 0 16:56:29 ? 0:00 /data/oracle/product/10.2/crs/bin/evmlogger.bin -o /data/oracle/product/10.2/cr
oracle 9091 9087 0 16:37:33 pts/2 0:00 bash
root 9858 9779 0 16:56:18 ? 0:00 /data/oracle/product/10.2/crs/bin/oprocd run -t 1000 -m 500 -f
oracle 9946 9941 0 16:56:20 ? 0:00 /bin/sh -c ulimit -c unlimited; cd /data/oracle/product/10.2/crs/log/racnode1/c
oracle 10832 1 0 16:56:49 ? 0:00 asm_psp0_+ASM1
oracle 9949 9946 0 16:56:20 ? 0:01 /data/oracle/product/10.2/crs/bin/ocssd.bin
root 11951 9203 0 16:57:29 pts/2 0:00 grep ora
oracle 10566 1 0 16:56:40 ? 0:00 /data/oracle/product/10.2/database/bin/tnslsnr LISTENER1_RACNODE1 -inherit
oracle 11334 1 0 16:57:06 ? 0:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle 10836 1 0 16:56:49 ? 0:00 asm_lmd0_+ASM1
oracle 10870 1 0 16:56:50 ? 0:00 asm_mman_+ASM1
oracle 11129 1 0 16:56:59 ? 0:00 /data/oracle/product/10.2/database/bin/racgimon daemon ora.racnode1.ASM1.asm
oracle 10883 1 0 16:56:50 ? 0:00 asm_gmon_+ASM1
oracle 10873 1 0 16:56:50 ? 0:00 asm_dbw0_+ASM1
oracle 10881 1 0 16:56:50 ? 0:00 asm_rbal_+ASM1
oracle 10834 1 0 16:56:49 ? 0:00 asm_lmon_+ASM1
oracle 10875 1 0 16:56:50 ? 0:00 asm_lgwr_+ASM1
oracle 10838 1 0 16:56:49 ? 0:00 asm_lms0_+ASM1
oracle 10879 1 0 16:56:50 ? 0:00 asm_smon_+ASM1
oracle 10828 1 0 16:56:49 ? 0:00 asm_pmon_+ASM1
oracle 10914 1 0 16:56:51 ? 0:00 asm_lck0_+ASM1
oracle 11168 1 0 16:57:01 ? 0:00 /data/oracle/product/10.2/crs/opmn/bin/ons -d
oracle 11236 1 0 16:57:03 ? 0:00 /data/oracle/product/10.2/database/bin/racgimon startd testrac
oracle 11298 11267 0 16:57:05 ? 0:00 /data/oracle/product/10.2/database/bin/sqlplus
oracle 11267 11266 0 16:57:04 ? 0:00 /data/oracle/product/10.2/database/bin/racgmdb -s open
oracle 11170 11168 0 16:57:01 ? 0:00 /data/oracle/product/10.2/crs/opmn/bin/ons -d
oracle 11313 1 0 16:57:06 ? 0:00 asm_asmb_+ASM1
oracle 11338 1 0 16:57:06 ? 0:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle 11336 1 0 16:57:06 ? 0:00 asm_o000_+ASM1
oracle 11431 1 0 16:57:10 ? 0:00 ora_diag_testrac1
oracle 11678 1 1 16:57:18 ? 0:00 ora_asmb_testrac1
oracle 11435 1 0 16:57:10 ? 0:00 ora_psp0_testrac1
oracle 11572 1 0 16:57:15 ? 0:00 ora_ckpt_testrac1
oracle 11529 1 1 16:57:13 ? 0:01 ora_lms0_testrac1
oracle 11451 1 1 16:57:10 ? 0:01 ora_lmd0_testrac1
oracle 11449 1 0 16:57:10 ? 0:00 ora_lmon_testrac1
oracle 11574 1 0 16:57:15 ? 0:00 ora_smon_testrac1
oracle 11569 1 0 16:57:14 ? 0:00 ora_lgwr_testrac1
oracle 11544 1 3 16:57:14 ? 0:02 ora_mman_testrac1
oracle 11547 1 0 16:57:14 ? 0:00 ora_dbw0_testrac1
oracle 11576 1 0 16:57:15 ? 0:00 ora_reco_testrac1
oracle 11578 1 0 16:57:15 ? 0:00 ora_cjq0_testrac1
oracle 11580 1 0 16:57:15 ? 0:00 ora_mmon_testrac1
oracle 11582 1 0 16:57:15 ? 0:00 ora_mmnl_testrac1
oracle 11584 1 0 16:57:15 ? 0:00 ora_d000_testrac1
oracle 11586 1 0 16:57:15 ? 0:00 ora_s000_testrac1
oracle 11891 1 0 16:57:26 ? 0:00 ora_o001_testrac1
oracle 11682 1 0 16:57:18 ? 0:00 ora_rbal_testrac1
oracle 11646 1 1 16:57:17 ? 0:00 ora_lck0_testrac1
oracle 11680 1 0 16:57:18 ? 0:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle 11771 1 0 16:57:22 ? 0:00 ora_o000_testrac1
oracle 11773 1 1 16:57:22 ? 0:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle 11906 11298 1 16:57:27 ? 0:00 oracletestrac1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/4227/viewspace-621703/ ，如需转载，请注明出处，否则将追究法律责任。
