AIX 5.3 重启系统后VG PERMISSION被改变导致Oracle10.2.0.5集群启动失败
客户的数据库要升级，从Oracle9.2.0.4双节点RAC升级到Oracle10.2.0.5双节点RAC。操作系统是AIX 5.3。服务器为P740 x 2。
主机、存储、数据库厂商之间配合的有些问题，导致安装期间问题不断。我负责安装数据库，被折腾到够呛。在不被通知的情况下重启主机、更换磁盘、更换背板、更换光纤一系列不可控的以外停机一次一次的冲击着OracleRAC，冲击着我的耐性。
有过共同经历的DBA都看到过那种双手一摊，一脸无辜的表情说出“我什么也没做”的情景。我只能按照各种日志上报错的时间来给个“提醒”，在xx天xx时xx分系统被意外关闭过。你有印象么....?
本篇博客记录了一次主机重启后导致的集群软件启动失败的案例。
环境介绍
OS： AIX 5.3
DB: Oracle 10.2.0.5 RAC
Instance: scg1, scgl2
Storage: ASM
故障现象
主机被重启，1号主机集群启动成功，2号主机集群启动失败。
启动2号主机集群
# /u01/app/oracle/product/10.2.0/crs_1/bin/crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
日志无任何输出
tail -f /u01/app/oracle/product/10.2.0/crs_1/log/scgl2/crsd/crsd.log
关闭2号主机集群
# /u01/app/oracle/product/10.2.0/crs_1/bin/crsctl stop crd
系统报错
OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [Read-only file system] [30]
故障分析
由于OCR所在磁盘不能写入数据导致集群启动失败。
检查ORC磁盘组的属组、权限都没有问题。
检查主机的VG PERMISSION 找的问题原因，该属性值是passive-only。正确的属性应该是read/write
解决方案
在2号机上操作
# lsvg datavg
VOLUME GROUP: datavg VG IDENTIFIER: 00f7639d00004c00000001482d5c96b7
VG STATE: active PP SIZE: 256 megabyte(s)
VG PERMISSION: passive-only TOTAL PPs: 3196 (818176 megabytes)
MAX LVs: 256 FREE PPs: 1176 (301056 megabytes)
LVs: 10 USED PPs: 2020 (517120 megabytes)
OPEN LVs: 0 QUORUM: 3 (Enabled)
TOTAL PVs: 4 VG DESCRIPTORS: 4
STALE PVs: 0 STALE PPs: 0
ACTIVE PVs: 4 AUTO ON: no
Concurrent: Enhanced-Capable Auto-Concurrent: Disabled
VG Mode: Concurrent
Node ID: 2 Active Nodes: 1
MAX PPs per VG: 32512
MAX PPs per PV: 1016 MAX PVs: 32
LTG size (Dynamic): 256 kilobyte(s) AUTO SYNC: no
HOT SPARE: no BB POLICY: relocatable
0，在两节点分别停止集群软件
/u01/app/oracle/product/10.2.0/crs_1/bin/crsctl stop crs -f
/u01/app/oracle/product/10.2.0/crs_1/bin/crsctl stop crs -f
1, 停止vg
varyoffvg datavg
2, 启动vg
varyonvg datavg
3, 更改datavg里面的10个lv的权限
# lsvg -l datavg
datavg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
ora_raw1_100gb raw 400 400 1 closed/syncd N/A
ora_raw2_100gb raw 400 400 1 closed/syncd N/A
ora_raw3_100gb raw 400 400 1 closed/syncd N/A
ora_raw4_100gb raw 400 400 1 closed/syncd N/A
ora_raw5_100gb raw 400 400 2 closed/syncd N/A
ora_raw6_1gb raw 4 4 1 closed/syncd N/A
ora_raw7_1gb raw 4 4 1 closed/syncd N/A
ora_raw8_1gb raw 4 4 1 closed/syncd N/A
ora_raw9_1gb raw 4 4 1 closed/syncd N/A
ora_raw10_1gb raw 4 4 1 closed/syncd N/A
chlv -p w ora_raw1_100gb
chlv -p w ora_raw2_100gb
chlv -p w ora_raw3_100gb
chlv -p w ora_raw4_100gb
chlv -p w ora_raw5_100gb
chlv -p w ora_raw6_1gb
chlv -p w ora_raw7_1gb
chlv -p w ora_raw8_1gb
chlv -p w ora_raw9_1gb
chlv -p w ora_raw10_1gb
4, 再次停止vg
varyoffvg datavg
5, 再次启动vg
varyonvg -c datavg
6,检查权限，已经从passive-only改成read/write
# lsvg datavg
VOLUME GROUP: datavg VG IDENTIFIER: 00f7639d00004c00000001482d5c96b7
VG STATE: active PP SIZE: 256 megabyte(s)
VG PERMISSION: read/write TOTAL PPs: 3196 (818176 megabytes)
MAX LVs: 256 FREE PPs: 1176 (301056 megabytes)
LVs: 10 USED PPs: 2020 (517120 megabytes)
OPEN LVs: 0 QUORUM: 3 (Enabled)
TOTAL PVs: 4 VG DESCRIPTORS: 4
STALE PVs: 0 STALE PPs: 0
ACTIVE PVs: 4 AUTO ON: no
Concurrent: Enhanced-Capable Auto-Concurrent: Disabled
VG Mode: Concurrent
Node ID: 2 Active Nodes: 1
MAX PPs per VG: 32512
MAX PPs per PV: 1016 MAX PVs: 32
LTG size (Dynamic): 256 kilobyte(s) AUTO SYNC: no
HOT SPARE: no BB POLICY: relocatable
7，分别启动2台主机的集群
# /u01/app/oracle/product/10.2.0/crs_1/bin/crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
# /u01/app/oracle/product/10.2.0/crs_1/bin/crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
# /u01/app/oracle/product/10.2.0/crs_1/bin/crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
# /u01/app/oracle/product/10.2.0/crs_1/bin/crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
8，启动数据库
SQL> select open_mode from gv$database;
OPEN_MODE
----------
READ WRITE
READ WRITE
本次故障处理花费了我好多时间，总结经验如下：
Oracle排错要准确，发现Oracle没有问题要敢于把问题抛出去，让主机、存储配合检查。
提升主机和存储的知识，在不依赖主机工程师和存储工程师的状态下多做些检查。
主机、存储、数据库厂商之间配合的有些问题，导致安装期间问题不断。我负责安装数据库，被折腾到够呛。在不被通知的情况下重启主机、更换磁盘、更换背板、更换光纤一系列不可控的以外停机一次一次的冲击着OracleRAC，冲击着我的耐性。
有过共同经历的DBA都看到过那种双手一摊，一脸无辜的表情说出“我什么也没做”的情景。我只能按照各种日志上报错的时间来给个“提醒”，在xx天xx时xx分系统被意外关闭过。你有印象么....?
本篇博客记录了一次主机重启后导致的集群软件启动失败的案例。
环境介绍
OS： AIX 5.3
DB: Oracle 10.2.0.5 RAC
Instance: scg1, scgl2
Storage: ASM
故障现象
主机被重启，1号主机集群启动成功，2号主机集群启动失败。
启动2号主机集群
# /u01/app/oracle/product/10.2.0/crs_1/bin/crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
日志无任何输出
tail -f /u01/app/oracle/product/10.2.0/crs_1/log/scgl2/crsd/crsd.log
关闭2号主机集群
# /u01/app/oracle/product/10.2.0/crs_1/bin/crsctl stop crd
系统报错
OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [Read-only file system] [30]
故障分析
由于OCR所在磁盘不能写入数据导致集群启动失败。
检查ORC磁盘组的属组、权限都没有问题。
检查主机的VG PERMISSION 找的问题原因，该属性值是passive-only。正确的属性应该是read/write
解决方案
在2号机上操作
# lsvg datavg
VOLUME GROUP: datavg VG IDENTIFIER: 00f7639d00004c00000001482d5c96b7
VG STATE: active PP SIZE: 256 megabyte(s)
VG PERMISSION: passive-only TOTAL PPs: 3196 (818176 megabytes)
MAX LVs: 256 FREE PPs: 1176 (301056 megabytes)
LVs: 10 USED PPs: 2020 (517120 megabytes)
OPEN LVs: 0 QUORUM: 3 (Enabled)
TOTAL PVs: 4 VG DESCRIPTORS: 4
STALE PVs: 0 STALE PPs: 0
ACTIVE PVs: 4 AUTO ON: no
Concurrent: Enhanced-Capable Auto-Concurrent: Disabled
VG Mode: Concurrent
Node ID: 2 Active Nodes: 1
MAX PPs per VG: 32512
MAX PPs per PV: 1016 MAX PVs: 32
LTG size (Dynamic): 256 kilobyte(s) AUTO SYNC: no
HOT SPARE: no BB POLICY: relocatable
0，在两节点分别停止集群软件
/u01/app/oracle/product/10.2.0/crs_1/bin/crsctl stop crs -f
/u01/app/oracle/product/10.2.0/crs_1/bin/crsctl stop crs -f
1, 停止vg
varyoffvg datavg
2, 启动vg
varyonvg datavg
3, 更改datavg里面的10个lv的权限
# lsvg -l datavg
datavg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
ora_raw1_100gb raw 400 400 1 closed/syncd N/A
ora_raw2_100gb raw 400 400 1 closed/syncd N/A
ora_raw3_100gb raw 400 400 1 closed/syncd N/A
ora_raw4_100gb raw 400 400 1 closed/syncd N/A
ora_raw5_100gb raw 400 400 2 closed/syncd N/A
ora_raw6_1gb raw 4 4 1 closed/syncd N/A
ora_raw7_1gb raw 4 4 1 closed/syncd N/A
ora_raw8_1gb raw 4 4 1 closed/syncd N/A
ora_raw9_1gb raw 4 4 1 closed/syncd N/A
ora_raw10_1gb raw 4 4 1 closed/syncd N/A
chlv -p w ora_raw1_100gb
chlv -p w ora_raw2_100gb
chlv -p w ora_raw3_100gb
chlv -p w ora_raw4_100gb
chlv -p w ora_raw5_100gb
chlv -p w ora_raw6_1gb
chlv -p w ora_raw7_1gb
chlv -p w ora_raw8_1gb
chlv -p w ora_raw9_1gb
chlv -p w ora_raw10_1gb
4, 再次停止vg
varyoffvg datavg
5, 再次启动vg
varyonvg -c datavg
6,检查权限，已经从passive-only改成read/write
# lsvg datavg
VOLUME GROUP: datavg VG IDENTIFIER: 00f7639d00004c00000001482d5c96b7
VG STATE: active PP SIZE: 256 megabyte(s)
VG PERMISSION: read/write TOTAL PPs: 3196 (818176 megabytes)
MAX LVs: 256 FREE PPs: 1176 (301056 megabytes)
LVs: 10 USED PPs: 2020 (517120 megabytes)
OPEN LVs: 0 QUORUM: 3 (Enabled)
TOTAL PVs: 4 VG DESCRIPTORS: 4
STALE PVs: 0 STALE PPs: 0
ACTIVE PVs: 4 AUTO ON: no
Concurrent: Enhanced-Capable Auto-Concurrent: Disabled
VG Mode: Concurrent
Node ID: 2 Active Nodes: 1
MAX PPs per VG: 32512
MAX PPs per PV: 1016 MAX PVs: 32
LTG size (Dynamic): 256 kilobyte(s) AUTO SYNC: no
HOT SPARE: no BB POLICY: relocatable
7，分别启动2台主机的集群
# /u01/app/oracle/product/10.2.0/crs_1/bin/crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
# /u01/app/oracle/product/10.2.0/crs_1/bin/crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
# /u01/app/oracle/product/10.2.0/crs_1/bin/crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
# /u01/app/oracle/product/10.2.0/crs_1/bin/crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
8，启动数据库
SQL> select open_mode from gv$database;
OPEN_MODE
----------
READ WRITE
READ WRITE
本次故障处理花费了我好多时间，总结经验如下：
Oracle排错要准确，发现Oracle没有问题要敢于把问题抛出去，让主机、存储配合检查。
提升主机和存储的知识，在不依赖主机工程师和存储工程师的状态下多做些检查。
来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/29047826/viewspace-1265183/ ，如需转载，请注明出处，否则将追究法律责任。
请登录后发表评论
登录
全部评论
