现在新安装的数据库无论是单机或RAC，版本基本都是Oracle 11.2.0.3了，其中经历了2次该版本RAC的bug导致节点crash的故障，与大家分享下。今年3月初的时候，某运营商的一套RAC数据库突然一个节点垮了，该数据库版本就是Linux64位系统Oracle 11.2.0.3.0的，出于考虑先恢复业务，当时负责该系统的同事就尝试去启动CRS和数据库，谁知道怎么也启动不了CRS，如下所示（由于日志比较多，只截取最关键的一部分）：：
grid@wxcs:~> sudo /Oracle/app/crs/bin/crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.
使用crsctl stat res -t -init查看CRS的状态，启动GPNP进程后一直存在报错，
2013-03-04 23:48:33.756: [ OCRMSG][2657056528]GIPC error [29] msg [gipcretConnectionRefused]
2013-03-04 23:48:47.758: [ OCRMSG][2657056528]GIPC error [29] msg [gipcretConnectionRefused]
其中cssd一直为STARING状态，其中后台cssd的日志为：
2013-03-03 22:40:25.319: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:26.095: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222804,
LATS 1783876, lastSeqNo 11222803, uniqueness 1361347261, timestamp 1362321629/975388116
既然“has a disk HB, but no network HB”明显就是心跳出问题，内部通信有问题，然后我们手工去ping，都没有问题，我们还在metalink上下载了mcasttest.pl工具来对对网卡进行了全面的测试，发现也是没有问题，这就奇怪了，没问题还报心跳错误？不是很甘心，决定使用终极办法，对故障机器进行重启尝试是否可以恢复故障，谁知道机器起来后，等了大半天，还是不能恢复，再重启了一次，还是如此，于是我们就猜测有可能是CRS软件出了什么问题，然后在metalink找了一天，最后终于找到了ID 1479380.1里面描述的症状：bug 13653178与bug 13899736导致Node cannot join the cluster after reboot or “interconnect restored”，里面的心跳错误也是一模一样，再找了下发现ID 1456977.1也发现出现类似的心跳错误，这个是由bug 13334158与bug 13811209引起，然后再详细对比了下其它日志报错内容，更精确定位为bug 13334158与bug 13811209。查了下PSU补丁，发现PSU5是把这4个bug都给修复的了，于是发出故障报告，建议打PSU5 patch 14038787 解决，该补丁包含了GRID和RDBMS的补丁。
最初我们的建议按照metalink上的建议就是找个维护时间，把两台机器同时重启（不要只重启节点2），估计可以恢复，由于正在开两会，所以局方这边说要封网暂时不打，就让一个节点先顶着先。于是我们自己先在虚拟机上测试下PSU5的补丁。谁知道过了两天，某航空的一套RAC数据库出了问题，该航空DBA说其中一个节点垮掉了，重启了机器好几次，故障节点的数据库服务始终起不来，都急死了。我详细看了下他发来日志，该数据库也是Linux64位系统Oracle11.2.0.3.0版本，然后发现跟前几天某运营商的故障症状一模一样啊，于是我告诉他这是个bug来的，我们有一套系统也出了这个问题。然后他问怎么解决，我说打PSU5，他说你们打了吗？我说现在两会封网要过几天，他想重启两台机器，但是又怕两个节点都起不来。该数据库是关于气象方面的，平时压力不算大，但是如果两个节点都垮掉的话，正在天空飞的飞机当然不会掉下来那么恐怖，但会影响气候方面的信息调度。于是他跟他的领导汇报了下这个故障，然后决定让我们在运营商那里先打，再看我们打完后的状况，直接当该运营商的数据库为测试机了啊:)。
过几天我们就开始打出问题的那套运营商数据库的PSU5，停掉了业务，在打之前先同时重启了两台机器，果然故障的CRS都恢复了正常，数据库也拉起来了。于是我们开始打PSU5，打得非常顺利，结果就恢复了故障。
过了几天去客户现场打PSU5，当时应用没有停，基于一个原则：要保证数据库故障恢复正常的情况下再打打PSU，免得故障出现重叠现象，即如果本来数据库有问题，打完补丁后才检查出来的话，一般人会认为是我们打补丁造成的，这样的话会搞得相当乱。于是在申请同意的情况下，先重启了两台服务器，和我们预期的一样，故障节点的CRS恢复了正常，在准备打PSU之前，客户DBA有意无意去测试了下业务，即用toad去连数据库，谁知道发现怎么也连不上去，这就奇怪了，CRS服务都是好的，数据库实例都是好的，偏偏就是连不上去，检查VIP服务都是存在的，但是去ping或tnsping只能通一个节点，确定配置也没变动，客户DBA说下午都还可以的，找了下原因没有看出异常，结果我们尝试再重启了下两台机器，这次更怪了，两个节点CRS和数据库实例还是都能起来，虽然两个节点的所有显示都正常，但是现在用工具是两个节点都连接不上了。然后客户DBA问我在运营商那边操作的时候有没有这种情况发生，我说我们把应用停了再重启机器的，重启后是测试不了业务的，就直接打PSU5的。三点多钟大半夜里心慌慌的，于是我们两个商量了半个钟头左右，看是否决定还打不打PSU5，最后商量的结果就是：决定还是打PSU5吧，因为有回退方案，即使打完没效果也可以干净回退。说不定这个问题就是bug引起的呢，只是oracle的官方都没有说出这种症状。当然也理解，oracle不可能在所有环境下对所有情况进行测试。于是开始打PSU5，打了大概40分钟左右，就打完了，我们再去ping及用toad去连数据库，这次两个节点都连上了，数据库瞬间恢复正常了，真神奇，看来真的是bug引起的。当然最近网上看到有人在不打补丁恢复故障节点的方法为：跑到机房里去把心跳线拔掉，然后再插上重启故障节点就可以临时解决，但是我这边倒没有去验证过。
所以建议以后在装完这种比较新版本数据库的时候，一定要记得打上PSU，免得出一些不必要的麻烦问题。
下面是打PSU5的过程，以供参考。
[root@QiXiang-DB1 backup]# su – grid
[grid@QiXiang-DB1 ~]$ pwd
/home/grid
[grid@QiXiang-DB1 ~]$ mkdir psu5
[grid@QiXiang-DB1 ~]$ chmod -R 777 psu5/
[root@QiXiang-DB1 backup]# mv p14727347_112030_Linux-x86-64.zip /home/grid
[root@QiXiang-DB1 backup]# cd /home/grid
[root@QiXiang-DB1 ~]# chown -R grid.oinstall p14727347_112030_Linux-x86-64.zip
[root@QiXiang-DB1 ~]# su – grid
[grid@QiXiang-DB1 ~]$ unzip p14727347_112030_Linux-x86-64.zip -d /home/grid/psu5
[grid@QiXiang-DB1 ~]$ /oracle/product/11.2.3/crs11g/OPatch/opatch version
Invoking OPatch 11.2.0.1.7
OPatch Version: 11.2.0.1.7
OPatch succeeded.
[root@QiXiang-DB1 psu5]# cd /oracle/product/11.2.3/crs11g/
[root@QiXiang-DB1 crs11g]# mv OPatch/ OPatch_bak
[root@QiXiang-DB1 backup]# mv p6880880_112000_Linux-x86-64.zip /home/grid/
[root@QiXiang-DB1 ~]# chown -R grid.oinstall /home/grid/p6880880_112000_Linux-x86-64.zip
[root@QiXiang-DB1 ~]# su – grid
[grid@QiXiang-DB1 ~]$ echo $ORACLE_HOME
/oracle/product/11.2.3/crs11g
[root@QiXiang-DB1 ~]# cd /home/grid/
[root@QiXiang-DB1 grid]# unzip p6880880_112000_Linux-x86-64.zip -d /oracle/product/11.2.3/crs11g
[root@QiXiang-DB1 grid]# cd /oracle/product/11.2.3/crs11g
[root@QiXiang-DB1 crs11g]# chown -R grid.oinstall OPatch
[root@QiXiang-DB1 crs11g]# su – grid
[grid@QiXiang-DB1 ~]$ /oracle/product/11.2.3/crs11g/OPatch/opatch version
OPatch Version: 11.2.0.3.3
OPatch succeeded.
[root@QiXiang-DB1 ~]# su – oracle
[oracle@QiXiang1 /home/oracle]$ echo $ORACLE_HOME
/oracle/product/11.2.3/db11g
[oracle@QiXiang1 /home/oracle]$ cd /oracle/product/11.2.3/db11g
[oracle@QiXiang1 /oracle/product/11.2.3/db11g]$ OPatch/opatch version
Invoking OPatch 11.2.0.1.7
OPatch Version: 11.2.0.1.7
OPatch succeeded.
[oracle@QiXiang1 /oracle/product/11.2.3/db11g]$ mv OPatch/ OPatch_bak
[root@QiXiang-DB1 ~]# cd /home/
[root@QiXiang-DB1 home]# ls -lrt
total 8
drwx—— 5 oracle oinstall 4096 Mar 12 14:59 oracle
drwx—— 7 grid oinstall 4096 Mar 20 22:15 grid
[root@QiXiang-DB1 home]# chmod -R 774 *
[root@QiXiang-DB1 home]# ls -l
total 8
drwxrwxr– 7 grid oinstall 4096 Mar 20 22:15 grid
drwxrwxr– 5 oracle oinstall 4096 Mar 12 14:59 oracle
[root@QiXiang-DB1 ~]# cd /home/grid/
[root@QiXiang-DB1 grid]# unzip p6880880_112000_Linux-x86-64.zip -d /oracle/product/11.2.3/db11g
[root@QiXiang-DB1 grid]# cd /oracle/product/11.2.3/db11g
[root@QiXiang-DB1 db11g]# chown -R oracle.oinstall OPatch
[root@QiXiangS-DB1 db11g]# su – oracle
[oracle@QiXiang1 /home/oracle]$ /oracle/product/11.2.3/db11g/OPatch/opatch version
OPatch Version: 11.2.0.3.3
OPatch succeeded.
[root@QiXiang-DB1 ~]# su – grid
[grid@QiXiang-DB1 ~]$ cd /oracle/product/11.2.3/crs11g/OPatch/ocm/bin/
[grid@QiXiang-DB1 bin]$ ./emocmrsp
OCM Installation Response Generator 10.3.4.0.0 – Production
Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y
The OCM configuration response file (ocm.rsp) was successfully created.
[root@QiXiang-DB1 ~]# su – grid
[grid@QiXiang-DB1 ~]$ echo $ORACLE_HOME
/oracle/product/11.2.3/crs11g
[grid@QiXiang-DB1 ~]$ /oracle/product/11.2.3/crs11g/OPatch/opatch lsinventory -detail -oh /oracle/product/11.2.3/crs11g
……
There are no Interim patches installed in this Oracle Home.
Rac system comprising of multiple nodes
Local node =QiXiang-DB1
Remote node = QiXiang-DB2
——————————————————————————–
OPatch succeeded.
[root@QiXiang-DB1 ~]# su – oracle
[oracle@QiXiang1 /home/oracle]$ /oracle/product/11.2.3/db11g/OPatch/opatch lsinventory -detail -oh /oracle/product/11.2.3/db11g
……
There are no Interim patches installed in this Oracle Home.
Rac system comprising of multiple nodes
Local node =QiXiang-db1
Remote node = QiXiang-db2
——————————————————————————–
OPatch succeeded.
备份目录：
su – oracle
tar cvf db11g.tar /oracle/product/11.2.3/db11g
su – grid
tar cvf crs11g.tar /oracle/product/11.2.3/crs11g
在第二个节点上做以上重复的步骤
[oracle@QiXiang1 /home/oracle]$ emctl stop dbconsole
开始打PSU补丁操作：
在节点1上执行：
[root@QiXiang-DB1 ~]#/oracle/product/11.2.3/crs11g/OPatch/opatch auto /home/grid/psu5 -ocmrf /oracle/product/11.2.3/crs11g/OPatch/ocm/bin/ocm.rsp
Executing /oracle/product/11.2.3/crs11g/perl/bin/perl /oracle/product/11.2.3/crs11g/OPatch/crs/patch11203.pl -patchdir /home/grid -patchn psu5 -ocmrf /oracle/product/11.2.3/crs11g/OPatch/ocm/bin/ocm.rsp -paramfile /oracle/product/11.2.3/crs11g/crs/install/crsconfig_params
/oracle/product/11.2.3/crs11g/crs/install/crsconfig_params
/oracle/product/11.2.3/crs11g/crs/install/s_crsconfig_defs
This is the main log file: /oracle/product/11.2.3/crs11g/cfgtoollogs/opatchauto2013-03-21_05-25-34.log
This file will show your detected configuration and all the steps that opatchauto attempted to do on your system: /oracle/product/11.2.3/crs11g/cfgtoollogs/opatchauto2013-03-21_05-25-34.report.log
2013-03-21 05:25:34: Starting Clusterware Patch Setup
Using configuration parameter file: /oracle/product/11.2.3/crs11g/crs/install/crsconfig_params
Unable to determine if /oracle/product/11.2.3/db11g is shared oracle home
Enter ‘yes’ if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
Unable to determine if /oracle/product/11.2.3/crs11g is shared oracle home
Enter ‘yes’ if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
patch /home/grid/psu5/15876003/custom/server/15876003 apply successful for home /oracle/product/11.2.3/db11g
patch /home/grid/psu5/14727310 apply successful for home /oracle/product/11.2.3/db11g
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘QiXiang-db1′
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘QiXiang-db1′
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘QiXiang-db1′
CRS-2673: Attempting to stop ‘ora.oc4j’ on ‘QiXiang-db1′
CRS-2673: Attempting to stop ‘ora.GRID_DG.dg’ on ‘QiXiang-db1′
CRS-2673: Attempting to stop ‘ora.registry.acfs’ on ‘QiXiang-db1′
CRS-2673: Attempting to stop ‘ora.DATA_DG.dg’ on ‘QiXiang-db1′
CRS-2673: Attempting to stop ‘ora.FLASH_DG.dg’ on ‘QiXiang-db1′
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘QiXiang-db1′
CRS-2673: Attempting to stop ‘ora.cvu’ on ‘QiXiang-db1′
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN1.lsnr’ on ‘QiXiang-db1′
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘QiXiang-db1′ succeeded
CRS-2673: Attempting to stop ‘ora.QiXiang-db1.vip’ on ‘QiXiang-db1′
CRS-2677: Stop of ‘ora.LISTENER_SCAN1.lsnr’ on ‘QiXiang-db1′ succeeded
CRS-2673: Attempting to stop ‘ora.scan1.vip’ on ‘QiXiang-db1′
CRS-2677: Stop of ‘ora.QiXiang-db1.vip’ on ‘QiXiang-db1′ succeeded
CRS-2672: Attempting to start ‘ora.QiXiang-db1.vip’ on ‘QiXiang-db2′
CRS-2677: Stop of ‘ora.scan1.vip’ on ‘QiXiang-db1′ succeeded
CRS-2672: Attempting to start ‘ora.scan1.vip’ on ‘QiXiang-db2′
CRS-2677: Stop of ‘ora.FLASH_DG.dg’ on ‘QiXiang-db1′ succeeded
CRS-2677: Stop of ‘ora.DATA_DG.dg’ on ‘QiXiang-db1′ succeeded
CRS-2677: Stop of ‘ora.registry.acfs’ on ‘QiXiang-db1′ succeeded
CRS-2676: Start of ‘ora.scan1.vip’ on ‘QiXiang-db2′ succeeded
CRS-2672: Attempting to start ‘ora.LISTENER_SCAN1.lsnr’ on ‘QiXiang-db2′
CRS-2676: Start of ‘ora.QiXiang-db1.vip’ on ‘QiXiang-db2′ succeeded
CRS-2676: Start of ‘ora.LISTENER_SCAN1.lsnr’ on ‘QiXiang-db2′ succeeded
CRS-2677: Stop of ‘ora.oc4j’ on ‘QiXiang-db1′ succeeded
CRS-2672: Attempting to start ‘ora.oc4j’ on ‘QiXiang-db2′
CRS-2677: Stop of ‘ora.cvu’ on ‘QiXiang-db1′ succeeded
CRS-2672: Attempting to start ‘ora.cvu’ on ‘QiXiang-db2′
CRS-2676: Start of ‘ora.cvu’ on ‘QiXiang-db2′ succeeded
CRS-2676: Start of ‘ora.oc4j’ on QiXiang-db2′ succeeded
CRS-2677: Stop of ‘ora.GRID_DG.dg’ on ‘QiXiang-db1′ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘QiXiang-db1′
CRS-2677: Stop of ‘ora.asm’ on ‘QiXiang-db1′ succeeded
CRS-2673: Attempting to stop ‘ora.ons’ on ‘QiXiang-db1′
CRS-2677: Stop of ‘ora.ons’ on ‘QiXiangs-db1′ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘QiXiang-db1′
CRS-2677: Stop of ‘ora.net1.network’ on ‘QiXiang-db1′ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘QiXiang-db1′ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘QiXiang-db1′ succeeded
CRS-2673: Attempting to stop ‘ora.drivers.acfs’ on ‘QiXiang-db1′
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘QiXiang-db1′
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘QiXiang-db1′
CRS-2673: Attempting to stop ‘ora.asm’ on QiXiangs-db1′
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘QiXiang-db1′
CRS-2677: Stop of ‘ora.evmd’ on ‘QiXiang-db1′ succeeded
CRS-2677: Stop of ‘ora.mdnsd’ on ‘acars-db1′ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘acars-db1′ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘QiXiang-db1′
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘QiXiangs-db1′ succeeded
CRS-2677: Stop of ‘ora.drivers.acfs’ on ‘QiXiang-db1′ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘QiXiang-db1′ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘QiXiang-db1′
CRS-2677: Stop of ‘ora.cssd’ on ‘QiXiang-db1′ succeeded
CRS-2673: Attempting to stop ‘ora.crf’ on ‘QiXiang-db1′
CRS-2677: Stop of ‘ora.crf’ on ‘QiXiang-db1′ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘QiXiang-db1′
CRS-2677: Stop of ‘ora.gipcd’ on ‘QiXiang-db1′ succeeded
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘QiXiang-db1′
CRS-2677: Stop of ‘ora.gpnpd’ on ‘QiXiang-db1′ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘QiXiang-db1′ has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /oracle/product/11.2.3/crs11g
patch /home/grid/psu5/15876003 apply successful for home /oracle/product/11.2.3/crs11g
patch /home/grid/psu5/14727310 apply successful for home /oracle/product/11.2.3/crs11g
CRS-4123: Oracle High Availability Services has been started.
在节点2上执行：
[root@QiXiang-DB2 ~]#/oracle/product/11.2.3/crs11g/OPatch/opatch auto /home/grid/psu5 -ocmrf /oracle/product/11.2.3/crs11g/OPatch/ocm/bin/ocm.rsp
日志省略
在其中一个节点上执行（只需要在一个节点上执行）：
cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP
SQL> @catbundle.sql psu apply
SQL> QUIT
SQL> select action,comments from registry$history;
ACTION COMMENTS
—————————— —————————————-
APPLY Patchset 11.2.0.2.0
APPLY Patchset 11.2.0.2.0
APPLY PSU 11.2.0.3.5
来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/29960155/viewspace-1363557/ ，如需转载，请注明出处，否则将追究法律责任。
