查询了一下Metalink，发现是Oracle的bug。Note:390591.1：Subject: RAC instances cannot be started after applying 10.2.0.2 patchset

Applies to:

Oracle Server - Enterprise Edition - Version: 10.2.0.2 to 10.2.0.2
This problem can occur on any platform.

Symptoms

After applying the 10.2.0.2 patchset the following problem occurs :
The instance can be started only on one node. This is the node where the Oracle Universal Installer was started.

The following messages occur on the offending nodes while trying to startup the DB using an spfile within sqlplus:

SQL> startup nomount
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+SISTEMA/pge/spfilepgec.ora'
ORA-17503: ksfdopn:2 Failed to open file +SISTEMA/pge/spfilepgec.ora
ORA-03113: end-of-file on communication channel
SQL>

and the following error can be seen in the alert log of de ASM instance:

Errors in file /opt/sw/app/oracle/admin/+ASM/udump/+asm1_ora_9677.trc:
ORA-07445: exception encountered: core dump [kkxsyn()+740] [SIGSEGV] [Address not mapped to object] [0x000000168] [] []
Wed Sep 6 18:56:08 2006
Trace dumping is performing id=[cdmp_20060906185608]

No error is dumped in the alert.log of the instance.

If a pfile is used an error message occurs in the alert log of the instance :


replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=23, OS id=928
Wed Sep 6 18:51:20 2006
Errors in file /opt/sw/app/oracle/admin/pge/udump/pgec2_ora_29898.trc:
ORA-07445: exception encountered: core dump [kkxcms()+1160] [SIGSEGV] [Address not mapped to object] [0x000000168] [] []
Wed Sep 6 18:51:22 2006
Trace dumping is performing id=[cdmp_20060906185122]

Cause

Installing the 10.2.0.2 patchset in a RAC installation on any Unix platform does not correctly update the libknlopt.a file on all nodes. The local node where the installer is run does update libknlopt.a but remote nodes do not get the updated file. This can lead to dumps or internal errors on the remote nodes if Oracle is subsequently relinked.

Solution

There are two solutions for this problem:

1) Manual copy of the "libknlopt.a" library to the offending nodes :

- ensure all instances are shut down
- manually copy $ORACLE_HOME/rdbms/lib/libknlopt.a from the local node to all remote nodes
- relink Oracle on all nodes :
make -f ins_rdbms.mk ioracle

2) Install the patchset on every node using the "-local" option:

On Unix:
runInstaller -updateNodeList -local ORACLE_HOME=$ORACLE_HOME CLUSTER_NODES=node1,node2,...

On Windows:
setup.exe -updateNodeList -local ORACLE_HOME=%ORACLE_HOME% CLUSTER_NODES=node1,node2,...