IPC send completion sync

-------------------
等待事件中出现大量IPC send completion sync
oracle support网站搜索后发现时bug7111619
升级补丁后不在出现
升级过程如下：

-------------------

检查已经安装的patch

[oracle@rac1 OPatch]$ ./opatch lsinventory
Invoking OPatch 10.2.0.4.2

Oracle Interim Patch Installer version 10.2.0.4.2
Copyright (c) 2007, Oracle Corporation.  All rights reserved.

Oracle Home       : /opt/oracle/product/10.2.0/db_1
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 10.2.0.4.2
OUI version       : 10.2.0.4.0
OUI location      : /opt/oracle/product/10.2.0/db_1/oui
Log file location : /opt/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch2010-11-25_14-03-24PM.log

Lsinventory Output file location : /opt/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2010-11-25_14-03-24PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (2):

Oracle Database 10g                                                  10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 3                            10.2.0.4.0
There are 2 products installed in this Oracle Home.

There are no Interim patches installed in this Oracle Home.

Rac system comprising of multiple nodes
  Local node = rac1
  Remote node = rac2

--------------------------------------------------------------------------------

OPatch succeeded.

--------------------------

检查perl版本，必须是5以上

[oracle@rac1 OPatch]$ perl -v

This is perl, v5.8.8 built for x86_64-linux-thread-multi

Copyright 1987-2006, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl".  If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

----------------------------

停掉节点1

[oracle@rac1 7111619]$ srvctl stop instance -d charge -i charge1 -o immediate

[oracle@rac1 7111619]$
[oracle@rac1 7111619]$ crs_stat -t
Name           Type           Target    State     Host       
------------------------------------------------------------
ora....E1.inst application    OFFLINE   OFFLINE              
ora....E2.inst application    ONLINE    ONLINE    rac2       
ora.CHARGE.db  application    ONLINE    ONLINE    rac1       
ora....SM1.asm application    ONLINE    ONLINE    rac1       
ora....C1.lsnr application    ONLINE    ONLINE    rac1       
ora.rac1.gsd   application    ONLINE    ONLINE    rac1       
ora.rac1.ons   application    ONLINE    ONLINE    rac1       
ora.rac1.vip   application    ONLINE    ONLINE    rac1       
ora....SM2.asm application    ONLINE    ONLINE    rac2       
ora....C2.lsnr application    ONLINE    ONLINE    rac2       
ora.rac2.gsd   application    ONLINE    ONLINE    rac2       
ora.rac2.ons   application    ONLINE    ONLINE    rac2       
ora.rac2.vip   application    ONLINE    ONLINE    rac2       

----------------------------

升级节点1

[oracle@rac1 7111619]$ cd /opt/oracle/product/10.2.0/db_1/OPatch/
[oracle@rac1 OPatch]$ ./opatch apply /opt/oracle/7111619/
Invoking OPatch 10.2.0.4.2

Oracle Interim Patch Installer version 10.2.0.4.2
Copyright (c) 2007, Oracle Corporation.  All rights reserved.

Oracle Home       : /opt/oracle/product/10.2.0/db_1
Central Inventory : /opt/oracle/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 10.2.0.4.2
OUI version       : 10.2.0.4.0
OUI location      : /opt/oracle/product/10.2.0/db_1/oui
Log file location : /opt/oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch2010-11-25_14-07-03PM.log

ApplySession applying interim patch '7111619' to OH '/opt/oracle/product/10.2.0/db_1'

Running prerequisite checks...

OPatch detected the node list and the local node from the inventory.  OPatch will patch the local system then propagate the patch to the remote nodes.

This node is part of an Oracle Real Application Cluster.
Remote nodes: 'rac2'
Local node: 'rac1'
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/opt/oracle/product/10.2.0/db_1')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '7111619' for restore. This might take a while...
Backing up files affected by the patch '7111619' for rollback. This might take a while...

Patching component oracle.rdbms, 10.2.0.4.0...
Updating archive file "/opt/oracle/product/10.2.0/db_1/lib/libserver10.a"  with "lib/libserver10.a/ksxp.o"
Running make for target ioracle
ApplySession adding interim patch '7111619' to inventory

Verifying the update...
Inventory check OK: Patch ID 7111619 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 7111619 are present in Oracle Home.

The local system has been patched.  You can restart Oracle instances on it.

Patching in rolling mode.

The node 'rac2' will be patched next.

Please shutdown Oracle instances running out of this ORACLE_HOME on 'rac2'.
(Oracle Home = '/opt/oracle/product/10.2.0/db_1')

Is the node ready for patching? [y|n]
y
User Responded with: Y
Updating nodes 'rac2'
   Apply-related files are:
     FP = "/opt/oracle/product/10.2.0/db_1/.patch_storage/7111619_Dec_31_2009_18_16_32/rac/copy_files.txt"
     DP = "/opt/oracle/product/10.2.0/db_1/.patch_storage/7111619_Dec_31_2009_18_16_32/rac/copy_dirs.txt"
     MP = "/opt/oracle/product/10.2.0/db_1/.patch_storage/7111619_Dec_31_2009_18_16_32/rac/make_cmds.txt"
     RC = "/opt/oracle/product/10.2.0/db_1/.patch_storage/7111619_Dec_31_2009_18_16_32/rac/remote_cmds.txt"

Instantiating the file "/opt/oracle/product/10.2.0/db_1/.patch_storage/7111619_Dec_31_2009_18_16_32/rac/copy_files.txt.instantiated" by replacing $ORACLE_HOME in "/opt/oracle/product/10.2.0/db_1/.patch_storage/7111619_Dec_31_2009_18_16_32/rac/copy_files.txt" with actual path.
Propagating files to remote nodes...
Instantiating the file "/opt/oracle/product/10.2.0/db_1/.patch_storage/7111619_Dec_31_2009_18_16_32/rac/copy_dirs.txt.instantiated" by replacing $ORACLE_HOME in "/opt/oracle/product/10.2.0/db_1/.patch_storage/7111619_Dec_31_2009_18_16_32/rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes...
Instantiating the file "/opt/oracle/product/10.2.0/db_1/.patch_storage/7111619_Dec_31_2009_18_16_32/rac/make_cmds.txt.instantiated" by replacing $ORACLE_HOME in "/opt/oracle/product/10.2.0/db_1/.patch_storage/7111619_Dec_31_2009_18_16_32/rac/make_cmds.txt" with actual path.
Running command on remote node 'rac2':
cd /opt/oracle/product/10.2.0/db_1/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/opt/oracle/product/10.2.0/db_1 || echo REMOTE_MAKE_FAILED::>&2

The node 'rac2' has been patched.  You can restart Oracle instances on it.

There were relinks on remote nodes.  Remember to check the binary size and timestamp on the nodes 'rac2' .
The following make commands were invoked on remote nodes:
'cd /opt/oracle/product/10.2.0/db_1/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/opt/oracle/product/10.2.0/db_1
'

OPatch succeeded.

-----------------------------

启动节点1

[oracle@rac1 OPatch]$ srvctl start instance -d charge -i charge1

[oracle@rac1 ~]$ crs_stat -t
Name           Type           Target    State     Host       
------------------------------------------------------------
ora....E1.inst application    ONLINE    ONLINE    rac1       
ora....E2.inst application    ONLINE    ONLINE    rac2       
ora.CHARGE.db  application    ONLINE    ONLINE    rac1       
ora....SM1.asm application    ONLINE    ONLINE    rac1       
ora....C1.lsnr application    ONLINE    ONLINE    rac1       
ora.rac1.gsd   application    ONLINE    ONLINE    rac1       
ora.rac1.ons   application    ONLINE    ONLINE    rac1       
ora.rac1.vip   application    ONLINE    ONLINE    rac1       
ora....SM2.asm application    ONLINE    ONLINE    rac2       
ora....C2.lsnr application    ONLINE    ONLINE    rac2       
ora.rac2.gsd   application    ONLINE    ONLINE    rac2       
ora.rac2.ons   application    ONLINE    ONLINE    rac2       
ora.rac2.vip   application    ONLINE    ONLINE    rac2       

重复步骤升级节点2
之后这个等待事件不在出现