环境：HP-UX 11.31 ia64
数据库版本：11.2.0.4
是否RAC：是
文件系统或ASM：ASM


事件描述：
最近新装了一套RAC，应用测试连接不上数据库，连接直接报TNS-12547: TNS:lost contact，
但查看监听状态，CRS状态，数据库状态均正常。CRS日志、CSS日志及AGENT日志均无报错。


处理过程详细：
一、连上数据库查看监听状态，显示正常：
racdb1:/home/oracle(racdb1)$lsnrctl status


LSNRCTL for HPUX: Version 11.2.0.4.0 - Production on 30-JUL-2014 10:13:02


Copyright (c) 1991, 2013, Oracle.  All rights reserved.


Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for HPUX: Version 11.2.0.4.0 - Production
Start Date                23-JUL-2014 16:54:43
Uptime                    0 days 17 hr. 18 min. 19 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /oracle/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /oracle/app/grid/diag/tnslsnr/racdb1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.192.10.210)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.192.10.212)(PORT=1521)))
Services Summary...
Service "racdb" has 1 instance(s).
  Instance "racdb1", status READY, has 1 handler(s) for this service...
Service "racdbXDB" has 1 instance(s).
  Instance "racdb1", status READY, has 1 handler(s) for this service...
The command completed successfully


二、连入数据库切换日志，日志可以切换，查看db alert日志无异常发现。
racdb1:/home/oracle(racdb1)$sqlplus / as sysdba


SQL*Plus: Release 11.2.0.4.0 Production on Wed Jul 30 10:14:31 2014


Copyright (c) 1982, 2013, Oracle.  All rights reserved.




Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options


SQL> alter system switch logfile;


System altered.


SQL> 


三、查看监听日志发现一直在报连接失败：
29-JUL-2014 16:52:30 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVER=DEDICATED)(SERVICE_NAME=racdb)(FAILOVER_MODE=(TYPE=SELECT)(METHOD=BASIC)(RETRIES=20
)(DELAY=5))) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.192.10.63)(PORT=13662)) * establish * racdb * 12518
TNS-12518: TNS:listener could not hand off client connection
 TNS-12547: TNS:lost contact
  TNS-12560: TNS:protocol adapter error
   TNS-00517: Lost contact
    HPUX Error: 32: Broken pipe
29-JUL-2014 16:52:31 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVER=DEDICATED)(SERVICE_NAME=racdb)(FAILOVER_MODE=(TYPE=SELECT)(METHOD=BASIC)(RETRIES=20
)(DELAY=5))) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.192.10.63)(PORT=13663)) * establish * racdb * 12518
TNS-12518: TNS:listener could not hand off client connection
 TNS-12547: TNS:lost contact
  TNS-12560: TNS:protocol adapter error
   TNS-00517: Lost contact
    HPUX Error: 32: Broken pipe
29-JUL-2014 16:52:32 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVER=DEDICATED)(SERVICE_NAME=racdb)(FAILOVER_MODE=(TYPE=SELECT)(METHOD=BASIC)(RETRIES=20
)(DELAY=5))) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.192.10.63)(PORT=13668)) * establish * racdb * 12518
TNS-12518: TNS:listener could not hand off client connection
 TNS-12547: TNS:lost contact
  TNS-12560: TNS:protocol adapter error
   TNS-00517: Lost contact
    HPUX Error: 32: Broken pipe


四、分析问题的原因，导致的原因有在32位平台中当listener.log超过2G会报这个错，
ORACLE_HOME下的一些执行文件权限不对也会导致相同的错误，但我们这个是64位的，排除第一种，
所以去查询执行文件的权限是否正常。
1)查看问题节点的oracle执行文件权限
racdb1:/oracle/app/grid/diag/tnslsnr/racdb1/listener/trace(racdb1)$ ls -l $ORACLE_HOME/bin/oracle
-rwxr-s--x   1 oracle     asmadmin   573971520 Jun 23 15:44 /oracle/app/oracle/product/11.2.0/db/bin/oracle
2)查看正常节点的oracle执行文件权限
racdb2:/home/oracle(racdb2)$ls -l $ORACLE_HOME/bin/oracle
-rwsr-s--x   1 oracle     asmadmin   573971520 Jun 23 16:20 /oracle/app/oracle/product/11.2.0/db/bin/oracle
3)对比执行文件发现文件权限不一致，问题节点少了S权限。


五、为了防止其他执行文件都有类似的权限问题，故对其作了relink all,重新同步执行文件。
racdb1:/oracle/app/grid/diag/tnslsnr/racdb1/listener/trace(racdb1)$cd $ORACLE_HOME/bin
racdb1:/oracle/app/oracle/product/11.2.0/db/bin(racdb1)$relink all
writing relink log to: /oracle/app/oracle/product/11.2.0/db/install/relink.log


racdb1:/oracle/app/oracle/product/11.2.0/db/bin(racdb1)$ ls -l $ORACLE_HOME/bin/oracle
-rwsr-s--x   1 oracle     oinstall   573971520 Jul 29 17:15 /oracle/app/oracle/product/11.2.0/db/bin/oracle


六、relink all之后权限正常了，但sqlplus进去发现没有连上数据库，shutdown数据库报错，CRS状态正常。
racdb1:/oracle/app/oracle/product/11.2.0/db/bin(racdb1)$sqlplus / as sysdba


SQL*Plus: Release 11.2.0.4.0 Production on Tue Jul 29 17:17:46 2014


Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected.
SQL> exit
Disconnected
racdb1:/oracle/app/oracle/product/11.2.0/db/bin(racdb1)$
racdb1:/oracle/app/oracle/product/11.2.0/db/bin(racdb1)$crsctl stat res -t 
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       racdb1                                      
               ONLINE  ONLINE       racdb2                                      
ora.gsd
               OFFLINE OFFLINE      racdb1                                      
               OFFLINE OFFLINE      racdb2                                      
ora.net1.network
               ONLINE  ONLINE       racdb1                                      
               ONLINE  ONLINE       racdb2                                      
ora.ons
               ONLINE  ONLINE       racdb1                                      
               ONLINE  ONLINE       racdb2                                      
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       racdb1                                      
ora.cvu
      1        ONLINE  ONLINE       racdb1                                      
ora.oc4j
      1        OFFLINE OFFLINE                                                   
ora.racdb.db
      1        ONLINE  ONLINE       racdb1                  Open                
      2        ONLINE  ONLINE       racdb2                  Open                
ora.racdb1.vip
      1        ONLINE  ONLINE       racdb1                                      
ora.racdb2.vip
      1        ONLINE  ONLINE       racdb2                                      
ora.scan1.vip
      1        ONLINE  ONLINE       racdb1  
      
racdb1:/oraclelog/diag/rdbms/racdb/racdb1/trace(racdb1)$sqlplus / as sysdba


SQL*Plus: Release 11.2.0.4.0 Production on Tue Jul 29 17:18:49 2014


Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected.
SQL> shutdown immediate
ORA-24324: service handle not initialized
ORA-24323: value not allowed
ORA-27140: attach to post/wait facility failed
ORA-27300: OS system dependent operation:invalid_egid failed with status: 1
ORA-27301: OS failure message: Not owner
ORA-27302: failure occurred at: skgpwinit6
ORA-27303: additional information: startup egid = 1100 (asmadmin), current egid = 1000 (oinstall)
SQL> exit
Disconnected


七、从报错日志我们可以清晰的看到数据库是asmadmin属组的执行文件启动的，而当前的属组是oinstall，
故将正常节点跟问题节点属组是asmadmin的文件全部找出来进行比对并修改问题节点文件的属组。
racdb1:/oraclelog/diag/rdbms/racdb/racdb1/trace(racdb1)$ls -l $ORACLE_HOME/bin/oracle
-rwsr-s--x   1 oracle     oinstall   573971520 Jul 29 17:15 /oracle/app/oracle/product/11.2.0/db/bin/oracle
问题节点：
racdb1:/oraclelog/diag/rdbms/racdb/racdb1/trace(racdb1)$find /oracle/app/oracle/product/11.2.0/db -group asmadmin | xargs ls -dlrt 
-rwxr-s--x   1 oracle     asmadmin   573971520 Jun 23 15:44 /oracle/app/oracle/product/11.2.0/db/bin/oracleO
-rwxrwx---   1 oracle     asmadmin     12288 Jun 23 16:50 /oracle/app/oracle/product/11.2.0/db/dbs/peshm_racdb_1/JOXSHM_EXT_5_racdb1_65557.so
-rwxrwx---   1 oracle     asmadmin      4096 Jun 23 16:50 /oracle/app/oracle/product/11.2.0/db/dbs/peshm_racdb_1/JOXSHM_EXT_4_racdb1_65557.so
-rwxrwx---   1 oracle     asmadmin      4096 Jun 23 16:50 /oracle/app/oracle/product/11.2.0/db/dbs/peshm_racdb_1/JOXSHM_EXT_3_racdb1_65557.so
-rwxrwx---   1 oracle     asmadmin      4096 Jun 23 16:50 /oracle/app/oracle/product/11.2.0/db/dbs/peshm_racdb_1/JOXSHM_EXT_2_racdb1_65557.so
。。。。。。
-rwxrwx---   1 oracle     asmadmin     16384 Jun 23 16:51 /oracle/app/oracle/product/11.2.0/db/dbs/peshm_racdb_1/JOXSHM_EXT_712_racdb1_65557.so
-rwxrwx---   1 oracle     asmadmin      8192 Jun 23 16:51 /oracle/app/oracle/product/11.2.0/db/dbs/peshm_racdb_1/JOXSHM_EXT_716_racdb1_65557.so
-rwxrwx---   1 oracle     asmadmin     16384 Jun 23 16:51 /oracle/app/oracle/product/11.2.0/db/dbs/peshm_racdb_1/JOXSHM_EXT_715_racdb1_65557.so
drwx------   2 oracle     asmadmin        96 Jul 21 17:18 /oracle/app/oracle/product/11.2.0/db/dbs/peshm_DBUA1804573_0
drwx------   2 oracle     asmadmin        96 Jul 22 14:44 /oracle/app/oracle/product/11.2.0/db/dbs/peshm_DBUA4427703_0
drwx------   2 oracle     asmadmin        96 Jul 22 16:32 /oracle/app/oracle/product/11.2.0/db/dbs/peshm_DBUA3234315_0
-rw-r-----   1 oracle     asmadmin       116 Jul 22 17:27 /oracle/app/oracle/product/11.2.0/db/rdbms/log/dp.log
-rw-r-----   1 oracle     asmadmin      1161 Jul 22 18:10 /oracle/app/oracle/product/11.2.0/db/rdbms/admin/catbundle_PSU_racdb_APPLY.sql
-rw-r-----   1 oracle     asmadmin      1210 Jul 22 18:10 /oracle/app/oracle/product/11.2.0/db/rdbms/admin/catbundle_PSU_racdb_ROLLBACK.sql
-rw-rw----   1 oracle     asmadmin      1544 Jul 23 17:57 /oracle/app/oracle/product/11.2.0/db/dbs/hc_racdb1.dat
-rw-r-----   1 oracle     asmadmin       770 Jun 23 11:15 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_25869_20140623111537590613143795.aud
-rw-r-----   1 oracle     asmadmin       768 Jun 23 12:03 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_1578_20140623120338071335143795.aud
-rw-r-----   1 oracle     asmadmin       768 Jun 23 14:18 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_4965_20140623141844878862143795.aud
-rw-r-----   1 oracle     asmadmin       769 Jun 23 14:18 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_4965_20140623141845143404143795.aud
-rw-r-----   1 oracle     asmadmin       768 Jun 23 14:18 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_4979_20140623141851427440143795.aud
-rw-r-----   1 oracle     asmadmin       770 Jun 23 15:20 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_13235_20140623152042817429143795.aud
-rw-r-----   1 oracle     asmadmin       770 Jun 23 16:08 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_26406_20140623160826869470143795.aud
-rw-r-----   1 oracle     asmadmin       768 Jun 23 17:00 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_3128_20140623170026275422143795.aud
-rw-r-----   1 oracle     asmadmin       768 Jun 25 17:18 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_3994_20140625171835029768143795.aud
-rw-r-----   1 oracle     asmadmin       759 Jun 25 17:22 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_4670_20140625172235345392143795.aud
-rw-r-----   1 oracle     asmadmin       761 Jul 18 12:01 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_26942_20140718120142032682143795.aud
-rw-r-----   1 oracle     asmadmin       776 Jul 21 17:18 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/DBUA1804573_ora_3770_20140721171810270743143795.aud
-rw-r-----   1 oracle     asmadmin       774 Jul 21 17:18 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/DBUA1804573_ora_3770_20140721171837523949143795.aud
-rw-r-----   1 oracle     asmadmin       967 Jul 21 17:18 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/DBUA1804573_ora_4455_20140721171837701239143795.aud
-rw-r-----   1 oracle     asmadmin       770 Jul 21 17:33 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_15458_20140721173308739486143795.aud
-rw-r-----   1 oracle     asmadmin       770 Jul 21 17:37 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_18255_20140721173722473913143795.aud
-rw-r-----   1 oracle     asmadmin       770 Jul 21 17:42 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_22195_20140721174257759089143795.aud
-rw-r-----   1 oracle     asmadmin       778 Jul 22 14:44 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/DBUA4427703_ora_22297_20140722144429274228143795.aud
-rw-r-----   1 oracle     asmadmin       776 Jul 22 14:44 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/DBUA4427703_ora_22297_20140722144455293169143795.aud
-rw-r-----   1 oracle     asmadmin       969 Jul 22 14:44 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/DBUA4427703_ora_22412_20140722144455382457143795.aud
-rw-r-----   1 oracle     asmadmin     21903 Jul 21 17:40 /oracle/app/oracle/product/11.2.0/db/ccr/state/racdb.ll
-rw-r-----   1 oracle     asmadmin       278 Jul 21 17:40 /oracle/app/oracle/product/11.2.0/db/ccr/state/racdb.ll-stat
-rw-r-----   1 oracle     asmadmin       770 Jul 22 14:47 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_24426_20140722144756679243143795.aud
-rw-r-----   1 oracle     asmadmin       778 Jul 22 16:32 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/DBUA3234315_ora_26529_20140722163240095776143795.aud
-rw-r-----   1 oracle     asmadmin       969 Jul 22 16:33 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/DBUA3234315_ora_27145_20140722163307623414143795.aud
-rw-r-----   1 oracle     asmadmin       776 Jul 22 16:33 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/DBUA3234315_ora_26529_20140722163307491047143795.aud
-rw-r-----   1 oracle     asmadmin       770 Jul 22 17:15 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_25586_20140722171512623374143795.aud
-rw-r-----   1 oracle     asmadmin       768 Jul 22 18:11 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_3152_20140722181155093598143795.aud
-rw-r--r--   1 oracle     asmadmin     12466 Jul 22 18:13 /oracle/app/oracle/product/11.2.0/db/ccr/state/racdb-RAC.ll-stat
-rw-r-----   1 oracle     asmadmin       759 Jul 23 09:24 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_5492_20140723092443335121143795.aud
-rw-r-----   1 oracle     asmadmin       770 Jul 23 09:44 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_18896_20140723094405646296143795.aud
-rw-r-----   1 oracle     asmadmin       770 Jul 23 09:52 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_24400_20140723095249279502143795.aud
-rw-r-----   1 oracle     asmadmin       770 Jul 23 17:56 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb1_ora_18965_20140723175608392118143795.aud
-rw-r--r--   1 oracle     asmadmin       891 Jul 29 01:01 /oracle/app/oracle/product/11.2.0/db/ccr/state/racdb-RAC.ccr
-rw-r--r--   1 oracle     asmadmin     67057 Jul 29 01:01 /oracle/app/oracle/product/11.2.0/db/ccr/state/racdb-RAC.ll


正常节点：
-rwsr-s--x   1 oracle     asmadmin   573962928 Jun 23 10:33 /oracle/app/oracle/product/11.2.0/db/bin/oracleO
-rw-r-----   1 oracle     asmadmin       768 Jun 23 11:15 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb2_ora_6906_20140623111553826625143795.aud
drwx------   2 oracle     asmadmin        96 Jun 23 11:15 /oracle/app/oracle/product/11.2.0/db/dbs/peshm_racdb_2
-rw-r-----   1 oracle     asmadmin       770 Jun 23 12:05 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb2_ora_12624_20140623120503589321143795.aud
-rw-r-----   1 oracle     asmadmin       769 Jun 23 14:18 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb2_ora_4941_20140623141844933369143795.aud
-rw-r-----   1 oracle     asmadmin       768 Jun 23 14:18 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb2_ora_4941_20140623141844871557143795.aud
-rw-r-----   1 oracle     asmadmin       768 Jun 23 14:18 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb2_ora_4953_20140623141851159498143795.aud
-rw-r-----   1 oracle     asmadmin       770 Jun 23 15:22 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb2_ora_13867_20140623152257095498143795.aud
-rwsr-s--x   1 oracle     asmadmin   573971520 Jun 23 16:20 /oracle/app/oracle/product/11.2.0/db/bin/oracle
-rw-r-----   1 oracle     asmadmin       768 Jun 23 16:46 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb2_ora_1343_20140623164613613196143795.aud
-rw-r-----   1 oracle     asmadmin       768 Jun 23 17:08 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb2_ora_4375_20140623170816905854143795.aud
-rw-r-----   1 oracle     asmadmin       770 Jun 25 17:26 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb2_ora_21799_20140625172612676435143795.aud
drwx------   2 oracle     asmadmin        96 Jul 14 14:43 /oracle/app/oracle/product/11.2.0/db/dbs/peshm_DBUA4321613_0
drwxr-xr-x   3 oracle     asmadmin        96 Jul 14 14:43 /oracle/app/oracle/product/11.2.0/db/log/diag
-rw-r-----   1 oracle     asmadmin       778 Jul 14 14:43 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/DBUA4321613_ora_26459_20140714144323250932143795.aud
-rw-r-----   1 oracle     asmadmin       969 Jul 14 14:43 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/DBUA4321613_ora_26548_20140714144347500235143795.aud
-rw-r-----   1 oracle     asmadmin       776 Jul 14 14:43 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/DBUA4321613_ora_26459_20140714144347408564143795.aud
drwxr-x---   2 oracle     asmadmin        96 Jul 14 14:43 /oracle/app/oracle/product/11.2.0/db/log/diag/rdbms
-rw-r-----   1 oracle     asmadmin       770 Jul 14 14:46 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb2_ora_26855_20140714144624985968143795.aud
-rw-r-----   1 oracle     asmadmin       770 Jul 21 17:43 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb2_ora_27480_20140721174300653127143795.aud
-rw-r-----   1 oracle     asmadmin       768 Jul 22 18:11 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb2_ora_2792_20140722181158000036143795.aud
-rw-r-----   1 oracle     asmadmin       770 Jul 23 09:46 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb2_ora_19286_20140723094656723711143795.aud
-rw-r-----   1 oracle     asmadmin        11 Jul 23 17:59 /oracle/app/oracle/product/11.2.0/db/log/diag/adrci_dir.mif
-rw-rw----   1 oracle     asmadmin      1544 Jul 23 18:00 /oracle/app/oracle/product/11.2.0/db/dbs/hc_racdb2.dat
-rw-r-----   1 oracle     asmadmin       770 Jul 23 09:55 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb2_ora_25292_20140723095555923366143795.aud
-rw-r-----   1 oracle     asmadmin       768 Jul 23 17:58 /oracle/app/oracle/product/11.2.0/db/rdbms/audit/racdb2_ora_5558_20140723175850541243143795.aud
-rw-r--r--   1 oracle     asmadmin       891 Jul 27 01:01 /oracle/app/oracle/product/11.2.0/db/ccr/state/racdb-RAC.ccr
-rw-r--r--   1 oracle     asmadmin     67057 Jul 29 01:01 /oracle/app/oracle/product/11.2.0/db/ccr/state/racdb-RAC.ll
修改问题节点文件的属组(修改前属组为oinstall，修改后属组为asmadmin)
root@racdb1:/oracle/app/oracle/product/11.2.0/db/bin#chown oracle:asmadmin oracle
root@racdb1:/oracle/app/oracle/product/11.2.0/db/bin#ls -lrt oracle
-rwsr-s--x   1 oracle     asmadmin   573971520 Jul 29 17:15 oracle
root@racdb1:/oracle/app/oracle/product/11.2.0/db/dbs#ls -lrtd /oracle/app/oracle/product/11.2.0/db/log/diag
drwxrwxr-x   3 oracle     oinstall        96 Jun 23 10:48 /oracle/app/oracle/product/11.2.0/db/log/diag
root@racdb1:/oracle/app/oracle/product/11.2.0/db/dbs#cd /oracle/app/oracle/product/11.2.0/db/log/    
root@racdb1:/oracle/app/oracle/product/11.2.0/db/log#chown oracle:asmadmin diag
root@racdb1:/oracle/app/oracle/product/11.2.0/db/log#ls -lrtd /oracle/app/oracle/product/11.2.0/db/log/diag
drwxrwxr-x   3 oracle     asmadmin        96 Jun 23 10:48 /oracle/app/oracle/product/11.2.0/db/log/diag
root@racdb1:/oracle/app/oracle/product/11.2.0/db/log#ls -lrtd /oracle/app/oracle/product/11.2.0/db/log/diag/rdbms
drwxr-x---   2 oracle     oinstall        96 Jul 22 16:33 /oracle/app/oracle/product/11.2.0/db/log/diag/rdbms
root@racdb1:/oracle/app/oracle/product/11.2.0/db/log#cd /oracle/app/oracle/product/11.2.0/db/log/diag/
root@racdb1:/oracle/app/oracle/product/11.2.0/db/log/diag#chown oracle:asmadmin rdbms
root@racdb1:/oracle/app/oracle/product/11.2.0/db/log/diag#ls -lrt
total 2
drwxr-x---   2 oracle     asmadmin        96 Jul 22 16:33 rdbms
-rw-r-----   1 oracle     oinstall        11 Jul 23 17:56 adrci_dir.mif
root@racdb1:/oracle/app/oracle/product/11.2.0/db/log/diag#chown oracle:asmadmin adrci_dir.mif
root@racdb1:/oracle/app/oracle/product/11.2.0/db/log/diag#cd /oracle/app/oracle/product/11.2.0/db/dbs/


八、将文件权限修改之后重启实例
racdb1:/home/oracle(racdb1)$sqlplus / as sysdba


SQL*Plus: Release 11.2.0.4.0 Production on Tue Jul 29 17:34:52 2014


Copyright (c) 1982, 2013, Oracle.  All rights reserved.




Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options


SQL> 
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> !crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       racdb1                                      
               ONLINE  ONLINE       racdb2                                      
ora.gsd
               OFFLINE OFFLINE      racdb1                                      
               OFFLINE OFFLINE      racdb2                                      
ora.net1.network
               ONLINE  ONLINE       racdb1                                      
               ONLINE  ONLINE       racdb2                                      
ora.ons
               ONLINE  ONLINE       racdb1                                      
               ONLINE  ONLINE       racdb2                                      
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       racdb1                                      
ora.cvu
      1        ONLINE  ONLINE       racdb1                                      
ora.oc4j
      1        OFFLINE OFFLINE                                                   
ora.racdb.db
      1        OFFLINE OFFLINE                               Instance Shutdown   
      2        ONLINE  ONLINE       racdb2                  Open                
ora.racdb1.vip
      1        ONLINE  ONLINE       racdb1                                      
ora.racdb2.vip
      1        ONLINE  ONLINE       racdb2                                      
ora.scan1.vip
      1        ONLINE  ONLINE       racdb1                                      


SQL> startup
ORACLE instance started.


Total System Global Area 1.3202E+11 bytes
Fixed Size                  2221816 bytes
Variable Size            4.1339E+10 bytes
Database Buffers         9.0463E+10 bytes
Redo Buffers              214671360 bytes
Database mounted.
Database opened.
SQL> 
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options


九、通过pl sql远程连接数据库，连接成功，查看listener.log，显示已连接上，无报错。
listener.log:
29-JUL-2014 18:03:27 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVER=DEDICATED)(SERVICE_NAME=racdb)(FAILOVER_MODE=(TYPE=SELECT)(METHOD=BASIC)(RETRIES=20
)(DELAY=5))) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.192.10.63)(PORT=36598)) * establish * racdb * 0
29-JUL-2014 18:03:28 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVER=DEDICATED)(SERVICE_NAME=racdb)(FAILOVER_MODE=(TYPE=SELECT)(METHOD=BASIC)(RETRIES=20
)(DELAY=5))) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.192.10.63)(PORT=36602)) * establish * racdb * 0
29-JUL-2014 18:03:29 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVER=DEDICATED)(SERVICE_NAME=racdb)(FAILOVER_MODE=(TYPE=SELECT)(METHOD=BASIC)(RETRIES=20
)(DELAY=5))) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.192.10.63)(PORT=36606)) * establish * racdb * 0
29-JUL-2014 18:03:30 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVER=DEDICATED)(SERVICE_NAME=racdb)(FAILOVER_MODE=(TYPE=SELECT)(METHOD=BASIC)(RETRIES=20
)(DELAY=5))) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.192.10.63)(PORT=36610)) * establish * racdb * 0
29-JUL-2014 18:03:31 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVER=DEDICATED)(SERVICE_NAME=racdb)(FAILOVER_MODE=(TYPE=SELECT)(METHOD=BASIC)(RETRIES=20
)(DELAY=5))) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.192.10.63)(PORT=36616)) * establish * racdb * 0