“oracle  node 1, rac0101, has a disk HB, but no network HB, DHB has r cfg , wrtcnt”

db突然停止运行
alert.log
Archived Log entry 52886 added for thread 2 sequence 27344 ID 0x980341ff dest 1:
Mon Jun 08 22:00:00 2015
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
Mon Jun 08 22:00:00 2015
Starting background process VKRM
Mon Jun 08 22:00:00 2015
VKRM started with pid=291, OS id=21332 
Tue Jun 09 00:11:51 2015
Thread 2 advanced to log sequence 27346 (LGWR switch)
  Current log# 7 seq# 27346 mem# 0: +DATA/oradb/onlinelog/group_7.453.868440067
  Current log# 7 seq# 27346 mem# 1: +DATA/oradb/onlinelog/group_7.457.868440069
Tue Jun 09 00:11:54 2015
Archived Log entry 52888 added for thread 2 sequence 27345 ID 0x980341ff dest 1:
Tue Jun 09 02:00:00 2015
Closing Resource Manager plan via scheduler window
Clearing Resource Manager plan via parameter
Tue Jun 09 12:08:08 2015
WARNING: db_recovery_file_dest is same as db_create_file_dest
Tue Jun 09 14:17:38 2015
NOTE: ASMB terminating
Errors in file /data/oracle/diag/rdbms/oradb/oradb2/trace/oradb2_asmb_25897.trc:
ORA-15064: ? ASM ??????
ORA-03113: ?????????
?? ID: 
?? ID: 386 ???: 5
Errors in file /data/oracle/diag/rdbms/oradb/oradb2/trace/oradb2_asmb_25897.trc:
ORA-15064: ? ASM ??????
ORA-03113: ?????????
?? ID: 
?? ID: 386 ???: 5
ASMB (ospid: 25897): terminating the instance due to error 15064
Instance terminated by ASMB, pid = 25897
Tue Jun 23 14:52:20 2015

alert_+ASM2.log

SQL> ALTER DISKGROUP ALL ENABLE VOLUME ALL /* asm agent *//* {0:8:9} */ 
SUCCESS: ALTER DISKGROUP ALL ENABLE VOLUME ALL /* asm agent *//* {0:8:9} */
Mon Jun 01 12:22:18 2015
WARNING: failed to online diskgroup resource ora.DATA.dg (unable to communicate with CRSD/OHASD)
Mon Jun 01 12:22:18 2015
NOTE: [crsd.bin@rac0102.tempus.cn (TNS V1-V3) 25061] opening OCR file
Starting background process ASMB
Mon Jun 01 12:22:18 2015
ASMB started with pid=25, OS id=25075 
Mon Jun 01 12:22:18 2015
NOTE: client +ASM2:+ASM registered, osid 25077, mbr 0x0
Mon Jun 01 12:22:19 2015
NOTE: Attempting voting file refresh on diskgroup DATA
NOTE: Voting file relocation is required in diskgroup DATA
NOTE: Attempting voting file relocation on diskgroup DATA
Mon Jun 01 12:25:35 2015
ALTER SYSTEM SET local_listener='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=172.16.2.43)(PORT=1521))))' SCOPE=MEMORY SI
D='+ASM2';
Mon Jun 01 12:26:05 2015
NOTE: client oradb2:oradb registered, osid 25903, mbr 0x1
Tue Jun 09 14:17:38 2015
Tue Jun 09 14:17:38 2015
NOTE: client exited [25061]NOTE: ASMB process exiting, either shutdown is in progress 


NOTE: or foreground connected to ASMB was killed. 
Tue Jun 09 14:17:39 2015
Received an instance abort message from instance 1
Tue Jun 09 14:17:39 2015
Received an instance abort message from instance 1
Please check instance 1 alert and LMON trace files for detail.
Please check instance 1 alert and LMON trace files for detail.
opidcl aborting process unknown ospid (25073) as a result of ORA-29709
LMD0 (ospid: 25013): terminating the instance due to error 481
Instance terminated by LMD0, pid = 25013


1.执行crsctl start crs提示服务已启动
2.执行crsctl check crs 
    CRS-4638: Oracle High Availability Services is online
    CRS-4535: Cannot communicate with Cluster Ready Services
    CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
    CRS-4534: Cannot communicate with Event Manager
3.crsctl stop crs -f 强制关闭后重启问题还是一样，cssd服务无法正常启动
4.正常重启系统后启动crsctl start crs还是不行，具体错误如下：
ocssd.log
2015-06-23 13:52:10.599: [    CSSD][1087736128]clssscSelect: cookie accept request 0x2aaab009e7c0
2015-06-23 13:52:10.599: [    CSSD][1087736128]clssscevtypSHRCON: getting client with cmproc 0x2aaab009e7c0
2015-06-23 13:52:10.599: [    CSSD][1087736128]clssgmRegisterClient: proc(4/0x2aaab009e7c0), client(1/0x2aaab009e930)
2015-06-23 13:52:10.599: [    CSSD][1087736128]clssgmJoinGrock: global grock CRF- new client 0x2aaab009e930 with con 0x37e8, request
ed num -1, flags 0x4000e00
2015-06-23 13:52:10.599: [    CSSD][1087736128]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group inf
ormation has been received from the master; group name CRF-, member number -1, flags 0x4000e00
2015-06-23 13:52:10.600: [    CSSD][1087736128]clssgmDiscEndpcl: gipcDestroy 0x37e8
2015-06-23 13:52:10.600: [    CSSD][1087736128]clssgmDeadProc: proc 0x2aaab009e7c0
2015-06-23 13:52:10.600: [    CSSD][1087736128]clssgmDestroyProc: cleaning up proc(0x2aaab009e7c0) con(0x37b9) skgpid  ospid 6002 wi
th 0 clients, refcount 0
2015-06-23 13:52:10.600: [    CSSD][1087736128]clssgmDiscEndpcl: gipcDestroy 0x37b9
2015-06-23 13:52:10.732: [    CSSD][1113405760]clssnmSendingThread: sending join msg to all nodes
2015-06-23 13:52:10.732: [    CSSD][1113405760]clssnmSendingThread: sent 4 join msgs to all nodes
2015-06-23 13:52:11.231: [    CSSD][1104025920]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0
2015-06-23 13:52:11.299: [    CSSD][1097464128]clssnmvDHBValidateNCopy: node 1, rac0101, has a disk HB, but no network HB, DHB has r
cfg 244238741, wrtcnt, 85705272, LATS 3151914, lastSeqNo 85705271, uniqueness 1433132495, timestamp 1435038730/2671193568
2015-06-23 13:52:12.233: [    CSSD][1104025920]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0
2015-06-23 13:52:12.301: [    CSSD][1097464128]clssnmvDHBValidateNCopy: node 1, rac0101, has a disk HB, but no network HB, DHB has r
cfg 244238741, wrtcnt, 85705273, LATS 3152914, lastSeqNo 85705272, uniqueness 1433132495, timestamp 1435038731/2671194568
2015-06-23 13:52:13.235: [    CSSD][1104025920]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0
2015-06-23 13:52:13.303: [    CSSD][1097464128]clssnmvDHBValidateNCopy: node 1, rac0101, has a disk HB, but no network HB, DHB has r
cfg 244238741, wrtcnt, 85705274, LATS 3153914, lastSeqNo 85705273, uniqueness 1433132495, timestamp 1435038732/2671195568
2015-06-23 13:52:14.237: [    CSSD][1104025920]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0
2015-06-23 13:52:14.306: [    CSSD][1097464128]clssnmvDHBValidateNCopy: node 1, rac0101, has a disk HB, but no network HB, DHB has r
cfg 244238741, wrtcnt, 85705275, LATS 3154914, lastSeqNo 85705274, uniqueness 1433132495, timestamp 1435038733/2671196578
2015-06-23 13:52:14.741: [    CSSD][1113405760]clssnmSendingThread: sending join msg to all nodes
2015-06-23 13:52:14.741: [    CSSD][1113405760]clssnmSendingThread: sent 4 join msgs to all nodes
2015-06-23 13:52:15.240: [    CSSD][1104025920]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0
2015-06-23 13:52:15.309: [    CSSD][1097464128]clssnmvDHBValidateNCopy: node 1, rac0101, has a disk HB, but no network HB, DHB has r
cfg 244238741, wrtcnt, 85705276, LATS 3155924, lastSeqNo 85705275, uniqueness 1433132495, timestamp 1435038734/2671197578
2015-06-23 13:52:15.609: [    CSSD][1087736128]clssscSelect: cookie accept request 0x2aaaac02a570
2015-06-23 13:52:15.610: [    CSSD][1087736128]clssgmAllocProc: (0x2aaab009e7c0) allocated
2015-06-23 13:52:15.610: [    CSSD][1087736128]clssgmClientConnectMsg: properties of cmProc 0x2aaab009e7c0 - 1,2,3,4,5
2015-06-23 13:52:15.610: [    CSSD][1087736128]clssgmClientConnectMsg: Connect from con(0x3884) proc(0x2aaab009e7c0) pid(6002) versi
on 11:2:1:4, properties: 1,2,3,4,5
2015-06-23 13:52:15.610: [    CSSD][1087736128]clssgmClientConnectMsg: msg flags 0x0000
2015-06-23 13:52:15.611: [    CSSD][1087736128]clssscSelect: cookie accept request 0x2aaab009e7c0
2015-06-23 13:52:15.611: [    CSSD][1087736128]clssscevtypSHRCON: getting client with cmproc 0x2aaab009e7c0
2015-06-23 13:52:15.611: [    CSSD][1087736128]clssgmRegisterClient: proc(4/0x2aaab009e7c0), client(1/0x2aaab00e3250)
2015-06-23 13:52:15.611: [    CSSD][1087736128]clssgmJoinGrock: global grock CRF- new client 0x2aaab00e3250 with con 0x38b3, request
ed num -1, flags 0x4000e00
2015-06-23 13:52:15.611: [    CSSD][1087736128]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group inf
ormation has been received from the master; group name CRF-, member number -1, flags 0x4000e00
2015-06-23 13:52:15.611: [    CSSD][1087736128]clssgmDiscEndpcl: gipcDestroy 0x38b3
2015-06-23 13:52:15.611: [    CSSD][1087736128]clssgmDeadProc: proc 0x2aaab009e7c0
2015-06-23 13:52:15.611: [    CSSD][1087736128]clssgmDestroyProc: cleaning up proc(0x2aaab009e7c0) con(0x3884) skgpid  ospid 6002 wi
th 0 clients, refcount 0
2015-06-23 13:52:15.611: [    CSSD][1087736128]clssgmDiscEndpcl: gipcDestroy 0x3884

提示两个节点通讯有问题，检查网络好像是正常的
执行crsctl stop has -f 强制停止所有服务，然后通过尝试把心跳线拔下稍等五分钟再插回去，
启动crsctl start has 
系统恢复正常。


以下为参考资料：

问题 2：CRS-4530：联系集群同步服务守护进程时出现通信故障，ocssd.bin 未运行状：

1. 命令“$GRID_HOME/bin/crsctl check crs”返回错误：
    CRS-4638: Oracle High Availability Services is online
    CRS-4535: Cannot communicate with Cluster Ready Services
    CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
    CRS-4534: Cannot communicate with Event Manager
2. 命令“ps -ef | grep d.bin”不显示类似于如下所示的行：
    oragrid 21543 1 1 22:24 ? 00:00:01 /u01/app/11.2.0/grid/bin/ocssd.bin
3. ocssd.bin 正在运行，但在 ocssd.log 中显示消息“CLSGPNP_CALL_AGAIN”后又中止运行
4. ocssd.log 显示如下内容：

   2012-01-27 13:42:58.796: [ CSSD][19]clssnmvDHBValidateNCopy: node 1, racnode1, has a disk HB, but no network HB, DHB has rcfg 223132864, wrtcnt, 1112, LATS 783238209,  
   lastSeqNo 1111, uniqueness 1327692232, timestamp 1327693378/787089065

5. 对于 3 个或更多节点的情况，2 个节点形成的集群一切正常，但是，当第 3 个节点加入时就出现故障，ocssd.log 显示如下内容：

   2012-02-09 11:33:53.048: [ CSSD][1120926016](:CSSNM00008:)clssnmCheckDskInfo: Aborting local node to avoid splitbrain. Cohort of 2 nodes with leader 2, racnode2, is smaller than   
   cohort of 2 nodes led by node 1, racnode1, based on map type 2
   2012-02-09 11:33:53.048: [ CSSD][1120926016]###################################
   2012-02-09 11:33:53.048: [ CSSD][1120926016]clssscExit: CSSD aborting from thread clssnmRcfgMgrThread

6. 10 分钟后 ocssd.bin 启动超时

   2012-04-08 12:04:33.153: [    CSSD][1]clssscmain: Starting CSS daemon, version 11.2.0.3.0, in (clustered) mode with uniqueness value 1333911873
   ......
   2012-04-08 12:14:31.994: [    CSSD][5]clssgmShutDown: Received abortive shutdown request from client.
   2012-04-08 12:14:31.994: [    CSSD][5]###################################
   2012-04-08 12:14:31.994: [    CSSD][5]clssscExit: CSSD aborting from thread GMClientListener
   2012-04-08 12:14:31.994: [    CSSD][5]###################################
   2012-04-08 12:14:31.994: [    CSSD][5](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally

可能的原因：

1. 表决磁盘丢失或无法访问
2. 多播未正常工作（对于 11.2.0.2 及以上版本）
3. 私网未工作，ping 或 traceroute 显示无法访问目标。或虽然 ping/traceroute 正常工作，但是在私网中启用了防火墙
4. 使用正常 ping 命令可对私网进行 ping 操作，但启用巨帧时（MTU：9000+），不能使用巨帧尺寸（如：ping -s 8900 ）进行 ping 操作。或部分集群节点设置了巨帧（MTU：9000），但问题节点未设置巨帧（MTU：1500）
5. gpnpd 未出现，卡在 dispatch 线程中， Bug 10105195
6. 通过 asm_diskstring 发现的磁盘太多，或由于 Bug 13454354 导致扫描太慢（仅在 Solaris 11.2.0.3 上出现）

解决方案：

1. 通过检查存储存取性、磁盘权限等恢复表决磁盘存取。
   如果 OCR ASM 磁盘组中的 voting disk已经丢失，以独占模式启动 CRS，并重建表决磁盘：
   # crsctl start crs -excl
   # crsctl replace votedisk <+OCRVOTE diskgroup>
2. 请参考 Document 1212703.1 ，了解多播功能的测试及修正
3. 咨询网络管理员，恢复私网访问或禁用私网防火墙（对于 Linux，请检查服务 iptables 状态和服务 ip6tables 状态）
4. 如果巨帧在网卡中启用，则联系网络管理员在交换机层也启用。
5. 终止正常运行节点上的 gpnpd.bin 进程，请参考 Document 10105195.8
   一旦以上问题得以解决，请重新启动 Grid Infrastructure。
   如果 ping/traceroute 对私网均可用，但是问题发生在从 11.2.0.1 至 11.2.0.2 升级过程中，请检查 
   Bug 13416559 获取解决方法。
6. 通过提供更加具体的 asm_diskstring，限制 ASM 扫描磁盘的数量，请参考 bug 13583387
   对于 Solaris 11.2.0.3，请应用补丁 13250497，请参阅 Document 1451367.1.