################关于Oracle 11G RAC双节点之间存在防火墙导致只能一个节点运行

#问题背景：

深信服云平台内部分布式防火墙开启后，所有的主机之间都会存在防火墙。开始不知道分布式防火墙的启动。

#问题现象：

故障节点启动集群时会出现异常，导致集群无法完成启动。alert日志:

2021-10-28 07:30:56.944: 

[ohasd(2313)]CRS-2112:The OLR service started on node cmsorcl2.

2021-10-28 07:30:56.955: 

[ohasd(2313)]CRS-1301:Oracle High Availability Service started on node cmsorcl2.

2021-10-28 07:30:56.958: 

[ohasd(2313)]CRS-8017:location: /etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred

2021-10-28 07:30:57.283: 

[/u01/app/11.2.0/grid/bin/oraagent.bin(7198)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/cmsorcl2/agent/ohasd/oraagent_grid/oraagent_grid.log"

2021-10-28 07:31:00.326: 

[/u01/app/11.2.0/grid/bin/orarootagent.bin(7202)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running). 

2021-10-28 07:31:02.503: 

[ohasd(2313)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running). 

2021-10-28 07:31:02.504: 

[gpnpd(8934)]CRS-2328:GPNPD started on node cmsorcl2. 

2021-10-28 07:31:04.840: 

[cssd(10891)]CRS-1713:CSSD daemon is started in clustered mode

2021-10-28 07:31:06.672: 

[ohasd(2313)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE

2021-10-28 07:31:06.673: 

[ohasd(2313)]CRS-2769:Unable to failover resource 'ora.diskmon'.

2021-10-28 07:31:30.624: 

[cssd(10891)]CRS-1707:Lease acquisition for node cmsorcl2 number 2 completed

2021-10-28 07:31:32.008: 

[cssd(10891)]CRS-1605:CSSD voting file is online: /dev/asm-diskc; details in /u01/app/11.2.0/grid/log/cmsorcl2/cssd/ocssd.log.

2021-10-28 07:31:32.048: 

[cssd(10891)]CRS-1605:CSSD voting file is online: /dev/asm-diskb; details in /u01/app/11.2.0/grid/log/cmsorcl2/cssd/ocssd.log.

2021-10-28 07:31:32.089: 

[cssd(10891)]CRS-1605:CSSD voting file is online: /dev/asm-diska; details in /u01/app/11.2.0/grid/log/cmsorcl2/cssd/ocssd.log.

2021-10-28 15:38:34.926: 

[/u01/app/11.2.0/grid/bin/cssdagent(10858)]CRS-5818:Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/11.2.0/grid/log/cmsorcl2/agent/ohasd/oracssdagent_root/oracssdagent_root.log.

2021-10-28 15:38:34.926: 

[cssd(10891)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/11.2.0/grid/log/cmsorcl2/cssd/ocssd.log

2021-10-28 15:38:34.926: 

[cssd(10891)]CRS-1603:CSSD on node cmsorcl2 shutdown by user.

2021-10-28 15:38:40.254: 

[ohasd(2313)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'cmsorcl2'.

2021-10-28 15:38:41.668: 

[cssd(6922)]CRS-1713:CSSD daemon is started in clustered mode

2021-10-28 15:38:43.435: 

[ohasd(2313)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE

2021-10-28 15:38:43.435: 

[ohasd(2313)]CRS-2769:Unable to failover resource 'ora.diskmon'.

2021-10-28 15:38:57.400: 

[cssd(6922)]CRS-1707:Lease acquisition for node cmsorcl2 number 2 completed

#解决方案：

css异常，一般都是节点间的访问异常，在ocssd.log中可查看具体信息，可看到一些私网端口访问异常的信息。

在防火墙中把数据库pub,priv,vip,scan IP之间的限制全部取消。

如果数据库不启用HAIP的话，这时数据库应该就恢复正常了；如果启用了HAIP（默认启用，169段ip），需要把HAIP之间的限制取消掉。如果不取消掉，asm实例无法启动。现象如下：

Wed Nov 17 22:05:45 2021

MMON started with pid=20, OS id=6979 

Wed Nov 17 22:05:45 2021

MMNL started with pid=21, OS id=6981 

lmon registered with NM - instance number 2 (internal mem no 1)

Wed Nov 17 22:07:45 2021

PMON (ospid: 6911): terminating the instance due to error 481              <-----问题现象，参考http://blog.itpub.net/29615408/viewspace-1384760/

Wed Nov 17 22:07:45 2021

ORA-1092 : opitsk aborting process

Wed Nov 17 22:07:45 2021

System state dump requested by (instance=2, osid=6911 (PMON)), summary=[abnormal instance termination].

System State dumped to trace file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_diag_6949_20211117220745.trc

Dumping diagnostic data in directory=[cdmp_20211117220745], requested by (instance=2, osid=6911 (PMON)), summary=[abnormal instance termination].

Instance terminated by PMON, pid = 6911

Wed Nov 17 22:08:06 2021

NOTE: No asm libraries found in the system

双节点之间HAIP有防火墙的现象提示不是很明显，asm的alert日志如上，手动启asm到nomount时报ORA-03113: end-of-file on communication channel ，集群告警日志和trc文件中的报错都没有明确指出HAIP的问题。PMON (ospid: 6911): terminating the instance due to error 481 的出现原因如下：

Case1: link local IP (169.254.x.x) is being used by other adapter/network

  Case2: firewall exists between nodes on private network (iptables etc)

  Case3: HAIP is up on some nodes but not on all

  Case4: HAIP is up on all nodes but some do not have route info

最终关闭了HAIP之间的防火墙策略后数据库恢复！