1.问题现象
1）RAC环境的第一个节点手工启动CRS后没有反应，CRS无法启动。
[root@rac1 ~]# crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
长时间等待……
[root@rac1 ~]# crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.
CRS没启动成功。
查看后台与CRS相关的进程
[root@rac1 ~]# ps -ef | grep d.bin | grep -v grep
[root@rac1 ~]# ps -ef | grep init | grep -v grep
root 1 0 0 14:22 ? 00:00:01 init [5]
此处显示，没有与CRS相关的进程信息。
长时间等待没能等来CRS的成功启动。crsd日志和ocssd日志没有任何信息输出。
[root@rac1 ~]# tail -30f /u01/crs1020/log/rac1/crsd/crsd.log
注释：此处无新信息输出。
[root@rac1 ~]# tail -30f /u01/crs1020/log/rac1/cssd/ocssd.log
注释：此处无新信息输出。
2）RAC第二节点手工启动正常
[root@rac2 ~]# crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
[root@rac2 ~]# crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora....B1.inst application 0/5 0/0 ONLINE OFFLINE
ora....B2.inst application 0/5 0/0 ONLINE OFFLINE
ora.RACDB.db application 0/1 0/1 ONLINE ONLINE rac2
ora....DB1.srv application 0/1 0/0 ONLINE OFFLINE
ora....DB2.srv application 0/1 0/0 ONLINE ONLINE rac2
ora.....taf.cs application 0/1 0/1 ONLINE ONLINE rac2
ora....SM1.asm application 0/5 0/0 ONLINE OFFLINE
ora....C1.lsnr application 0/5 0/0 ONLINE OFFLINE
ora.rac1.gsd application 0/5 0/0 ONLINE OFFLINE
ora.rac1.ons application 0/3 0/0 ONLINE OFFLINE
ora.rac1.vip application 0/0 0/0 ONLINE ONLINE rac2
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2
ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.gsd application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2
ora.rac2.vip application 0/0 0/0 ONLINE ONLINE rac2
[root@rac2 ~]# ps -ef | grep init | grep -v grep
root 1 0 0 14:16 ? 00:00:01 init [5]
root 3081 1 0 14:18 ? 00:00:05 /bin/sh /etc/init.d/init.cssd fatal
root 4767 3081 0 14:21 ? 00:00:00 /bin/sh /etc/init.d/init.cssd daemon
[root@rac2 ~]# ps -ef | grep d.bin | grep -v grep
root 3082 1 1 14:18 ? 00:00:23 /u01/crs1020/bin/crsd.bin reboot
oracle 4764 3080 0 14:21 ? 00:00:01 /u01/crs1020/bin/evmd.bin
oracle 4989 4942 0 14:21 ? 00:00:04 /u01/crs1020/bin/ocssd.bin
2.问题原因
如果细心观察上面的输出信息，便会找到问题的根源。问题节点上有关“init.*”的进程没有启动。
[root@rac1 ~]# ps -ef | grep init | grep -v grep
root 1 0 0 14:22 ? 00:00:01 init [5]
启动这些进程使用操作系统/etc/inittab文件来实现的。
3.问题处理
1）修复/etc/inittab文件
[root@rac1 ~]# vi /etc/inittab
……省略部分信息……
# Run xdm in runlevel 5
x:5:respawn:/etc/X11/prefdm -nodaemon
#h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1
#h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1
#h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1
将最后三行前面的注释符号“#”去掉，修改后如下：
# Run xdm in runlevel 5
x:5:respawn:/etc/X11/prefdm -nodaemon
h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1
h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1
h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1
2）重新加载/etc/inittab
[root@rac1 ~]# init q
[root@rac1 ~]# ps -ef | grep init | grep -v grep
root 1 0 0 14:22 ? 00:00:01 init [5]
root 3419 1 5 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.evmd run
root 3420 1 8 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.cssd fatal
root 3421 1 5 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.crsd run
root 3450 3419 4 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.cssd startcheck
root 3451 3420 0 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.cssd startcheck
root 3471 3451 0 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.cssd runcheck
root 3473 3450 0 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.cssd runcheck
root 3477 3473 0 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.cssd runcheck
[root@rac1 ~]# ps -ef | grep init | grep -v grep
root 1 0 0 14:22 ? 00:00:01 init [5]
root 3419 1 0 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.evmd run
root 3420 1 1 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.cssd fatal
root 3421 1 0 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.crsd run
root 3450 3419 0 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.cssd startcheck
root 3451 3420 0 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.cssd startcheck
root 3679 3421 0 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.cssd startcheck
注意，这里有一个现象。即便这里不使用crsctl start crs命令手工去启动CRS，CRS同样会自动启动。
3）确认启动成功后的CRS状态
[root@rac1 ~]# crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora....B1.inst application 0/5 0/0 ONLINE ONLINE rac1
ora....B2.inst application 0/5 0/0 ONLINE ONLINE rac2
ora.RACDB.db application 0/1 0/1 ONLINE ONLINE rac1
ora....DB1.srv application 0/1 0/0 ONLINE ONLINE rac1
ora....DB2.srv application 0/1 0/0 ONLINE ONLINE rac2
ora.....taf.cs application 0/1 0/1 ONLINE ONLINE rac1
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac1
ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.gsd application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1
ora.rac1.vip application 0/0 0/0 ONLINE ONLINE rac1
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2
ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.gsd application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2
ora.rac2.vip application 0/0 0/0 ONLINE ONLINE rac2
4）确认成功启动后的与CRS相关的后台进程信息
[root@rac1 ~]# ps -ef | grep init | grep -v grep
root 1 0 0 14:22 ? 00:00:01 init [5]
root 3420 1 0 14:51 ? 00:00:01 /bin/sh /etc/init.d/init.cssd fatal
root 3810 3420 0 14:52 ? 00:00:00 /bin/sh /etc/init.d/init.cssd daemon
[root@rac1 ~]# ps -ef | grep d.bin | grep -v grep
root 3421 1 2 14:51 ? 00:00:08 /u01/crs1020/bin/crsd.bin reboot
oracle 3796 3419 1 14:52 ? 00:00:03 /u01/crs1020/bin/evmd.bin
oracle 3972 3924 0 14:52 ? 00:00:02 /u01/crs1020/bin/ocssd.bin
4.小结
这是一个人为误调整inittab文件导致的问题。从问题现象和处理过程中我们应该总结出：在任何实际调整之前，请充分评估和测试修改的影响范围。
Good luck.
secooler
11.01.05
-- The End --
1）RAC环境的第一个节点手工启动CRS后没有反应，CRS无法启动。
[root@rac1 ~]# crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
长时间等待……
[root@rac1 ~]# crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.
CRS没启动成功。
查看后台与CRS相关的进程
[root@rac1 ~]# ps -ef | grep d.bin | grep -v grep
[root@rac1 ~]# ps -ef | grep init | grep -v grep
root 1 0 0 14:22 ? 00:00:01 init [5]
此处显示，没有与CRS相关的进程信息。
长时间等待没能等来CRS的成功启动。crsd日志和ocssd日志没有任何信息输出。
[root@rac1 ~]# tail -30f /u01/crs1020/log/rac1/crsd/crsd.log
注释：此处无新信息输出。
[root@rac1 ~]# tail -30f /u01/crs1020/log/rac1/cssd/ocssd.log
注释：此处无新信息输出。
2）RAC第二节点手工启动正常
[root@rac2 ~]# crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
[root@rac2 ~]# crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora....B1.inst application 0/5 0/0 ONLINE OFFLINE
ora....B2.inst application 0/5 0/0 ONLINE OFFLINE
ora.RACDB.db application 0/1 0/1 ONLINE ONLINE rac2
ora....DB1.srv application 0/1 0/0 ONLINE OFFLINE
ora....DB2.srv application 0/1 0/0 ONLINE ONLINE rac2
ora.....taf.cs application 0/1 0/1 ONLINE ONLINE rac2
ora....SM1.asm application 0/5 0/0 ONLINE OFFLINE
ora....C1.lsnr application 0/5 0/0 ONLINE OFFLINE
ora.rac1.gsd application 0/5 0/0 ONLINE OFFLINE
ora.rac1.ons application 0/3 0/0 ONLINE OFFLINE
ora.rac1.vip application 0/0 0/0 ONLINE ONLINE rac2
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2
ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.gsd application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2
ora.rac2.vip application 0/0 0/0 ONLINE ONLINE rac2
[root@rac2 ~]# ps -ef | grep init | grep -v grep
root 1 0 0 14:16 ? 00:00:01 init [5]
root 3081 1 0 14:18 ? 00:00:05 /bin/sh /etc/init.d/init.cssd fatal
root 4767 3081 0 14:21 ? 00:00:00 /bin/sh /etc/init.d/init.cssd daemon
[root@rac2 ~]# ps -ef | grep d.bin | grep -v grep
root 3082 1 1 14:18 ? 00:00:23 /u01/crs1020/bin/crsd.bin reboot
oracle 4764 3080 0 14:21 ? 00:00:01 /u01/crs1020/bin/evmd.bin
oracle 4989 4942 0 14:21 ? 00:00:04 /u01/crs1020/bin/ocssd.bin
2.问题原因
如果细心观察上面的输出信息，便会找到问题的根源。问题节点上有关“init.*”的进程没有启动。
[root@rac1 ~]# ps -ef | grep init | grep -v grep
root 1 0 0 14:22 ? 00:00:01 init [5]
启动这些进程使用操作系统/etc/inittab文件来实现的。
3.问题处理
1）修复/etc/inittab文件
[root@rac1 ~]# vi /etc/inittab
……省略部分信息……
# Run xdm in runlevel 5
x:5:respawn:/etc/X11/prefdm -nodaemon
#h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1
#h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1
#h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1
将最后三行前面的注释符号“#”去掉，修改后如下：
# Run xdm in runlevel 5
x:5:respawn:/etc/X11/prefdm -nodaemon
h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1
h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1
h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1
2）重新加载/etc/inittab
[root@rac1 ~]# init q
[root@rac1 ~]# ps -ef | grep init | grep -v grep
root 1 0 0 14:22 ? 00:00:01 init [5]
root 3419 1 5 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.evmd run
root 3420 1 8 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.cssd fatal
root 3421 1 5 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.crsd run
root 3450 3419 4 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.cssd startcheck
root 3451 3420 0 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.cssd startcheck
root 3471 3451 0 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.cssd runcheck
root 3473 3450 0 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.cssd runcheck
root 3477 3473 0 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.cssd runcheck
[root@rac1 ~]# ps -ef | grep init | grep -v grep
root 1 0 0 14:22 ? 00:00:01 init [5]
root 3419 1 0 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.evmd run
root 3420 1 1 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.cssd fatal
root 3421 1 0 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.crsd run
root 3450 3419 0 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.cssd startcheck
root 3451 3420 0 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.cssd startcheck
root 3679 3421 0 14:51 ? 00:00:00 /bin/sh /etc/init.d/init.cssd startcheck
注意，这里有一个现象。即便这里不使用crsctl start crs命令手工去启动CRS，CRS同样会自动启动。
3）确认启动成功后的CRS状态
[root@rac1 ~]# crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora....B1.inst application 0/5 0/0 ONLINE ONLINE rac1
ora....B2.inst application 0/5 0/0 ONLINE ONLINE rac2
ora.RACDB.db application 0/1 0/1 ONLINE ONLINE rac1
ora....DB1.srv application 0/1 0/0 ONLINE ONLINE rac1
ora....DB2.srv application 0/1 0/0 ONLINE ONLINE rac2
ora.....taf.cs application 0/1 0/1 ONLINE ONLINE rac1
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac1
ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.gsd application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1
ora.rac1.vip application 0/0 0/0 ONLINE ONLINE rac1
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2
ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.gsd application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2
ora.rac2.vip application 0/0 0/0 ONLINE ONLINE rac2
4）确认成功启动后的与CRS相关的后台进程信息
[root@rac1 ~]# ps -ef | grep init | grep -v grep
root 1 0 0 14:22 ? 00:00:01 init [5]
root 3420 1 0 14:51 ? 00:00:01 /bin/sh /etc/init.d/init.cssd fatal
root 3810 3420 0 14:52 ? 00:00:00 /bin/sh /etc/init.d/init.cssd daemon
[root@rac1 ~]# ps -ef | grep d.bin | grep -v grep
root 3421 1 2 14:51 ? 00:00:08 /u01/crs1020/bin/crsd.bin reboot
oracle 3796 3419 1 14:52 ? 00:00:03 /u01/crs1020/bin/evmd.bin
oracle 3972 3924 0 14:52 ? 00:00:02 /u01/crs1020/bin/ocssd.bin
4.小结
这是一个人为误调整inittab文件导致的问题。从问题现象和处理过程中我们应该总结出：在任何实际调整之前，请充分评估和测试修改的影响范围。
Good luck.
secooler
11.01.05
-- The End --
来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/70034820/viewspace-2994430/ ，如需转载，请注明出处，否则将追究法律责任。
请登录后发表评论
登录
全部评论
最新文章
- 中国OCM联盟隆重落户ITPUB社区，百位OCM已入驻，欢迎更多的OCM加入我们哦！
- OCMU联盟-一个有温度的联盟
- 丢掉盔甲，我们依然是骄傲的骑士——Secooler“三问四看”话认证培训
- 【数据安全】一次惊心动魄的ASM磁盘头损坏故障处理过程带来的深思
- 【参数】恢复db_recovery_file_dest_size参数为默认值“0”方法
- 【如何做快乐的DBA】ITPUB社区第一期博客专家专题“如何做快乐的DBA”
- 【Bitmap Index】B-Tree索引与Bitmap位图索引的锁代价比较研究
- 【RMAN】使用RMAN的Duplicate功能创建物理DataGuard报错（ORA-17627、ORA-17629）处理
- 【CUBE】Oracle分组函数之CUBE魅力
- 【ROLLUP】Oracle分组函数之ROLLUP魅力
