ITPub博客
首页 博客 专栏 学院 社区 IT新媒体评选
发博

首页 > 数据库 > Oracle > 一个CRS CRS-5818 gpnpd、mdnsd进程无法启动案例处理

一个CRS CRS-5818 gpnpd、mdnsd进程无法启动案例处理

原创 Oracle 作者：startay 时间：2016-05-12 22:39:09 0 删除 编辑

crsctl start crs 启动集群后，资源死活拉不起来，重启机器后还是这样。 查看日志发现下面信息:

  1.
  2. 2016-05-12 20:22:18.341:
  3. [gpnpd(4653074)]CRS-2329:GPNPD on node pos5gpp1 shutdown.
  4. 2016-05-12 20:24:15.080:
  5. [/grid/app/11.2/bin/oraagent.bin(5832974)]CRS-5818:Aborted command 'start' for resource 'ora.gpnpd'. Details at (:CRSAGF00113:) {0:0:
  6. 2} in /grid/app/11.2/log/pos5gpp1/agent/ohasd/oraagent_grid/oraagent_grid.log.
  7. 2016-05-12 20:24:19.084:
  8. [ohasd(2359742)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gpnpd'. Details at (:CRSPE00111:) {0:0
  9. :2} in /grid/app/11.2/log/pos5gpp1/ohasd/ohasd.log.
  10. 2016-05-12 20:24:20.560:
  11. [mdnsd(6619232)]CRS-5602:mDNS service stopping by request.
  12. 2016-05-12 20:26:20.471:
  13. [/grid/app/11.2/bin/oraagent.bin(5832976)]CRS-5818:Aborted command 'start' for resource 'ora.mdnsd'. Details at (:CRSAGF00113:) {0:0:
  14. 2} in /grid/app/11.2/log/pos5gpp1/agent/ohasd/oraagent_grid/oraagent_grid.log.
  15. 2016-05-12 20:26:24.476:
  16. [ohasd(2359742)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.mdnsd'. Details at (:CRSPE00111:) {0:0
  17. :2} in /grid/app/11.2/log/pos5gpp1/ohasd/ohasd.log.
  18. 2016-05-12 20:26:29.033:
  19. [gpnpd(3539086)]CRS-2329:GPNPD on node pos5gpp1 shutdown
检查oraagent_grid.log 报错如下
  1. [ clsdmc][1800]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=pos5gpp1DBG_MDNSD)) with status 9
  2. 2016-05-12 20:46:05.720: [ora.mdnsd][1800]{0:0:2} [start] Error = error 9 encountered when connecting to MDNSD
  3. 2016-05-12 20:46:06.720: [ora.mdnsd][1800]{0:0:2} [start] without returnbuf
  4. 2016-05-12 20:46:06.884: [ COMMCRS][1034]clsc_connect: (111ed9190) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=pos5gpp1DBG_MDNSD))
ohasd.log发下以下报错
  1. ================================================================================
  2. 2016-05-12 20:47:19.870: [ default][1]gpnpd START pid=6357104 Oracle Grid Plug-and-Play Daemon
  3. 2016-05-12 20:47:19.870: [ GPNP][1]clsgpnp_Init: [at clsgpnp0.c:586 clsgpnp_Init] '/grid/app/11.2' in effect as GPnP home base.
  4. 2016-05-12 20:47:19.870: [ GPNP][1]clsgpnp_Init: [at clsgpnp0.c:632 clsgpnp_Init] GPnP pid=6357104, GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:ORA_DAEMON_LOGGING_LEVELS, apitl:0, complog:1, tstenv:0, devenv:0, envopt:0, flags=3
  5. 2016-05-12 20:47:22.896: [ GPNP][1]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:399 clsgpnpkwf_initwfloc] Using FS Wallet Location : /grid/app/11.2/gpnp/pos5gpp1/wallets/peer/

  7. [ CLWAL][1]clsw_Initialize: OLR initlevel [70000]
  8. 2016-05-12 20:47:22.988: [ COMMCRS][1029]clsclisten: Permission denied for (ADDRESS=(PROTOCOL=ipc)(KEY=pos5gpp1DBG_GPNPD))

  10. 2016-05-12 20:47:22.988: [ clsdmt][772]Fail to listen to (ADDRESS=(PROTOCOL=ipc)(KEY=pos5gpp1DBG_GPNPD))
  11. 2016-05-12 20:47:22.988: [ clsdmt][772]Terminating process
  12. 2016-05-12 20:47:22.988: [ GPNP][772]CLSDM requested exit
  13. 2016-05-12 20:47:22.988: [ default][772]GPNPD on node pos5gpp1 shutdown.

通过METALINK 发现解决方案 “诊断 Grid Infrastructure 启动问题 (文档 ID 1623340.1)”

当网络的 socket 文件权限或者属主设置不正确的时候，我们通常会上面的类似报错信息。

网络的 socket 文件可能位于目录： /tmp/.oracle, /var/tmp/.oracle or /usr/tmp/.oracle 中。

查看我们的socket文件所属信息：(所属用户基本全是oracle，但我们的CRS的所属应该grid)  <--问题出在这里
  1. grid@aaggpp1:/grid/app/11.2/bin/>ls -l /tmp/.oracle
  2. total 8
  3. srwxrwxrwx 1 oracle oinstall 0 Sep 1 2015 mdnsd
  4. -rw-r--r-- 1 oracle oinstall 8 Sep 1 2015 mdnsd.pid
  5. prw-r--r-- 1 oracle oinstall 0 Jun 10 2015 npohasd
  6. srwxrwxrwx 1 oracle oinstall 0 Sep 1 2015 ora_gipc_GPNPD_pos5gpp1
  7. -rw-r--r-- 1 oracle oinstall 0 Jun 10 2015 ora_gipc_GPNPD_pos5gpp1_lock
  8. srwxrwxrwx 1 oracle oinstall 0 Sep 1 2015 ora_gipc_gipcd_pos5gpp1
  9. -rw-r--r-- 1 oracle oinstall 0 Sep 1 2015 ora_gipc_gipcd_pos5gpp1_lock
  10. srwxrwxrwx 1 root system 0 Jan 4 02:17 ora_gipc_spos5gpp1gridpos5gpp-clusterCRFM_CLIIPC
  11. -rw-r--r-- 1 oracle oinstall 0 Jun 10 2015 ora_gipc_spos5gpp1gridpos5gpp-clusterCRFM_CLIIPC_lock
  12. srwxrwxrwx 1 root system 0 Jan 4 02:17 ora_gipc_spos5gpp1gridpos5gpp-clusterCRFM_SIPC
  13. -rw-r--r-- 1 oracle oinstall 0 Jun 10 2015 ora_gipc_spos5gpp1gridpos5gpp-clusterCRFM_SIPC_lock
  14. srwxrwxrwx 1 oracle oinstall 0 Sep 21 2015 s#48758948.1
  15. srwxrwxrwx 1 oracle oinstall 0 Jun 8 2015 s#66978252.1
  16. srwxrwxrwx 1 oracle oinstall 0 Sep 23 2015 s#8455124.1
  17. srwxrwxrwx 1 oracle oinstall 0 Sep 1 2015 sAevm
  18. srwxrwxrwx 1 oracle oinstall 0 Sep 1 2015 sCRSD_IPC_SOCKET_11
  19. -rw-r--r-- 1 oracle oinstall 0 Jun 10 2015 sCRSD_IPC_SOCKET_11_lock
  20. srwxrwxrwx 1 oracle oinstall 0 Sep 1 2015 sCRSD_UI_SOCKET
  21. srwxrwxrwx 1 oracle oinstall 0 Sep 1 2015 sCevm
  22. srwxrwxrwx 1 oracle oinstall 0 Sep 23 2015 sEXTPROC1521
  23. srwxrwxrwx 1 oracle oinstall 0 Sep 21 2015 sLISTENER
  24. srwxrwxrwx 1 oracle oinstall 0 Sep 1 2015 sOCSSD_LL_pos5gpp1_
  25. -rw-r--r-- 1 oracle oinstall 0 Jun 10 2015 sOCSSD_LL_pos5gpp1__lock
  26. srwxrwxrwx 1 oracle oinstall 0 Sep 1 2015 sOCSSD_LL_pos5gpp1_pos5gpp-cluster
  27. -rw-r--r-- 1 oracle oinstall 0 Jun 10 2015 sOCSSD_LL_pos5gpp1_pos5gpp-cluster_lock
  28. srwxrwxrwx 1 root system 0 May 12 20:13 sOHASD_IPC_SOCKET_11
  29. -rw-r--r-- 1 oracle oinstall 0 Jun 10 2015 sOHASD_IPC_SOCKET_11_lock
  30. srwxrwxrwx 1 root system 0 May 12 20:13 sOHASD_UI_SOCKET
  31. srwxrwxrwx 1 oracle oinstall 0 Sep 1 2015 sOracle_CSS_LclLstnr_pos5gpp-cluster_1
  32. -rw-r--r-- 1 oracle oinstall 0 Jun 10 2015 sOracle_CSS_LclLstnr_pos5gpp-cluster_1_lock
  33. srwxrwxrwx 1 oracle oinstall 0 Sep 1 2015 sSYSTEM.evm.acceptor.auth
  34. srwxrwxrwx 1 oracle oinstall 0 Sep 1 2015 sora_crsqs
  35. srwxrwxrwx 1 oracle oinstall 0 Sep 1 2015 spos5gpp1DBG_CRSD
  36. srwxrwxrwx 1 oracle oinstall 0 Sep 1 2015 spos5gpp1DBG_CSSD
  37. srwxrwxrwx 1 oracle oinstall 0 Sep 1 2015 spos5gpp1DBG_CTSSD
  38. srwxrwxrwx 1 oracle oinstall 0 Sep 1 2015 spos5gpp1DBG_EVMD
  39. srwxrwxrwx 1 oracle oinstall 0 Sep 1 2015 spos5gpp1DBG_GIPCD
  40. srwxrwxrwx 1 oracle oinstall 0 Sep 1 2015 spos5gpp1DBG_GPNPD
  41. srwxrwxrwx 1 root system 0 Dec 17 15:17 spos5gpp1DBG_LOGD
  42. srwxrwxrwx 1 oracle oinstall 0 Sep 1 2015 spos5gpp1DBG_MDNSD
  43. srwxrwxrwx 1 root system 0 Jan 4 02:17 spos5gpp1DBG_MOND
  44. srwxrwxrwx 1 root system 0 May 12 20:13 spos5gpp1DBG_OHASD
  45. srwxrwxrwx 1 oracle oinstall 0 Sep 1 2015 sprocr_local_conn_0_PROC
  46. -rw-r--r-- 1 oracle oinstall 0 Jun 10 2015 sprocr_local_conn_0_PROC_lock
  47. srwxrwxrwx 1 root system 0 May 12 20:13 sprocr_local_conn_0_PROL
  48. -rw-r--r-- 1 oracle oinstall 0 Jun 10 2015 sprocr_local_conn_0_PROL_lock

我们已经找到原因，下面有两种方法解决。
第一，批量修改权限
第二，删除scoket文件, crs重启后会重建。

第一种方法：批量修改
  1. chown grid:oinstall mdnsd
  2. chown grid:oinstall mdnsd.pid
  3. chown root:system npohasd
  4. chown grid:oinstall ora_gipc_GPNPD_pos5gpp1
  5. chown grid:oinstall ora_gipc_GPNPD_pos5gpp1_lock
  6. chown grid:oinstall ora_gipc_gipcd_pos5gpp1
  7. chown grid:oinstall ora_gipc_gipcd_pos5gpp1_lock
  8. chown root:system ora_gipc_spos5gpp1gridpos5gpp-clusterCRFM_CLIIPC
  9. chown root:system ora_gipc_spos5gpp1gridpos5gpp-clusterCRFM_CLIIPC_lock
  10. chown root:system ora_gipc_spos5gpp1gridpos5gpp-clusterCRFM_SIPC
  11. chown root:system ora_gipc_spos5gpp1gridpos5gpp-clusterCRFM_SIPC_lock
  12. chown oracle:oinstall s#14287242.1
  13. chown grid:oinstall s#48955460.1
  14. chown grid:oinstall s#5046366.1
  15. chown grid:oinstall sAevm
  16. chown root:system sCRSD_IPC_SOCKET_11
  17. chown root:system sCRSD_IPC_SOCKET_11_lock
  18. chown root:system sCRSD_UI_SOCKET
  19. chown grid:oinstall sCevm
  20. chown grid:oinstall sLISTENER
  21. chown grid:oinstall sLISTENER_SCAN1
  22. chown grid:oinstall sOCSSD_LL_pos5gpp1_
  23. chown grid:oinstall sOCSSD_LL_pos5gpp1__lock
  24. chown grid:oinstall sOCSSD_LL_pos5gpp1_pos5gpp-cluster
  25. chown grid:oinstall sOCSSD_LL_pos5gpp1_pos5gpp-cluster_lock
  26. chown root:system sOHASD_IPC_SOCKET_11
  27. chown root:system sOHASD_IPC_SOCKET_11_lock
  28. chown root:system sOHASD_UI_SOCKET
  29. chown grid:oinstall sOracle_CSS_LclLstnr_pos5gpp-cluster_2
  30. chown grid:oinstall sOracle_CSS_LclLstnr_pos5gpp-cluster_2_lock
  31. chown grid:oinstall sSYSTEM.evm.acceptor.auth
  32. chown root:system sora_crsqs
  33. chown root:system spos5gpp1DBG_CRSD
  34. chown grid:oinstall spos5gpp1DBG_CSSD
  35. chown root:system spos5gpp1DBG_CTSSD
  36. chown grid:oinstall spos5gpp1DBG_EVMD
  37. chown grid:oinstall spos5gpp1DBG_GIPCD
  38. chown grid:oinstall spos5gpp1DBG_GPNPD
  39. chown root:system spos5gpp1DBG_LOGD
  40. chown grid:oinstall spos5gpp1DBG_MDNSD
  41. chown root:system spos5gpp1DBG_MOND
  42. chown root:system spos5gpp1DBG_OHASD
  43. chown root:system sprocr_local_conn_0_PROC
  44. chown root:system sprocr_local_conn_0_PROC_lock
  45. chown root:system sprocr_local_conn_0_PROL
  46. chown root:system sprocr_local_conn_0_PROL_lock

第二种方法，使用root 用户停掉 GI，删除这些 socket 文件，并重新启动 GI。

安装上面解决方法处理后，CRS恢复正常。
  1. grid@aadgpp1:/home/grid/>crsctl check crs
  2. CRS-4638: Oracle High Availability Services is online
  3. CRS-4537: Cluster Ready Services is online
  4. CRS-4529: Cluster Synchronization Services is online
  5. CRS-4533: Event Manager is online
  6. grid@aadgpp1:/home/grid/>ps -ef|grep d.bin
  7.     grid 4063236 1 0 21:37:23 - 0:00 /grid/app/11.2/bin/mdnsd.bin
  8.     root 5308482 1 0 21:38:05 - 0:01 /grid/app/11.2/bin/crsd.bin reboot
  9.     root 6029390 1 0 21:37:26 - 0:01 /grid/app/11.2/bin/osysmond.bin
  10.     root 6291706 1 0 21:37:07 - 0:01 /grid/app/11.2/bin/ohasd.bin reboot
  11.     grid 6422532 6815890 0 21:37:28 - 0:01 /grid/app/11.2/bin/ocssd.bin
  12.     grid 3342780 1 1 21:38:05 - 0:00 /grid/app/11.2/bin/evmd.bin
  13.     grid 5046692 1 0 21:37:24 - 0:00 /grid/app/11.2/bin/gpnpd.bin
  14.     grid 6095300 1 0 21:37:26 - 0:00 /grid/app/11.2/bin/gipcd.bin
  15.     root 6488494 1 0 21:37:56 - 0:00 /grid/app/11.2/bin/octssd.bin reboot


需要注意：

  1. Caution:
  2. After installation is complete, do not remove manually or run cron jobs that remove /tmp/.oracle or /var/tmp/.oracle directories or their files while Oracle software is running on the server. If you remove these files, then the Oracle software can encounter intermittent hangs. Oracle Clusterware installations can fail with the error:
  3. CRS-0184: Cannot communicate with the CRS daemon.
安装完成后,在正常运行的RAC下，不能手工移动或者删除/tmp/.oracle,/var/tmp/.oracle，如果移除了，会造成间断性的hang,oracle集群软件也会有报错。

记住：
如果RAC正在运行千万不能删除！
如果文件删除了，重新CRS会自动重新建立！
如果目录删除了，那就只能参考下面的命令重建了
Create the /var/tmp and /var/tmp/.oracle directory:
/bin/mkdir -p /var/tmp/.oracle
/bin/chmod 01777 /var/tmp/
/bin/chown root /var/tmp/ 
/bin/chmod 01777 /var/tmp/.oracle
/bin/chown root /var/tmp/.oracle


另外补充说明(隐藏SOCKET文件)：
  1. The hidden directory '/var/tmp/.oracle' (or /tmp/.oracle on some platforms) or its content was removed while instances & the CRS stack were up and running. Typically this directory contains a number of "special" socket files that are used by local clients to connect via the IPC protocol (sqlnet) to various Oracle processes including the TNS listener, the CSS, CRS & EVM daemons or even database or ASM instances. These files are created when the "listening" process starts.
这些socket 文被用作 本地客户端使用进程间通信协议（ipc）和不同的oracle的进程通信，而这些进程包括：tns 监听，css ，crs，evm 守护进程；甚至数据库和asm 实例。这些socket 由‘主动监听’的进程创建。在这里oracle tns listener 创建这些socket 文件主要使用用作pmon 和 tnslsnr  通信从报错信息里就可以看出问题.

来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/17086096/viewspace-2098909/ ，如需转载，请注明出处，否则将追究法律责任。

0
0
上一篇： 利用UDEV管理配置ASMDISK
下一篇： ORACLE 数字类型（number）的值长时，科学计数显示问题
请登录后发表评论 登录
全部评论
startay

注册时间：2013-07-29

  • 博文量
    45
  • 访问量
    117886

最新文章

支持我们 作者招募 用户协议 FAQ Contact Us

北京盛拓优讯信息技术有限公司. 版权所有  京ICP备16024965号-8 北京市公安局海淀分局网监中心备案编号：11010802021510 niuxiaotong@pcpop.com 17352615567 未成年人举报专区

广播电视节目制作经营许可证(京) 字第1234号 中国互联网协会会员

违法和不良信息举报电话:18101376593