#背景：

AIX Oracle11.2.0.4 RAC

节点2实例异常重启。

#数据库alert日志：

Sun Jan 02 23:08:13 2022

LMON (ospid: 4981218) waits for event 'rdbms ipc message' for 90 secs.

Sun Jan 02 23:08:28 2022

Errors in file /u01/app/oracle/diag/rdbms/bjsbjl/bjsbjl2/trace/bjsbjl2_lmhb_3736326.trc  (incident=336164):

ORA-29770: global enqueue process LMON (OSID 4981218) is hung for more than 70 seconds

Incident details in: /u01/app/oracle/diag/rdbms/bjsbjl/bjsbjl2/incident/incdir_336164/bjsbjl2_lmhb_3736326_i336164.trc

Sun Jan 02 23:08:36 2022

Sweep [inc][336164]: completed

Sweep [inc2][336164]: completed

Sun Jan 02 23:08:40 2022

ERROR: Some process(s) is not making progress.

LMHB (ospid: 3736326) is terminating the instance.

Please check LMHB trace file for more details.

Please also check the CPU load, I/O load and other system properties for anomalous behavior

ERROR: Some process(s) is not making progress.

LMHB (ospid: 3736326): terminating the instance due to error 29770

Sun Jan 02 23:08:40 2022

System state dump requested by (instance=2, osid=3736326 (LMHB)), summary=[abnormal instance termination].

System State dumped to trace file /u01/app/oracle/diag/rdbms/bjsbjl/bjsbjl2/trace/bjsbjl2_diag_6619226_20220102230840.trc

Instance terminated by LMHB, pid = 3736326

#lmbh的trace文件：

*

略

*

Short stack dump: ORA-32516: cannot wait for process 'Unix process pid: 4981218, image: oracle@gshsqjdb02 (LMON)' to finish executing ORADEBUG command 'SHORT_STACK'; wait time exceeds 2495 ms

*

略

*

kjfmGCR_HBCheckAll: 1 process failed HBStatusCheck

kjfmGCR_HBAnalyze: LMON has no heartbeat

*** 2022-01-02 23:08:28.904

==============================

LMON (ospid: 4981218) has not moved for 113 sec (1641136108.1641135995)

Incident 336164 created, dump file: /u01/app/oracle/diag/rdbms/bjsbjl/bjsbjl2/incident/incdir_336164/bjsbjl2_lmhb_3736326_i336164.trc

ORA-29770: global enqueue process LMON (OSID 4981218) is hung for more than 70 seconds

ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 4981218, image: oracle@gshsqjdb02 (LMON)'; prior command execution time exceeds 2486 ms

*

略

*

#MOS上有篇文档：

OERR: ORA-29770 "global enqueue process %s (OSID %s) is hung for more than %s seconds" Reference Note (Doc ID 2010718.1)

其中找到29770关于LMON的一个bug：

Bug Fixed Description

27254851 18.3, 19.1 INSTANCE CRASHED WITH ORA-29770 DUE TO LMON HAS NOT HEARTBEAT

该问题在18，19版本才修复。