一、故障现象

2012年2月25日13:15:51，接到电话说某库2个节点出现宕机，13：36赶到某地,最后14：00到达6楼，之后查看数据库的状况,2个节点的实例都正常,据说是异常宕库的，到底是不是呢？来分析下！

二、处理过程

查看数据库的相关报警日志和审计日志.

数据库alert_kf2.log日志:

Sat Feb 25 01:17:38 2012

Shutting down instance: further logons disabled

Shutting down instance (immediate)

Sat Feb 25 01:17:59 2012

ALTER DATABASE DISMOUNT

Completed: ALTER DATABASE DISMOUNT

Sat Feb 25 01:45:14 2012

Starting ORACLE instance (force)

Sat Feb 25 01:47:57 2012

Database Characterset is ZHS16GBK

replication_dependency_tracking turned off (no async multimaster replication found)

Completed: ALTER DATABASE OPEN

数据库alert_kf1.log日志:

Sat Feb 25 01:22:07 2012

Shutting down instance: further logons disabled

Shutting down instance (immediate)

Sat Feb 25 01:22:22 2012

ALTER DATABASE DISMOUNT

Completed: ALTER DATABASE DISMOUNT

Sat Feb 25 01:45:14 2012

Starting ORACLE instance (force)

Sat Feb 25 01:45:14 2012

Global Enqueue Service Resources = 33288, pool = 16

Sat Feb 25 01:48:03 2012

Database Characterset is ZHS16GBK

replication_dependency_tracking turned off (no async multimaster replication found)

Completed: ALTER DATABASE OPEN

在节点kf1上的审计日志:

$ cat ora_1753524.aud

Sat Feb 25 01:22:07 2012

ACTION : 'CONNECT'

DATABASE USER: '/'

PRIVILEGE : SYSDBA

CLIENT USER: oracle

CLIENT TERMINAL:

STATUS: 0

Sat Feb 25 01:22:22 2012

ACTION : 'SHUTDOWN'

DATABASE USER: '/'

PRIVILEGE : SYSDBA

CLIENT USER: oracle

CLIENT TERMINAL:

STATUS: 0

$ cat ora_1744902.aud

Instance name: kf1

Redo thread mounted by this instance: 0

Oracle process number: 0

1744902

Sat Feb 25 01:45:14 2012

ACTION : 'CONNECT'

DATABASE USER: '/'

PRIVILEGE : SYSDBA

CLIENT USER: oracle

CLIENT TERMINAL: Not Available

STATUS: 0

Sat Feb 25 01:45:14 2012

ACTION : 'STARTUP'

DATABASE USER: '/'

PRIVILEGE : SYSDBA

CLIENT USER: oracle

CLIENT TERMINAL: Not Available

STATUS: 0

在节点kf2上的审计日志:

$ more ora_2740690.aud

Instance name: kf2

Sat Feb 25 01:17:38 2012

ACTION : 'CONNECT'

DATABASE USER: '/'

PRIVILEGE : SYSDBA

CLIENT USER: oracle

CLIENT TERMINAL:

STATUS: 0

Sat Feb 25 01:17:59 2012

ACTION : 'SHUTDOWN'

DATABASE USER: '/'

PRIVILEGE : SYSDBA

CLIENT USER: oracle

CLIENT TERMINAL:

STATUS: 0

$ more ora_2601180.aud

Instance name: kf2

Redo thread mounted by this instance: 0

Oracle process number: 0

Sat Feb 25 01:45:14 2012

ACTION : 'STARTUP'

DATABASE USER: '/'

PRIVILEGE : SYSDBA

CLIENT USER: oracle

CLIENT TERMINAL: Not Available

STATUS: 0

1、 从两个节点的后台alert日志及audit 日志可以看出，在2012-02-25 01:17:38有人停了kf2实例, 01:22:07停了kf1实例, 01:45:14启动了2个实例.
三、处理结果
在事实面前，不得不承认是他Shutdown的,哎。。。。,何必了，何苦呢！