故障经过:
          2012-03-20,17:00 下班后,客户电话通知,应用出现异常,报如下错误,应用不能插入数据.
           java.io.IOException: ORA-01157: cannot identify/lock data file 336 - see DBWR trace file
          ORA-01110: data file 336: '/dev/rdb04_0036'
处理经过:
          刚到家,穿拖鞋准备吃饭,没办法,来事了,赶紧打开电脑,连上vpn看下吧,先看下aler日志
    节点1:              
    Tue Mar 20 16:52:11 2012
    Thread 1 advanced to log sequence 35890 (LGWR switch)
     Current log# 3 seq# 35890 mem# 0: /dev/redo_1_3
    Tue Mar 20 16:58:52 2012
    alter tablespace TBS_IOM_PART03 add datafile '/dev/rdb04_0036'  size 8191m
    Tue Mar 20 16:59:24 2012
    Completed: alter tablespace TBS_IOM_PART03 add datafile '/dev/rdb04_0036'  size 8191m
    Tue Mar 20 17:05:34 2012
    Thread 1 advanced to log sequence 35891 (LGWR switch)
     Current log# 2 seq# 35891 mem# 0: /dev/redo_1_2
    节点2:
    Tue Mar 20 16:50:52 2012
    Thread 2 advanced to log sequence 21668 (LGWR switch)
    Current log# 5 seq# 21668 mem# 0: /dev/redo_2_2
   Tue Mar 20 16:59:24 2012
   Errors in file /oracle/admin/fwkt/bdump/fwkt2_dbw0_925932.trc:
    ORA-01157: cannot identify/lock data file 336 - see DBWR trace file
    ORA-01110: data file 336: '/dev/rdb04_0036'
    ORA-27041: unable to open file
   IBM AIX RISC System/6000 Error: 13: Permission denied
   Additional information: 11
   Tue Mar 20 16:59:24 2012
   Errors in file /oracle/admin/fwkt/bdump/fwkt2_dbw0_925932.trc:
   ORA-01186: file 336 failed verification tests
   ORA-01157: cannot identify/lock data file 336 - see DBWR trace file
   ORA-01110: data file 336: '/dev/rdb04_0036'
   Tue Mar 20 16:59:24 2012
   File 336 not verified due to error ORA-01157
   Tue Mar 20 16:59:24 2012
   Errors in file /oracle/admin/fwkt/bdump/fwkt2_dbw0_925932.trc:
   ORA-01157: cannot identify/lock data file 336 - see DBWR trace file
   ORA-01110: data file 336: '/dev/rdb04_0036'
   ORA-27041: unable to open file
   IBM AIX RISC System/6000 Error: 13: Permission denied
   Additional information: 11
   Tue Mar 20 16:59:24 2012
   Errors in file /oracle/admin/fwkt/bdump/fwkt2_dbw0_925932.trc:
   ORA-01186: file 336 failed verification tests
   ORA-01157: cannot identify/lock data file 336 - see DBWR trace file
   ORA-01110: data file 336: '/dev/rdb04_0036'
   File 336 not verified due to error ORA-01157
   Tue Mar 20 17:13:26 2012
   Thread 2 advanced to log sequence 21669 (LGWR switch)
   Current log# 6 seq# 21669 mem# 0: /dev/redo_2_3
   Tue Mar 20 17:26:28 2012
结论: 
    应该是有人在节点1上增加了个数据文件,'/dev/rdb04_0036' ,但是这个Raw Device在节点2上权限应该是不对的,
    去节 点2上看下
    # ls -tlr /dev/rdb04_0036
     crw-rw----   1 root     system       48, 36 Mar 20 16:41 /dev/rdb04_0036
   我晕, 果然没有修改权限,赶紧修改吧,chown oracle:dba  /dev/rdb04_0036
    修改之后,不要以为就完事了,这个时候,在节点2上当执行如下select count(1) from dba_data_files 还是报同样的
    错误.
    SQL> select count(1) from dba_data_files;
    select count(1) from dba_data_files
     ERROR at line 1:
    ORA-01157: cannot identify/lock data file 336 - see DBWR trace file
    ORA-01110: data file 336: '/dev/rdb04_0036'
     修改完权限之后,还要加一步,'alter system check datafiles',这才算完事.继续吃饭了,晕,饭都凉了.