今天青海同事说应用不能录入数据：
查看alert日志出现大量的下面信息
ORA-00603: ORACLE server session terminated by fatal error
ORA-00372: file 3 cannot be modified at this time
ORA-01110: data file 3: '+FLASHRECOVER/orcl/datafile/sysaux.266.764271775'
ORA-00372: file 3 cannot be modified at this time
ORA-01110: data file 3: '+FLASHRECOVER/orcl/datafile/sysaux.266.764271775'
Mon Feb 20 15:15:40 2012
Trace dumping is performing id=[cdmp_20120220151540]
Mon Feb 20 15:15:42 2012
Errors in file /opt/oracle/admin/orcl/bdump/orcl1_pmon_10355.trc:
ORA-00376: Message 376 not found; No message file for product=RDBMS, facility=ORA; arguments: [3]
ORA-01110: Message 1110 not found; No message file for product=RDBMS, facility=ORA; arguments: [3] [+FLASHRECOVER/orcl/datafile/sysaux.266.764271775]
Mon Feb 20 15:15:42 2012
Errors in file /opt/oracle/admin/orcl/bdump/orcl1_pmon_10355.trc:
ORA-00376: Message 376 not found; No message file for product=RDBMS, facility=ORA; arguments: [3]
ORA-01110: Message 1110 not found; No message file for product=RDBMS, facility=ORA; arguments: [3] [+FLASHRECOVER/orcl/datafile/sysaux.266.764271775]
Mon Feb 20 15:15:42 2012
Trace dumping is performing id=[cdmp_20120220151542]
Mon Feb 20 15:15:47 2012
Errors in file /opt/oracle/admin/orcl/bdump/orcl1_q001_10774.trc:
ORA-00376: file 3 cannot be read at this time
ORA-01110: data file 3: '+FLASHRECOVER/orcl/datafile/sysaux.266.764271775'

猜测是+FLASHRECOVER/orcl/datafile/sysaux.266.764271775文件损坏。

查看crs等状态，有些服务已经停止。
rac1:/opt/oracle/product/10.2/crs/bin # ./crs_stat -t
名称           类型           目标      状态      主机       
------------------------------------------------------------
ora.orcl.db    application    ONLINE    ONLINE    rac1       
ora....l1.inst application    ONLINE    ONLINE    rac1       
ora....l2.inst application    ONLINE    ONLINE    rac2          
ora....SM1.asm application    ONLINE    ONLINE    rac1       
ora....C1.lsnr application    ONLINE    OFFLINE   rac1       
ora.rac1.gsd   application    ONLINE    ONLINE    rac1       
ora.rac1.ons   application    ONLINE    ONLINE    rac1       
ora.rac1.vip   application    ONLINE    ONLINE    rac1       
ora....SM2.asm application    ONLINE    ONLINE    rac2         
ora....C2.lsnr application    ONLINE    OFFLINE   rac2            
ora.rac2.gsd   application    ONLINE    ONLINE    rac2          
ora.rac2.ons   application    ONLINE    OFFLINE   rac2          
ora.rac2.vip   application    ONLINE    ONLINE    rac1       
检查数据文件状态。3号文件确实是损坏，查看日志是昨天下午三点多出现问题。尝试恢复.
SQL> select file#,status from v$datafile;

     FILE# STATUS
---------- -------
         1 SYSTEM
         2 ONLINE
         3 RECOVER
         4 ONLINE
         5 ONLINE
         6 ONLINE
         7 ONLINE
         8 ONLINE
         9 ONLINE
        10 ONLINE
        11 ONLINE
        12 ONLINE
        13 ONLINE

SQL> recover datafile 3;
ORA-00279: change 24192605 generated at 02/20/2012 08:01:27 needed for thread 1
ORA-00289: suggestion : +FLASHRECOVER
ORA-00280: change 24192605 for thread 1 is in sequence #800
Specify log: {=suggested | filename | AUTO | CANCEL}
auto
ORA-00308: cannot open archived log '+FLASHRECOVER'
ORA-17503: ksfdopn:2 Failed to open file +FLASHRECOVER
ORA-15045: ASM file name '+FLASHRECOVER' is not in reference form
ORA-15173: entry 'ARCHIVELOG' does not exist in directory 'ORCL'
ORA-00308: cannot open archived log '+FLASHRECOVER'
ORA-17503: ksfdopn:2 Failed to open file +FLASHRECOVER
ORA-15045: ASM file name '+FLASHRECOVER' is not in reference form
不能打开asm。。。特么悲剧呢，难道asm出问题了。。查看asm日志并有错误提示，
ora.orcl.db    application    ONLINE    ONLINE    rac1       
ora....l1.inst application    ONLINE    ONLINE    rac1       
ora....l2.inst application    ONLINE    ONLINE    rac2          
ora....SM1.asm application    ONLINE    ONLINE    rac1       
ora....C1.lsnr application    ONLINE    OFFLINE          
ora.rac1.gsd   application    ONLINE    ONLINE    rac1       
ora.rac1.ons   application    ONLINE    ONLINE    rac1       
ora.rac1.vip   application    ONLINE    ONLINE    rac1       
ora....SM2.asm application    ONLINE    ONLINE             
ora....C2.lsnr application    ONLINE    OFFLINE               
ora.rac2.gsd   application    ONLINE    ONLINE    rac2          
ora.rac2.ons   application    ONLINE    OFFLINE             
ora.rac2.vip   application    ONLINE    ONLINE    rac1       

rac1:/opt/oracle/product/10.2/crs/bin # ps -ef | grep ASM
root      7127 23948  0 14:51 pts/0    00:00:00 grep ASM
oracle   10071     1  0  2011 ?        00:00:00 asm_pmon_+ASM1
oracle   10073     1  0  2011 ?        00:00:01 asm_diag_+ASM1
oracle   10075     1  0  2011 ?        00:00:00 asm_psp0_+ASM1
oracle   10077     1  0  2011 ?        00:00:03 asm_lmon_+ASM1
oracle   10079     1  0  2011 ?        00:00:15 asm_lmd0_+ASM1
oracle   10081     1  0  2011 ?        00:00:08 asm_lms0_+ASM1
oracle   10085     1  0  2011 ?        00:00:00 asm_mman_+ASM1
oracle   10087     1  0  2011 ?        00:00:01 asm_dbw0_+ASM1
oracle   10089     1  0  2011 ?        00:00:02 asm_lgwr_+ASM1
oracle   10091     1  0  2011 ?        00:00:01 asm_ckpt_+ASM1
oracle   10093     1  0  2011 ?        00:00:00 asm_smon_+ASM1
oracle   10095     1  0  2011 ?        00:00:00 asm_rbal_+ASM1
oracle   10097     1  0  2011 ?        00:00:00 asm_gmon_+ASM1
oracle   10170     1  0  2011 ?        00:00:25 asm_lck0_+ASM1

查看备份脚本，原来是每隔8个小时就备份一次归档日志，然后清理掉原有日志。进入rman 重新尝试回复。
oracle@rac1:~> crontab -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.XXXXbc394M installed on Thu Oct 20 11:57:20 2011)
# (Cron version V5.0 -- $Id: crontab.c,v 1.12 2004/01/23 18:56:42 vixie Exp $)
0 0 * * 0 sh /opt/oracle/backup/script/dbback.sh 2>&1 >/dev/null
0 0,8,16 * * * sh /opt/oracle/backup/script/backuparchive.sh 2>&1 >/dev/null

oracle@rac1:~> /opt/oracle/product/10.2/db_1/bin/rman target /

Recovery Manager: Release 10.2.0.4.0 - Production on 星期二 2月 21 13:59:20 2012

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

connected to target database: ORCL (DBID=1292052561, not open)

RMAN> recover datafile 3;

Starting recover at 2012-02-21 13:59:25
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=1643 instance=orcl1 devtype=DISK

starting media recovery

channel ORA_DISK_1: starting archive log restore to default destination
channel ORA_DISK_1: restoring archive log
archive log thread=2 sequence=798
channel ORA_DISK_1: reading from backup piece /opt/oracle/backup/arcback/arc_775728097_1681
channel ORA_DISK_1: restored backup piece 1
piece handle=/opt/oracle/backup/arcback/arc_775728097_1681 tag=TAG20120220T080137
channel ORA_DISK_1: restore complete, elapsed time: 00:00:02
channel ORA_DISK_1: starting archive log restore to default destination
channel ORA_DISK_1: restoring archive log
archive log thread=2 sequence=800
channel ORA_DISK_1: reading from backup piece /opt/oracle/backup/arcback/arc_775756834_1685
channel ORA_DISK_1: restored backup piece 1
piece handle=/opt/oracle/backup/arcback/arc_775756834_1685 tag=TAG20120220T160034
channel ORA_DISK_1: restore complete, elapsed time: 00:00:02
channel ORA_DISK_1: starting archive log restore to default destination
channel ORA_DISK_1: restoring archive log
archive log thread=2 sequence=799
channel ORA_DISK_1: restoring archive log
archive log thread=1 sequence=801
channel ORA_DISK_1: reading from backup piece /opt/oracle/backup/arcback/arc_775756834_1684
channel ORA_DISK_1: restored backup piece 1
piece handle=/opt/oracle/backup/arcback/arc_775756834_1684 tag=TAG20120220T160034
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting archive log restore to default destination
channel ORA_DISK_1: restoring archive log
archive log thread=1 sequence=800
channel ORA_DISK_1: reading from backup piece /opt/oracle/backup/arcback/arc_775756834_1683
channel ORA_DISK_1: restored backup piece 1
piece handle=/opt/oracle/backup/arcback/arc_775756834_1683 tag=TAG20120220T160034
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
archive log filename=+FLASHRECOVER/orcl/archivelog/2012_02_21/thread_1_seq_800.362.775835975 thread=1 sequence=800
archive log filename=+FLASHRECOVER/orcl/archivelog/2012_02_21/thread_2_seq_798.366.775835969 thread=2 sequence=798
archive log filename=+FLASHRECOVER/orcl/archivelog/2012_02_21/thread_2_seq_799.359.775835973 thread=2 sequence=799
media recovery complete, elapsed time: 00:00:03
channel ORA_DISK_1: starting archive log restore to default destination
channel ORA_DISK_1: restoring archive log
archive log thread=2 sequence=802
channel ORA_DISK_1: reading from backup piece /opt/oracle/backup/arcback/arc_775785626_1689
channel ORA_DISK_1: restored backup piece 1
piece handle=/opt/oracle/backup/arcback/arc_775785626_1689 tag=TAG20120221T000026
channel ORA_DISK_1: restore complete, elapsed time: 00:00:02
channel ORA_DISK_1: starting archive log restore to default destination
channel ORA_DISK_1: restoring archive log
archive log thread=2 sequence=801
channel ORA_DISK_1: restoring archive log
archive log thread=1 sequence=803
channel ORA_DISK_1: reading from backup piece /opt/oracle/backup/arcback/arc_775785626_1688
channel ORA_DISK_1: restored backup piece 1
piece handle=/opt/oracle/backup/arcback/arc_775785626_1688 tag=TAG20120221T000026
channel ORA_DISK_1: restore complete, elapsed time: 00:00:02
channel ORA_DISK_1: starting archive log restore to default destination
channel ORA_DISK_1: restoring archive log
archive log thread=1 sequence=802
channel ORA_DISK_1: reading from backup piece /opt/oracle/backup/arcback/arc_775785626_1687
channel ORA_DISK_1: restored backup piece 1
piece handle=/opt/oracle/backup/arcback/arc_775785626_1687 tag=TAG20120221T000026
channel ORA_DISK_1: restore complete, elapsed time: 00:00:02
channel ORA_DISK_1: starting archive log restore to default destination
channel ORA_DISK_1: restoring archive log
archive log thread=1 sequence=804
channel ORA_DISK_1: reading from backup piece /opt/oracle/backup/arcback/arc_775814425_1691
channel ORA_DISK_1: restored backup piece 1
piece handle=/opt/oracle/backup/arcback/arc_775814425_1691 tag=TAG20120221T080025
channel ORA_DISK_1: restore complete, elapsed time: 00:00:02
channel ORA_DISK_1: starting archive log restore to default destination
channel ORA_DISK_1: restoring archive log
archive log thread=2 sequence=803
channel ORA_DISK_1: restoring archive log
archive log thread=1 sequence=805
channel ORA_DISK_1: reading from backup piece /opt/oracle/backup/arcback/arc_775814425_1692
channel ORA_DISK_1: restored backup piece 1
piece handle=/opt/oracle/backup/arcback/arc_775814425_1692 tag=TAG20120221T080025
channel ORA_DISK_1: restore complete, elapsed time: 00:00:02
channel ORA_DISK_1: starting archive log restore to default destination
channel ORA_DISK_1: restoring archive log
archive log thread=2 sequence=804
channel ORA_DISK_1: reading from backup piece /opt/oracle/backup/arcback/arc_775814425_1693
channel ORA_DISK_1: restored backup piece 1
piece handle=/opt/oracle/backup/arcback/arc_775814425_1693 tag=TAG20120221T080025
channel ORA_DISK_1: restore complete, elapsed time: 00:00:02
Finished recover at 2012-02-21 13:59:50

RMAN> exit

Recovery Manager complete.

SQL> alter database datafile '+FLASHRECOVER/orcl/datafile/sysaux.266.764271775' online;

Database altered.

SQL> select file#,status from v$datafile;

     FILE# STATUS
---------- -------
         1 SYSTEM
         2 ONLINE
         3 ONLINE
         4 ONLINE
         5 ONLINE
         6 ONLINE
         7 ONLINE
         8 ONLINE
         9 ONLINE
        10 ONLINE
        11 ONLINE

     FILE# STATUS
---------- -------
        12 ONLINE
        13 ONLINE

13 rows selected.

SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options