[故障描述]
今天进行节点2的asm diskgroup 检查时发现，
asmcmd lsdg 命令没有结果输出，然后去检查节点2的相关日志,
数据库日志无异常报错，cluster 日志也无异常报错，但ASM日志报以下的错误：
2011.9.20 11:26:01 ASM报以下的错误

Sep 20 11:26:01 2011
WARNNING: cache read a corrupted block group=1(CRS) fn=1 blk=7 from disk 0(CRS_0000)
Errors in file /software/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_46793154.trc:
ORA-15196: invalid ASM block header [kfc.c:25165] [obj_kfbl] [1] [7] [2147483649 != 1]
NOTE: a corrupted block from group CRS was dumped to /software/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_46793154.trc
WARNNING: cache read(retry) a corrupted block group=1(CRS) fn=1 blk=7 from disk 0(CRS_0000)
Errors in file /software/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_46793154.trc:
ORA-15196: invalid ASM block header [kfc.c:25165] [obj_kfbl] [1] [7] [2147483649 != 1]
ORA-15196: invalid ASM block header [kfc.c:25165] [obj_kfbl] [1] [7] [2147483649 != 1]
ERROR: cache failed to read group=1(CRS) fn=1 blk=7 from disk(s): 0(CRS_0000)
ORA-15196: invalid ASM block header [kfc.c:25165] [obj_kfbl] [1] [7] [2147483649 != 1]
ORA-15196: invalid ASM block header [kfc.c:25165] [obj_kfbl] [1] [7] [2147483649 != 1]
System State dumped to trace file /software/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_46793154.trc
NOTE: AMDU dump of disk group CRS created at /software/oracle/diag/asm/+asm/+ASM2/trace
NOTE: cache initiating offline of disk 0 group CRS
NOTE: process 46793154 initiating offline of disk 0.2209752627 (CRS_0000) with mask 0x7e in group 1

上述日志报CRS diskgroup里有一个逻辑block出现损坏。导致diskgroup CRS offline.

OCR diskgoup 是RAC的一个重要组成部分，其存放着ocr file 和vote disk file，用于维护在cluster 中高可用性组件的信息。例如，
Cluster节点列表等。

当前只有节点2出现上述的错误，在节点2上，crs diskgroup offline,其他diskgroup 正常，所以节点2实例还是正常，可以提供访问。
而主节点1并没有出现相应的错误，数据库及应用连接正常。

[解决方案]
1、检查各个diskgroup的状态是否正常，节点1 全正常，节点2 CRS diskgroup 不正常
SQL> select name,state from v$asm_diskgroup;

NAME                           STATE
------------------------------ -----------
CRS                            MOUNTED
DGSYSTEM                       MOUNTED
DG01                      MOUNTED
RAID5_DG01                MOUNTED
RECOVER_DG                MOUNTED
SSD_DG01                  MOUNTED

6 rows selected.

SQL> alter diskgroup SSD_DG01 check;

Diskgroup altered.

SQL> alter diskgroup RECOVER_DG check;

Diskgroup altered.

SQL> alter diskgroup RAID5_DG01 check;

Diskgroup altered.

SQL> alter diskgroup DG01 check;

Diskgroup altered.

SQL> alter diskgroup DGSYSTEM check;

Diskgroup altered.

SQL> alter diskgroup CRS check;

Diskgroup altered.

节点2 CRS diskgroup报以下的错：
SQL> alter diskgroup CRS check;
alter diskgroup CRS check
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15130: diskgroup "CRS" is being dismounted

2、备份节点1 CRS diskgroup 的元信息
[nhdb18:grid]asmcmd md_backup crsdgbk -G CRS
Disk group metadata to be backed up: CRS
Current alias directory path: cluster
Current alias directory path: cluster/ASMPARAMETERFILE
Current alias directory path: cluster/OCRFILE

3、检查节点1是否有应用在跑，如有，通知应用组停止应用。
4、停止Goldengate

5、停止节点2 实例
6、恢复CRS diskgroup 元数据
   asmcmd md_restore crsdgbk -G CRS

7、把crs diskgroup 加载，并检查是否正常
  alter diskgroup crs mount;
  alter diskgroup CRS check;

8、把节点2实例启动，通知应用组启动相关的应用及Goldengate。

整个处理过程大概需要一个小时左右。

[后续处理]
可以在线处理：
1、CRS diskgroup 增加mirror disk,以防crs disk出现物理损坏。
2、定期备份ASM 磁盘组元数据。
3、加强ASM 和 CRS 日志的监控检查。