AsM丢魂记？（二）
昨天的问题，貌似已经结束。今天朋友又打来电话，数据库又宕机了。难道ASM的磁盘不够，整个系统在用户写入数据库的时候，磁盘满了，无法写入，造成数据库崩溃？
1、问题继续分析：
经过我不懈的努力，将节点恢复到目前状态：
[oracle@rac2 bin]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.orcl.db application ONLINE OFFLINE
ora....l1.inst application ONLINE OFFLINE
ora....l2.inst application ONLINE OFFLINE
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
数据库的实例始终是启动不了。我一直在纳闷，今天见鬼了。。。。。
尝试手工启动数据库：
[oracle@rac2 bin]$ sqlplus /nolog
SQL*Plus: Release 10.2.0.2.0 - Production on Thu Mar 5 01:06:14 2009
Copyright (c) 1982, 2005, Oracle. All Rights Reserved.
SQL> conn sys/oracle as sysdba
Connected to an idle instance.
SQL> startup
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/orcl/spfileorcl.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/orcl/spfileorcl.ora
ORA-15077: could not locate ASM instance serving a required diskgroup
SQL>
SQL> exit
Disconnected
没有磁盘？为什么？
[oracle@rac2 bin]$ sqlplus /nolog
SQL*Plus: Release 10.2.0.2.0 - Production on Thu Mar 5 01:06:14 2009
Copyright (c) 1982, 2005, Oracle. All Rights Reserved.
SQL> conn sys/oracle as sysdba
Connected to an idle instance.
SQL> startup
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/orcl/spfileorcl.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/orcl/spfileorcl.ora
ORA-15077: could not locate ASM instance serving a required diskgroup
SQL>
SQL> exit
Disconnected
[oracle@rac2 bin]$ crs_start ora.orcl.orcl2.inst
Attempting to start `ora.orcl.orcl2.inst` on member `rac2`
Start of `ora.orcl.orcl2.inst` on member `rac2` failed.
rac1 : CRS-1018: Resource ora.rac2.vip (application) is already running on rac2
CRS-0215: Could not start resource 'ora.orcl.orcl2.inst'.
这样貌似也启动不了，报没有看到资源的错误。为什么？ASM已经启动了！
确认一下：
[oracle@rac2 bin]$ ps -ef|grep ASM
oracle 30320 1 0 00:06 ? 00:00:00 asm_pmon_+ASM2
oracle 30322 1 0 00:06 ? 00:00:00 asm_diag_+ASM2
oracle 30324 1 0 00:06 ? 00:00:00 asm_psp0_+ASM2
oracle 30326 1 0 00:06 ? 00:00:01 asm_lmon_+ASM2
oracle 30328 1 0 00:06 ? 00:00:00 asm_lmd0_+ASM2
oracle 30330 1 0 00:06 ? 00:00:00 asm_lms0_+ASM2
oracle 30334 1 0 00:06 ? 00:00:00 asm_mman_+ASM2
oracle 30336 1 0 00:06 ? 00:00:00 asm_dbw0_+ASM2
oracle 30348 1 0 00:06 ? 00:00:00 asm_lgwr_+ASM2
oracle 30365 1 0 00:06 ? 00:00:00 asm_ckpt_+ASM2
oracle 30367 1 0 00:06 ? 00:00:00 asm_smon_+ASM2
oracle 30369 1 0 00:06 ? 00:00:00 asm_rbal_+ASM2
oracle 30371 1 0 00:06 ? 00:00:00 asm_gmon_+ASM2
oracle 30423 1 0 00:06 ? 00:00:00 asm_lck0_+ASM2
oracle 30491 1 0 00:06 ? 00:00:00 /u01/app/oracle/product/10.2.0/asm/bin/racgimon daemon ora.rac2.ASM2.asm
oracle 25371 16831 0 01:08 pts/2 00:00:00 grep ASM
看来是没问题！ASM进程是启动的。
root@rac2 ~]# /etc/init.d/oracleasm listdisks
[root@rac2 ~]# /etc/init.d/oracleasm enable
Writing Oracle ASM library driver configuration: [ OK ]
Scanning system for ASM disks: [ OK ]
[root@rac2 ~]# /etc/init.d/oracleasm listdisks
扫描一下ASM是否启动，发现找不到ASM添加的磁盘。
基于操作系统层面，查看一下磁盘（其实这一步做不做都是多余）
[root@rac2 ~]# df -m
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/sda2 10077 884 8681 10% /
/dev/sda1 190 14 168 8% /boot
none 3986 0 3986 0% /dev/shm
/dev/sda7 4031 47 3779 2% /tmp
/dev/sda8 95178 30170 60174 34% /u01
/dev/sda6 10077 2151 7414 23% /usr
/dev/sda5 10077 159 9407 2% /var
172.16.1.160:/u01/rac1Arch
95186 22229 68122 25% /u01/rac1Arch
/dev/sdb1 1024 112 913 11% /u03
但是，我做到这个步骤，就明白可能存储有问题：
[root@rac2 ~]# fdisk -l
Disk /dev/sda: 146.6 GB, 146695782400 bytes
255 heads, 63 sectors/track, 17834 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 25 200781 83 Linux
/dev/sda2 26 1330 10482412+ 83 Linux
/dev/sda3 1331 2375 8393962+ 82 Linux swap
/dev/sda4 2376 17834 124174417+ 5 Extended
/dev/sda5 2376 3680 10482381 83 Linux
/dev/sda6 3681 4985 10482381 83 Linux
/dev/sda7 4986 5507 4192933+ 83 Linux
/dev/sda8 5508 17834 99016596 83 Linux
Disk /dev/sdb: 1073 MB, 1073741824 bytes
34 heads, 61 sectors/track, 1011 cylinders
Units = cylinders of 2074 * 512 = 1061888 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 1011 1048376+ 83 Linux
Disk /dev/sdd: 1073 MB, 1073741824 bytes
34 heads, 61 sectors/track, 1011 cylinders
Units = cylinders of 2074 * 512 = 1061888 bytes
Device Boot Start End Blocks Id System
/dev/sdd1 1 1011 1048376+ 83 Linux
根本没发现ASM的卷组！看来该卷组根本没有挂载上来！
[root@rac2 ~]# ls -lat /dev/raw/raw*
ls: /dev/raw/raw*: No such file or directory
没有这个目录！
[root@rac2 dev]# su - oracle
[oracle@rac2 ~]$ srvctl config asm -n rac1
+ASM1 /u01/app/oracle/product/10.2.0/asm
[oracle@rac2 ~]$ srvctl config asm -n rac2
+ASM2 /u01/app/oracle/product/10.2.0/asm
[oracle@rac2 ~]$ export ORACLE_SID=+ASM2
[oracle@rac2 ~]$ sqlplus /nolog
SQL*Plus: Release 10.2.0.2.0 - Production on Thu Mar 5 01:42:05 2009
Copyright (c) 1982, 2005, Oracle. All Rights Reserved.
SQL> conn / as sysdba
Connected to an idle instance.
SQL> exit
Disconnected
[oracle@rac2 ~]$ srvctl status asm -n rac1
ASM instance +ASM1 is running on node rac1.
[oracle@rac2 ~]$
这样就明白了，ASM的卷组丢失，但是ASM的进程仍旧运行，看来这次ASM的魂魄是丢失啦。。。。。
来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/8334342/viewspace-561445/ ，如需转载，请注明出处，否则将追究法律责任。
