环境：两个节点的Solaris10 + Oracle10g RAC + ASM环境

bash-3.00# su - oracle

Oracle Corporation SunOS 5.10 Generic Patch January 2005

You have new mail.

[dbserv2]@dbserv-db2[/export/home/oracle]$crs_stat -t -v

Name Type R/RA F/FT Target State Host

----------------------------------------------------------------------

ora....SM1.asm application 0/5 0/0 ONLINE OFFLINE

ora....B1.lsnr application 0/5 0/0 ONLINE OFFLINE

ora....db1.gsd application 0/5 0/0 ONLINE OFFLINE

ora....db1.ons application 0/3 0/0 ONLINE OFFLINE

ora....db1.vip application 0/0 0/0 ONLINE ONLINE dbserv-db2

ora....SM2.asm application 0/5 0/0 ONLINE ONLINE dbserv-db2

ora....B2.lsnr application 0/5 0/0 ONLINE ONLINE dbserv-db2

ora....db2.gsd application 0/5 0/0 ONLINE ONLINE dbserv-db2

ora....db2.ons application 0/3 0/0 ONLINE ONLINE dbserv-db2

ora....db2.vip application 0/0 0/0 ONLINE ONLINE dbserv-db2

ora.dbserv.db application 0/0 0/1 ONLINE ONLINE dbserv-db2

ora....s1.inst application 0/5 0/0 ONLINE OFFLINE

ora....s2.inst application 0/5 0/0 ONLINE ONLINE dbserv-db2

[dbserv2]@dbserv-db2[/export/home/oracle]$srvctl start listener -n dbserv-db1

CRS-0223: Resource 'ora.dbserv-db1.LISTENER_dbserv-DB1.lsnr' has placement error.

[dbserv2]@dbserv-db2[/export/home/oracle]$srvctl start asm -n dbserv-db1 -i +ASM1

PRKS-1009 : Failed to start ASM instance "+ASM1" on node "dbserv-db1", [CRS-0223: Resource 'ora.dbserv-db1.ASM1.asm' has placement error.]

[@more@]

1、在节点1的CRS_HOME/bin下以root用户执行如下启动crs命令，crs_stat -t -v结果根以上一样。

bash-3.00# ./crsctl start crs

Attempting to start CRS stack

The CRS stack will be started shortly

2、初步怀疑节点1可能认不到存储，但是format命令在两个节点的结果一样：

bash-3.00# format

Searching for disks...done

3、查看crsd日志：

[dbserv1]@dbserv-db1[/oracle/product/10.2/crs/log]$ls

crs dbserv-db1

[dbserv1]@dbserv-db1[/oracle/product/10.2/crs/log]$cd xj*

[dbserv1]@dbserv-db1[/oracle/product/10.2/crs/log/dbserv-db1]$ls

admin alertdbserv-db1.log client crsd cssd evmd racg

[dbserv1]@dbserv-db1[/oracle/product/10.2/crs/log/dbserv-db1]$cd crsd*

[dbserv1]@dbserv-db1[/oracle/product/10.2/crs/log/dbserv-db1/crsd]$ls

core.2011-04-17-16:29:16 crsd.log

[dbserv1]@dbserv-db1[/oracle/product/10.2/crs/log/dbserv-db1/crsd]$tail -f crsd.log

2011-07-26 04:35:55.365: [ CRSRES][569004] StopResource: setting CLI values

2011-07-26 04:35:55.390: [ CRSRES][569004] Attempting to stop `ora.dbserv.db` on member `dbserv-db2`

2011-07-26 04:35:56.036: [ CRSRES][569004] Stop of `ora.dbserv.db` on member `dbserv-db2` succeeded.

2011-07-26 04:35:56.053: [ CRSRES][568997] Stop of `ora.dbserv.dbserv1.inst` on member `dbserv-db1` succeeded.

2011-07-26 04:35:57.866: [ CRSRES][569006] `ora.dbserv.dbserv1.inst` is already OFFLINE.

2011-07-26 04:37:28.528: [ CRSRES][569013] startRunnable: setting CLI values

2011-07-26 04:37:28.530: [ CRSRES][569013] Attempting to start `ora.dbserv.dbserv1.inst` on member `dbserv-db1`

2011-07-26 04:37:30.453: [ CRSRES][569013] Start of `ora.dbserv.dbserv1.inst` on member `dbserv-db1` succeeded.

2011-07-26 04:37:30.504: [ CRSRES][569020] Attempting to start `ora.dbserv.db` on member `dbserv-db2`

2011-07-26 04:37:31.982: [ CRSRES][569020] Start of `ora.dbserv.db` on member `dbserv-db2` succeeded.

^C

4、查看IP地址及Hosts文件，也属于正常

bash-3.00# ifconfig -a | more

bash-3.00# cat /etc/hosts

5、有查看了如下，都属于正常

[dbserv1]@dbserv-db1[/var/adm]$crsctl query css votedisk

0. 0 /dev/rac/vote1

1. 0 /dev/rac/vote2

2. 0 /dev/rac/vote3

located 3 votedisk(s).

[dbserv1]@dbserv-db1[/var/adm]$cd /dev/rac

[dbserv1]@dbserv-db1[/dev/rac]$ls -ltr

[dbserv1]@dbserv-db1[/opt]$cd /var/opt/oracle

[dbserv1]@dbserv-db1[/var/opt/oracle]$ls

ocr.loc oprocd oraInst.loc oratab scls_scr

[dbserv1]@dbserv-db1[/var/opt/oracle]$cat oraInst.loc

inventory_loc=/oracle/oraInventory

inst_group=oinstall

[dbserv1]@dbserv-db1[/var/opt/oracle]$cat ocr.loc

ocrconfig_loc=/dev/rac/ocr1

ocrmirrorconfig_loc=/dev/rac/ocr2

local_only=FALSE

[dbserv1]@dbserv-db1[/var/opt/oracle]$cat oratab

#

6、查看系统日志，也属正常：

[dbserv1]@dbserv-db1[/var/adm]$ls

acct exacct log messages.0 messages.2 pool sm.bin streams utmpx wtmpx

aculog lastlog messages messages.1 messages.3 sa spellhist sulog vold.log

[dbserv1]@dbserv-db1[/var/adm]$tail -10f messages

Jul 7 05:00:00 dbserv-db1 sendmail[20005]: [ID 702911 mail.crit] My unqualified host name (dbserv-db1) unknown; sleeping for retry

Jul 7 05:01:00 dbserv-db1 sendmail[20005]: [ID 702911 mail.alert] unable to qualify my own domain name (dbserv-db1) -- using short name

Jul 8 05:00:00 dbserv-db1 sendmail[10035]: [ID 702911 mail.crit] My unqualified host name (dbserv-db1) unknown; sleeping for retry

Jul 8 05:01:00 dbserv-db1 sendmail[10035]: [ID 702911 mail.alert] unable to qualify my own domain name (dbserv-db1) -- using short name

Jul 9 05:00:00 dbserv-db1 sendmail[773]: [ID 702911 mail.crit] My unqualified host name (dbserv-db1) unknown; sleeping for retry

7、虽然节点1的小型机重启 两三次，但故障现象还是一样，/var/adm/下的系统日志没有什么可参考的错误。最终存储工程师通过串口线实时观察boot过程的日志输出，发现在启动过程中某块硬盘挂载点报错。

8、当启动完毕后查看vsftab文件，发现有个挂载点，跟系统日志的磁盘标示相同，观察确定为没有用的（过去忘了注释）挂载点，注释掉后重新启动了服务器后节点1的crs自动启动了。

bash-3.00# more /etc/vfstab

#device device mount FS fsck mount mount

#to mount to fsck point type pass at boot options

#

fd - /dev/fd fd - no -

/proc - /proc proc - no -

/dev/md/dsk/d10 - - swap - no -

/dev/md/dsk/d0 /dev/md/rdsk/d0 / ufs 1 no -

/dev/md/dsk/d20 /dev/md/rdsk/d20 /var ufs 1 no -

/dev/md/dsk/d30 /dev/md/rdsk/d30 /opt ufs 2 yes -

/dev/md/dsk/d40 /dev/md/rdsk/d40 /oracle ufs 2 yes -

/devices - /devices devfs - no -

sharefs - /etc/dfs/sharetab sharefs - no -

ctfs - /system/contract ctfs - no -

objfs - /system/object objfs - no -

swap - /tmp tmpfs - yes -

#/dev/dsk/c3t6000B0C100050000d0s6 /dev/rdsk/c3t6000B0C100050000d0s6 /aaa ufs 2 yes -

bash-3.00#