ORA-15055: unable to connect to ASM instance 造成 RAC的db不能启动
$ crsctl status res ora.racdb.db
NAME=ora.racdb.db
TYPE=ora.database.type
TARGET=ONLINE , ONLINE
STATE=OFFLINE, OFFLINE
启动crs，发现db服务没起来，如是尝试单独启动
$ crs_start ora.racdb.db
Attempting to start `ora.racdb.db` on member `rac01`
Attempting to start `ora.racdb.db` on member `rac02`
ORA-00205: error in identifying control file, check alert log for more info
Start of `ora.racdb.db` on member `rac01` failed.
Attempting to stop `ora.racdb.db` on member `rac01`
ORA-00205: error in identifying control file, check alert log for more info
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
Start of `ora.racdb.db` on member `rac02` failed.
Attempting to stop `ora.racdb.db` on member `rac02`
Stop of `ora.racdb.db` on member `rac01` succeeded.
CRS-2632: There are no more servers to try to place resource 'ora.racdb.db' on that would satisfy its placement policy
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
Stop of `ora.racdb.db` on member `rac02` succeeded.
CRS-0223: Resource 'ora.racdb.db 1 1' has placement error.
CRS-0215: Could not start resource 'ora.racdb.db 2 1'.
还是不能启动。
检查oracle警告日志
# tail -200 /app/oracle/product/oracle/diag/rdbms/racdb/racdb1/trace/alert_racdb1.log
lmon registered with NM - instance number 1 (internal mem no 0)
***********************************************************************
Fatal NI connect error 12547, connecting to:
(DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/app/grid/11.2.0/bin/oracle)(ARGV0=oracle+ASM1_asmb_racdb1)(ENVS='ORACLE_HOME=/app/grid/11.2.0,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser)(CONNECT_DATA=(CID=(PROGRAM=oracle@rac01)(HOST=rac01)(USER=oracle))))
VERSION INFORMATION:
TNS for Linux: Version 11.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.1.0 - Production
Time: 08-OCT-2013 14:38:07
Tracing not turned on.
Tns error struct:
ns main err code: 12547
TNS-12547: TNS:lost contact
ns secondary err code: 12560
nt main err code: 517
TNS-00517: Lost contact
nt secondary err code: 32
nt OS err code: 0
ERROR: Failed to connect with connect string: (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/app/grid/11.2.0/bin/oracle)(ARGV0=oracle+ASM1_asmb_racdb1)(ENVS='ORACLE_HOME=/app/grid/11.2.0,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser))
Errors in file /app/oracle/product/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_asmb_7238.trc:
ORA-15055: unable to connect to ASM instance
ORA-12547: TNS:lost contact
Reconfiguration started (old inc 0, new inc 4)
List of instances:
1 2 (myinst: 1)
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
Tue Oct 08 14:38:08 2013
LCK0 started with pid=23, OS id=7254
Starting background process RSMN
Tue Oct 08 14:38:08 2013
RSMN started with pid=28, OS id=7257
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
Reusing ORACLE_BASE from an earlier startup = /app/grid/crs
Tue Oct 08 14:38:09 2013
ALTER DATABASE MOUNT
This instance was first to mount
Starting background process ASMB
Tue Oct 08 14:38:09 2013
ASMB started with pid=30, OS id=7267
***********************************************************************
Fatal NI connect error 12547, connecting to:
(DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/app/grid/11.2.0/bin/oracle)(ARGV0=oracle+ASM1_asmb_racdb1)(ENVS='ORACLE_HOME=/app/grid/11.2.0,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser)(CONNECT_DATA=(CID=(PROGRAM=oracle@rac01)(HOST=rac01)(USER=oracle))))
VERSION INFORMATION:
TNS for Linux: Version 11.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.1.0 - Production
Time: 08-OCT-2013 14:38:09
Tracing not turned on.
Tns error struct:
ns main err code: 12547
TNS-12547: TNS:lost contact
ns secondary err code: 12560
nt main err code: 517
TNS-00517: Lost contact
nt secondary err code: 32
nt OS err code: 0
ERROR: Failed to connect with connect string: (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/app/grid/11.2.0/bin/oracle)(ARGV0=oracle+ASM1_asmb_racdb1)(ENVS='ORACLE_HOME=/app/grid/11.2.0,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser))
Starting background process ASMB
Tue Oct 08 14:38:09 2013
ASMB started with pid=31, OS id=7280
***********************************************************************
Fatal NI connect error 12547, connecting to:
(DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/app/grid/11.2.0/bin/oracle)(ARGV0=oracle+ASM1_asmb_racdb1)(ENVS='ORACLE_HOME=/app/grid/11.2.0,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser)(CONNECT_DATA=(CID=(PROGRAM=oracle@rac01)(HOST=rac01)(USER=oracle))))
VERSION INFORMATION:
TNS for Linux: Version 11.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.1.0 - Production
Time: 08-OCT-2013 14:38:09
Tracing not turned on.
Tns error struct:
ns main err code: 12547
TNS-12547: TNS:lost contact
ns secondary err code: 12560
nt main err code: 517
TNS-00517: Lost contact
nt secondary err code: 32
nt OS err code: 0
ERROR: Failed to connect with connect string: (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/app/grid/11.2.0/bin/oracle)(ARGV0=oracle+ASM1_asmb_racdb1)(ENVS='ORACLE_HOME=/app/grid/11.2.0,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser))
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATA/racdb/control02.ctl'
ORA-17503: ksfdopn:2 Failed to open file +DATA/racdb/control02.ctl
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15055: unable to connect to ASM instance
ORA-12547: TNS:lost contact
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATA/racdb/control01.ctl'
ORA-17503: ksfdopn:2 Failed to open file +DATA/racdb/control01.ctl
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15055: unable to connect to ASM instance
ORA-12547: TNS:lost contact
Starting background process ASMB
ORA-205 signalled during: ALTER DATABASE MOUNT...
Tue Oct 08 14:38:09 2013
ASMB started with pid=30, OS id=7290
Tue Oct 08 14:38:09 2013
Shutting down instance (abort)
License high water mark = 1
USER (ospid: 7294): terminating the instance
Instance terminated by USER, pid = 7294
Tue Oct 08 14:38:11 2013
Instance shutdown complete
发现 asm 报错，不能连接。如是，找到asm的日志
# tail -f /app/grid/crs/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log
...
Tue Oct 08 14:57:17 2013
Errors in file /app/grid/crs/diag/asm/+asm/+ASM1/trace/+ASM1_ora_8661.trc:
ORA-27070: async read/write failed
WARNING: IO Failed. group:0 disk(number.incarnation):0.0xe96ba85e disk_path:/dev/random
AU:0 disk_offset(bytes):0 io_size:4096 operation:Read type:asynchronous
result:I/O error process_id:8661
Errors in file /app/grid/crs/diag/asm/+asm/+ASM1/trace/+ASM1_ora_8661.trc:
ORA-27070: async read/write failed
WARNING: IO Failed. group:0 disk(number.incarnation):0.0xe96ba85e disk_path:/dev/random
AU:0 disk_offset(bytes):0 io_size:4096 operation:Read type:asynchronous
result:I/O error process_id:8661
根据日志信息找到报错的trace文件。
# tail -100 /app/grid/crs/diag/asm/+asm/+ASM1/trace/+ASM1_ora_8661.trc
...
*** 2013-10-08 14:57:17.778
ORA-27070: async read/write failed
WARNING: IO Failed. group:0 disk(number.incarnation):0.0xe96ba85e disk_path:/dev/random
AU:0 disk_offset(bytes):0 io_size:4096 operation:Read type:asynchronous
result:I/O error process_id:8661
subsys:System iop:0x7fd245f3f108 bufp:0x7fd246078600 osderr:0x0 osderr1:0x0
*** 2013-10-08 14:57:27.144
ORA-27070: async read/write failed
WARNING: IO Failed. group:0 disk(number.incarnation):0.0xe96ba85e disk_path:/dev/random
AU:0 disk_offset(bytes):0 io_size:4096 operation:Read type:asynchronous
result:I/O error process_id:8661
subsys:System iop:0x7fd245f3efe0 bufp:0x7fd246078600 osderr:0x0 osderr1:0x0
貌似是 disk_path:/dev/random 的 IO错误。奇怪，怎么多了这个磁盘，于是切换到grid用户
SQL> select name,path from v$asm_disk;
NAME PATH
------------------------------ --------------------------------------------------
/dev/random
FRA_0000 /dev/asm-diskd
CRS_0000 /dev/asm-diskb
DATA_0000 /dev/asm-diskc
发现asm disk磁盘里面多了一条记录“/dev/random”。
SQL> select name,state,total_mb from v$asm_diskgroup;
NAME STATE TOTAL_MB
------------------------------ ----------- ----------
FRA MOUNTED 51199
CRS MOUNTED 51199
DATA MOUNTED 51199
SQL> select group_number, name, path, total_mb, free_mb, total_mb-free_mb used_mb from v$asm_disk_stat;
GROUP_NUMBER NAME PATH TOTAL_MB FREE_MB USED_MB
------------ ------------------------------ -------------------------------------------------- ---------- ---------- ----------
3 FRA_0000 /dev/asm-diskd 51199 51097 102
1 CRS_0000 /dev/asm-diskb 51199 50803 396
2 DATA_0000 /dev/asm-diskc 51199 45917 5282
经过朋友指点
$ sqlplus / as sysasm
SQL> show parameter disk
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups string DATA, FRA
asm_diskstring string /dev
修改第二个参数，把范围进一步精确。
SQL> alter system set asm_diskstring='/dev/asm-disk*' scope=both sid='+ASM1';
SQL> alter system set asm_diskstring='/dev/asm-disk*' scope=both sid='+ASM2';
修改后，重启crs服务。分别在2个节点上执行如下：
# /app/grid/11.2.0/bin/crsctl stop crs
# /app/grid/11.2.0/bin/crsctl start crs
自此，不再报关于“/dev/random”磁盘IO错误。
来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/9399028/viewspace-773919/ ，如需转载，请注明出处，否则将追究法律责任。
请登录后发表评论
登录
全部评论
