11.2.0.3 rac-单机的DG

由于数据库部署监控，部署时把DG端的tnsname.ora文件覆盖了，到时DG和主库RAC节点的通讯中断，然后就出现奇怪的问题；

DG节点：

alert.log:

ERROR: Failed to connect with connect string: (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/11.2.0.3/
grid/bin/oracle)(ARGV0=oracle+ASM_pr00_popdg)(ENVS='ORACLE_HOME=/u01/app/11.2.0.3/grid,ORACLE_SID=+ASM')(ARGS
='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser))
WARNING: ASM communication error: op 0 state 0x0 (15055)
ERROR: direct connection failure with ASM
NOTE: Deferred communication with ASM instance
Errors in file /u01/app/oracle/diag/rdbms/popdg/popdg/trace/popdg_pr00_8502.trc:
ORA-15055: unable to connect to ASM instance
ORA-12547: TNS:lost contact
ORA-00308: cannot open archived log '+RECO_DM02/popdg/archivelog/2013_04_26/thread_2_seq_31174.2074.813766677
'
ORA-17503: ksfdopn:2 Failed to open file +RECO_DM02/popdg/archivelog/2013_04_26/thread_2_seq_31174.2074.81376
6677
ORA-15055: unable to connect to ASM instance
ORA-12547: TNS:lost contact
NOTE: deferred map free for map id 946645589
Recovery interrupted!

-------------------------------------------------------------------------------------------------------------

重新配置tnsnames.ora文件，保证DG和主库之间tnsping 正常通讯；

-------------------------------------------------------------------------------------------------------------

取消应用日志发现异常：

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
*
ERROR at line 1:
ORA-16136: Managed Standby Recovery not active

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ ONLY

很明确之前数据的恢复进程已经挂起了！

关闭并启动DG数据库

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area 7.2421E+10 bytes
Fixed Size                  2235808 bytes
Variable Size            5.1540E+10 bytes
Database Buffers         2.0670E+10 bytes
Redo Buffers              209866752 bytes
ORA-00205: error in identifying control file, check alert log for more info

意外发生了，

检查DG的alert.log:

ERROR: Failed to connect with connect string: (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/11.2.0.3/grid/bin/oracle)(ARGV0=oracle+ASM_asmb_popdg)(ENVS='ORACLE_HOME=/u01/app/11.2.0.3/grid,ORACLE_SID=+ASM')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser))
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+RECO_DM02/popdg/controlfile/standbyctl02.ctl.257.799339773'
ORA-17503: ksfdopn:2 Failed to open file +RECO_DM02/popdg/controlfile/standbyctl02.ctl.257.799339773
ORA-15001: diskgroup "RECO_DM02" does not exist or is not mounted
ORA-15055: unable to connect to ASM instance
ORA-12547: TNS:lost contact
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATA_DM02/popdg/controlfile/standbyctl01.ctl.257.799339639'
ORA-17503: ksfdopn:2 Failed to open file +DATA_DM02/popdg/controlfile/standbyctl01.ctl.257.799339639
ORA-15001: diskgroup "DATA_DM02" does not exist or is not mounted
ORA-15055: unable to connect to ASM instance
ORA-12547: TNS:lost contact
ORA-205 signalled during: alter database mount...
Fri Apr 26 15:40:48 2013
Starting background process ASMB
Fri Apr 26 15:40:48 2013
ASMB started with pid=27, OS id=31912

***********************************************************************

Fatal NI connect error 12547, connecting to:
(DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/11.2.0.3/grid/bin/oracle)(ARGV0=oracle+ASM_asmb_popdg)(ENVS='ORACLE_HOME=/u01/app/11.2.0.3/grid,ORACLE_SID=+ASM')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser)(CONNECT_DATA=(CID=(PROGRAM=oracle@popdg)(HOST=popdg)(USER=oracle))))

全部都是这种错误；

-------------------------------------------------------------------------------------------------------------

怀疑是DG节点的ASM盘组出现问题，检查DG节点的crs状态和磁盘组状态：

[grid@popdg ~]$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....DM02.dg ora....up.type ONLINE    ONLINE    popdg
ora....ER.lsnr ora....er.type ONLINE    ONLINE    popdg
ora....DM02.dg ora....up.type ONLINE    ONLINE    popdg
ora.asm        ora.asm.type   ONLINE    ONLINE    popdg
ora.cssd       ora.cssd.type ONLINE    ONLINE    popdg
ora.diskmon    ora....on.type OFFLINE   OFFLINE
ora.evmd       ora.evm.type   ONLINE    ONLINE    popdg
ora.ons        ora.ons.type   OFFLINE   OFFLINE

SQL> select name,state from v$asm_diskgroup;

NAME         STATE
---------   ----------
DATA_DM02    MOUNTED
RECO_DM02    MOUNTED

一切都正常！！！！！

------------------------------------------------------------------------------------------------------------

那么问题出现在哪儿，怀疑是ORACLE_HOME/bin和grid_home/bin下的oracel执行文件出现问题，于是检查两个bin下的oracle文件的执行权限：

[grid@node bin]$ls -l oracle

-rwxr-xr-x oracle

[oracle@node bin]$ ls -l oracle

-r-srws--x oracle

--------------------------------------------------------------------------------------------------------------

很明显grid_home/bin下的oracle执行文件的权限被修改了：

修改执行权限：

chmod 6751 oracle

然后启动数据库ok!!!

这问题相当的诡异，还没有研究明白为什么grid_home/bin下的oracle执行文件的权限会被修改？什么操作引起的权限修改？按道理说覆盖了tnsnames.ora文件，DG和主库的通讯中断，不会传输日志，不能进行恢复，为什么直接导致那么多问题出现，还有待研究！

ITPub博客

DATAGUARD异常处理

最新文章