1 rman报错信息

$ rman

Recovery Manager: Release 10.2.0.1.0 - Production on Mon Jun 23 18:51:35 2014

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-00554: initialization of internal recovery manager package failed

RMAN-03000: recovery manager compiler component initialization failed

RMAN-06035: wrong version of recover.bsq, expecting 10.2.0.1, found 10.2.0.4

2 经过比对，发现两个节点的lib文件不一致，经咨询，应该是在几年前升级PSU导致的问题。

在 2009年12月28日做数据库升级，过程中有部分的功能升级失败，导致节点2的RMAN不能使用。

节点 1

-rwxr-x---   1 oracle     oinstall   36877360 Dec 28  2009 /oracle/oracle10g2r/dbs/lib/libclntsh.so.10.1

$ -r-xr-xr-x   1 oracle     oinstall   5337112 Apr 22  2008 /oracle/oracle10g2r/dbs/lib/libnnz10.so

$ -r-xr-xr-x   1 bin        bin          85568 Feb 16  2007 /usr/lib/hpux64/librt.so.1

$ -r-xr-xr-x   1 bin        bin          91184 Jun  4  2009 /usr/lib/hpux64/libnss_dns.so.1

$ -r-xr-xr-x   1 bin        bin          78488 Jun  9  2009 /usr/lib/hpux64/libdl.so.1

$ -r-xr-xr-x   1 bin        bin        6580584 Dec  4  2008 /usr/lib/hpux64/libm.so.1

$ -r-xr-xr-x   1 bin        bin        1535512 Jun  6  2009 /usr/lib/hpux64/libpthread.so.1

$ -r-xr-xr-x   1 bin        bin         635200 Dec 13  2008 /usr/lib/hpux64/libunwind.so.1

$ -r-xr-xr-x   1 bin        bin        1510856 Jun  4  2009 /usr/lib/hpux64/libnsl.so.1

$ -r-xr-xr-x   1 bin        bin        4898536 Apr  8  2009 /usr/lib/hpux64/libc.so.1

$ -r-xr-xr-x   1 bin        bin          85568 Feb 16  2007 /usr/lib/hpux64/librt.so.1

$ -r-xr-xr-x   1 bin        bin          91184 Jun  4  2009 /usr/lib/hpux64/libnss_dns.so.1

$ -r-xr-xr-x   1 bin        bin          78488 Jun  9  2009 /usr/lib/hpux64/libdl.so.1

$ -r-xr-xr-x   1 bin        bin        6580584 Dec  4  2008 /usr/lib/hpux64/libm.so.1

$ -r-xr-xr-x   1 bin        bin        1535512 Jun  6  2009 /usr/lib/hpux64/libpthread.so.1

$ -r-xr-xr-x   1 bin        bin         635200 Dec 13  2008 /usr/lib/hpux64/libunwind.so.1

$ -r-xr-xr-x   1 bin        bin        1510856 Jun  4  2009 /usr/lib/hpux64/libnsl.so.1

$ -r-xr-xr-x   1 bin        bin          78488 Jun  9  2009 /usr/lib/hpux64/libdl.so.1

$ -r-xr-xr-x   1 bin        bin          85680 Feb 16  2007 /usr/lib/hpux64/libuca.so.1

节点 2

  -rwxr-xr-x   1 oracle     oinstall   36155448 Dec 26  2009 /oracle/oracle10g2r/dbs/lib/libclntsh.so.10.1

$ -r-xr-xr-x   1 oracle     oinstall   5227072 Feb  4  2006 /oracle/oracle10g2r/dbs/lib/libnnz10.so

$ -r-xr-xr-x   1 bin        bin          85568 Feb 16  2007 /usr/lib/hpux64/librt.so.1

$ -r-xr-xr-x   1 bin        bin          91184 Jun  4  2009 /usr/lib/hpux64/libnss_dns.so.1

$ -r-xr-xr-x   1 bin        bin          78488 Jun  9  2009 /usr/lib/hpux64/libdl.so.1

$ -r-xr-xr-x   1 bin        bin        6580584 Dec  4  2008 /usr/lib/hpux64/libm.so.1

$ -r-xr-xr-x   1 bin        bin        1535512 Jun  6  2009 /usr/lib/hpux64/libpthread.so.1

$ -r-xr-xr-x   1 bin        bin         635200 Dec 13  2008 /usr/lib/hpux64/libunwind.so.1

$ -r-xr-xr-x   1 bin        bin        1510856 Jun  4  2009 /usr/lib/hpux64/libnsl.so.1

$ -r-xr-xr-x   1 bin        bin        4898536 Apr  8  2009 /usr/lib/hpux64/libc.so.1

$ -r-xr-xr-x   1 bin        bin          85568 Feb 16  2007 /usr/lib/hpux64/librt.so.1

$ -r-xr-xr-x   1 bin        bin          91184 Jun  4  2009 /usr/lib/hpux64/libnss_dns.so.1

$ -r-xr-xr-x   1 bin        bin          78488 Jun  9  2009 /usr/lib/hpux64/libdl.so.1

$ -r-xr-xr-x   1 bin        bin        6580584 Dec  4  2008 /usr/lib/hpux64/libm.so.1

$ -r-xr-xr-x   1 bin        bin        1535512 Jun  6  2009 /usr/lib/hpux64/libpthread.so.1

$ -r-xr-xr-x   1 bin        bin         635200 Dec 13  2008 /usr/lib/hpux64/libunwind.so.1

$ -r-xr-xr-x   1 bin        bin        1510856 Jun  4  2009 /usr/lib/hpux64/libnsl.so.1

$ -r-xr-xr-x   1 bin        bin          78488 Jun  9  2009 /usr/lib/hpux64/libdl.so.1

$ -r-xr-xr-x   1 bin        bin          85680 Feb 16  2007 /usr/lib/hpux64/libuca.so.1

发现节点 1和节点2上有部分的库文件不一致。

三 故障处理

1 关闭节点2数据库

2 在节点2上重命名文件：

$ mv /oracle/oracle10g2r/dbs/lib/libclntsh.so.10.1 /oracle/oracle10g2r/dbs/lib/libclntsh.so.10.1.bak623

$ mv /oracle/oracle10g2r/dbs/lib/libnnz10.so /oracle/oracle10g2r/dbs/lib/libnnz10.so.bak623

$ ls -l /oracle/oracle10g2r/dbs/lib/libclntsh.so.10.1 /oracle/oracle10g2r/dbs/lib/libnnz10.so

/oracle/oracle10g2r/dbs/lib/libclntsh.so.10.1 not found

/oracle/oracle10g2r/dbs/lib/libnnz10.so not found

$ ls -l /oracle/oracle10g2r/dbs/lib/libclntsh.so.10.1.bak2014623 /oracle/oracle10g2r/dbs/lib/libnnz10.so.bak2014623

/oracle/oracle10g2r/dbs/lib/libclntsh.so.10.1.bak2014623 not found

/oracle/oracle10g2r/dbs/lib/libnnz10.so.bak2014623 not found

$ ls -l /oracle/oracle10g2r/dbs/lib/libclntsh.so.10.1.bak623 /oracle/oracle10g2r/dbs/lib/libnnz10.so.bak623

-rwxr-xr-x   1 oracle     oinstall   36155448 Dec 26  2009 /oracle/oracle10g2r/dbs/lib/libclntsh.so.10.1.bak623

-r-xr-xr-x   1 oracle     oinstall   5227072 Feb  4  2006 /oracle/oracle10g2r/dbs/lib/libnnz10.so.bak623

3 登陆到节点1，拷贝节点1的文件到节点2

$ scp /oracleacle10g2r/dbs/lib/libclntsh.so.10.1 oracle@cwgkdb2:/oracle/oracle10g2r/dbs/lib

Password:

libclntsh.so.10.1                                                                                                                      100%   35MB  35.2MB/s  29.1MB/s   00:01    

$ scp /oracle/oracle10g2r/dbs/lib/libnnz10.so  oracle@cwgkdb2:/oracle/oracle10g2r/dbs/lib

Password:

libnnz10.so                                                                                                                            100% 5212KB   5.1MB/s   5.1MB/s   00:01    

$

$ ls -l  /oracle/oracle10g2r/dbs/lib/libclntsh.so.10.1 /oracle/oracle10g2r/dbs/lib/libnnz10.so

-rwxr-x---   1 oracle     oinstall   36877360 Jun 23 19:55 /oracle/oracle10g2r/dbs/lib/libclntsh.so.10.1

-r-xr-xr-x   1 oracle     oinstall   5337112 Jun 23 19:55 /oracle/oracle10g2r/dbs/lib/libnnz10.so

4 启动数据库

startup

5 rman 验证

$ sqlplus / as sysdba

SQL*Plus: Release 10.2.0.4.0 - Production on Mon Jun 23 19:56:10 2014

Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.

Connected to an idle instance.

SQL> startup

ORACLE instance started.

Total System Global Area 3.2212E+10 bytes

Fixed Size                  2213440 bytes

Variable Size            8561899968 bytes

Database Buffers         2.3606E+10 bytes

Redo Buffers               42598400 bytes

Database mounted.

Database opened.

SQL> exit

Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production

With the Partitioning, Real Application Clusters, OLAP, Data Mining

and Real Application Testing options

$ rman

Recovery Manager: Release 10.2.0.4.0 - Production on Mon Jun 23 19:56:59 2014

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

RMAN> exit

Recovery Manager complete.

$ rman target /

Recovery Manager: Release 10.2.0.4.0 - Production on Mon Jun 23 19:57:20 2014

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

connected to target database: CTESTPD (DBID=543556404)

RMAN> exit

Recovery Manager complete.