• Oracle 版本：11.2.0.4

• Oracle 部署架构：双节点的RAC

• 主机类型：HP unix

• 补丁信息：31326410（11G HPUNIX  2020年10月的补丁）

• 处理思路：补丁打了一半报错，为了节约时间，我们想继续下去，同时也在问题发生后找到了导致问题的原因
  

第一 打补丁方式

Oracle 补丁包中推荐了好几种打的方法，此次我们选择如下方式：

GI: /grid/app/11.2.0.4/grid/OPatch/opatch auto /software/31326410/31305209    -oh /grid/app/11.2.0.4/grid 

ORACLE: /oracle/app/oracle/11.2.0.4/db_1/OPatch/opatch auto /software/31326410/31305209 -oh /oracle/app/oracle/11.2.0.4/db_1

OJVM:/software/31326410/31219953]opatch apply

第二 报错信息

在打补丁过程 GI 报错

jkzcdb1#[/]/grid/app/11.2.0.4/grid/OPatch/opatch auto /oracle/software/11G/patch/31326410/31305209 -oh /grid/app/11.2.0.4/grid
Executing /grid/app/11.2.0.4/grid/perl/bin/perl /grid/app/11.2.0.4/grid/OPatch/crs/patch11203.pl -patchdir /oracle/software/11G/patch/31326410 -patchn 31305209 -oh /grid/app/11.2.0.4/grid -paramfile /grid/app/11.2.0.4/grid/crs/install/crsconfig_params

This is the main log file: /grid/app/11.2.0.4/grid/cfgtoollogs/opatchauto2021-06-04_20-47-47.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/grid/app/11.2.0.4/grid/cfgtoollogs/opatchauto2021-06-04_20-47-47.report.log

2021-06-04 20:47:47: Starting Clusterware Patch Setup
Using configuration parameter file: /grid/app/11.2.0.4/grid/crs/install/crsconfig_params

Stopping CRS...
Stopped CRS successfully
patch /oracle/software/11G/patch/31326410/31305209/29938455  apply successful for home  /grid/app/11.2.0.4/grid
patch /oracle/software/11G/patch/31326410/31305209/31103343  apply failed  for home  /grid/app/11.2.0.4/grid

Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.
Oracle Grid Infrastructure stack start initiated but failed to complete at /oracle/software/11G/patch/31326410/31305209/29938455/files/crs/install/crsconfig_lib.pm line 11821.

在打补丁过程 DATABASE 报错信息：

root@jkzcdb1:[/]/oracle/app/oracle/11.2.0.4/db_1/OPatch/opatch auto /oracle/software/31326410/31305209 -oh /oracle/app/oracle/11.2.0.4/db_1
Executing /grid/app/11.2.0.4/grid/perl/bin/perl /oracle/app/oracle/11.2.0.4/db_1/OPatch/crs/patch11203.pl -patchdir /oracle/software/31326410 -patchn 31305209 -oh /oracle/app/oracle/11.2.0.4/db_1 -paramfile /grid/app/11.2.0.4/grid/crs/install/crsconfig_params

This is the main log file: /oracle/app/oracle/11.2.0.4/db_1/cfgtoollogs/opatchauto2021-06-04_22-03-39.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/oracle/app/oracle/11.2.0.4/db_1/cfgtoollogs/opatchauto2021-06-04_22-03-39.report.log

2021-06-04 22:03:39: Starting Clusterware Patch Setup
Using configuration parameter file: /grid/app/11.2.0.4/grid/crs/install/crsconfig_params

Stopping RAC /oracle/app/oracle/11.2.0.4/db_1 ...
Stopped RAC /oracle/app/oracle/11.2.0.4/db_1 successfully

patch /oracle/software/31326410/31305209/29938455/custom/server/29938455  apply successful for home  /oracle/app/oracle/11.2.0.4/db_1
patch /oracle/software/31326410/31305209/31103343  apply failed  for home  /oracle/app/oracle/11.2.0.4/db_1

Starting RAC /oracle/app/oracle/11.2.0.4/db_1 ...
Started RAC /oracle/app/oracle/11.2.0.4/db_1 successfully

opatch auto succeeded.

第三 处理方法，查找补丁日志，手动继续

查找补丁的log,如上方的GI 日志 /grid/app/11.2.0.4/grid/cfgtoollogs/opatchauto2021-06-04_20-47-47.log，

DB 的日志/oracle/app/oracle/11.2.0.4/db_1/cfgtoollogs/opatchauto2021-06-04_22-03-39.log

例如DB：如下错误信息：

2021-06-04 22:06:08: patch /oracle/software/31326410/31305209/29938455/custom/server/29938455  apply successful for home  /oracle/app/oracle/11.2.0.4/db_1

2021-06-04 22:06:08: Executing command /oracle/app/oracle/11.2.0.4/db_1/OPatch/opatch napply /oracle/software/31326410/31305209/31103343 -local -silent -oh /oracle/app/oracle/11.2.0.4/db_1
-invPtrLoc /oracle/app/oracle/11.2.0.4/db_1/oraInst.loc as oracle
2021-06-04 22:06:08: Running as user oracle: /oracle/app/oracle/11.2.0.4/db_1/OPatch/opatch napply /oracle/software/31326410/31305209/31103343 -local -silent -oh /oracle/app/oracle/11.2.0.4
/db_1 -invPtrLoc /oracle/app/oracle/11.2.0.4/db_1/oraInst.loc
2021-06-04 22:06:08: s_run_as_user2: Running /bin/su oracle -c ' /oracle/app/oracle/11.2.0.4/db_1/OPatch/opatch napply /oracle/software/31326410/31305209/31103343 -local -silent -oh /oracle
/app/oracle/11.2.0.4/db_1 -invPtrLoc /oracle/app/oracle/11.2.0.4/db_1/oraInst.loc '
2021-06-04 22:10:18: Removing file /var/tmp/vaa027086
2021-06-04 22:10:18: Successfully removed file: /var/tmp/vaa027086
2021-06-04 22:10:18: /bin/su exited with rc=73

第四 解决过程

同时我们要注意在PSU打补丁步骤中，需要集群unlock和patch,我们需要注意

打补丁前：root 用户 /grid/app/11.2.0.4/grid/perl/bin/perl /grid/app/11.2.0.4/grid/crs/install/rootcrs.pl -unlock

grid或者Oracle 用户打相关的补丁

打补丁后：root /grid/app/11.2.0.4/grid/perl/bin/perl /grid/app/11.2.0.4/grid/crs/install/rootcrs.pl -patch

GI 的详细如下：

root@jkzcdb1:[/]/grid/app/11.2.0.4/grid/perl/bin/perl /grid/app/11.2.0.4/grid/crs/install/rootcrs.pl -unlock
Using configuration parameter file: /grid/app/11.2.0.4/grid/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'jkzcdb1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'jkzcdb1'
CRS-2675: Stop of 'ora.mdnsd' on 'jkzcdb1' failed
CRS-2679: Attempting to clean 'ora.mdnsd' on 'jkzcdb1'
CRS-2681: Clean of 'ora.mdnsd' on 'jkzcdb1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'jkzcdb1' has completed
CRS-4133: Oracle High Availability Services has been stopped.

jkzcdb1#[/oracle/software]su - grid
(c)Copyright 1983-2006 Hewlett-Packard Development Company, L.P.
(c)Copyright 1979, 1980, 1983, 1985-1993 The Regents of the Univ. of California
(c)Copyright 1980, 1984, 1986 Novell, Inc.
(c)Copyright 1986-2000 Sun Microsystems, Inc.
(c)Copyright 1985, 1986, 1988 Massachusetts Institute of Technology
(c)Copyright 1989-1993  The Open Software Foundation, Inc.
(c)Copyright 1990 Motorola, Inc.
(c)Copyright 1990, 1991, 1992 Cornell University
(c)Copyright 1989-1991 The University of Maryland
(c)Copyright 1988 Carnegie Mellon University
(c)Copyright 1991-2006 Mentat Inc.
(c)Copyright 1996 Morning Star Technologies, Inc.
(c)Copyright 1996 Progressive Systems, Inc.

Confidential computer software. Valid license from HP required for
possession, use or copying.  Consistent with FAR 12.211 and 12.212,
Commercial Computer Software, Computer Software Documentation, and
Technical Data for Commercial Items are licensed to the U.S. Government
under vendor's standard commercial license.
grid@jkzcdb1:[/home/grid]
grid@jkzcdb1:[/home/grid]410/31305209/31103343 -local -silent -oh /grid/app/11.2.0.4/grid -invPtrLoc /grid/app/11.2.0.4/grid/oraInst.loc                      <
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.


Oracle Home       : /grid/app/11.2.0.4/grid
Central Inventory : /grid/app/oraInventory
   from           : /grid/app/11.2.0.4/grid/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /grid/app/11.2.0.4/grid/cfgtoollogs/opatch/opatch2021-06-04_21-41-11PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   17478514  18031668  18522509  19121551  19769489  20299013  20760982  21352635  21948347  22502456  23054359  24006111  24732075  25869727  26609445  26392168  26925576  27338049  27734982  28204707  28729262  29141056  29497421  29913194  30298532  30670774  31103343  

Do you want to proceed? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/grid/app/11.2.0.4/grid')


Is the local system ready for patching? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
Backing up files...
Applying sub-patch '17478514' to OH '/grid/app/11.2.0.4/grid'
ApplySession: Optional component(s) [ oracle.sdo, 11.2.0.4.0 ] , [ oracle.sysman.agent, 10.2.0.4.5 ] , [ oracle.xdk, 11.2.0.4.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle

...........................................省略
Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...
chmod: invalid mode
Composite patch 31103343 successfully applied.
Log file location: /grid/app/11.2.0.4/grid/cfgtoollogs/opatch/opatch2021-06-04_21-41-11PM_1.log

OPatch succeeded.
grid@jkzcdb1:[/home/grid]

root@jkzcdb1:[/]/grid/app/11.2.0.4/grid/crs/install/rootcrs.pl -patch
Using configuration parameter file: /grid/app/11.2.0.4/grid/crs/install/crsconfig_params
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

第五 最终结果

grid@jkzcdb1:[/home/grid]opatch lspatches
31103343;Database Patch Set Update : 11.2.0.4.200714 (31103343)
29938455;OCW Patch Set Update : 11.2.0.4.191015 (29938455)
OPatch succeeded.

oracle@jkzcdb1:[/oracle/software/31326410/31219953]opatch lspatches
31219953;OJVM PATCH SET UPDATE 11.2.0.4.200714
31103343;Database Patch Set Update : 11.2.0.4.200714 (31103343)
29938455;OCW Patch Set Update : 11.2.0.4.191015 (29938455)

OPatch succeeded.

第六 补丁第一次failed 原因

root 用户下的LD_LIBARAY_PATH 环境变量问题导致！