一、正常流程
环境： CentOS 7.6 Oracle 19c RAC
补丁工具： p6880880_190000_Linux-x86-64.zip
补丁集： p31326369_190000_Linux-x86-64.zip 包括 GI 、 DB 、 OJVM
1. 前期准备
1.1 清理 trace 、 aud 文件等
1.2 关闭集群（注意两个节点可以同时进行 1.3 ）
#/u01/app/19.0.0/grid/bin/crsctl stop crs
1.3 备份 /u01
#tar cvpzf /root/u01bak.tar.gz /u01
1.4 启动集群
#/u01/app/19.0.0/grid/bin/crsctl start crs
2.grid 安装 OPatch
#mv /u01/app/19.0.0/grid/OPatch /u01/app/19.0.0/grid/OPatch_bak_20201010
#unzip /software/p6880880_190000_Linux-x86-64.zip -d /u01/app/19.0.0/grid
#chown -R grid:oinstall /u01/app/19.0.0/grid/OPatch
#/u01/app/19.0.0/grid/OPatch/opatch version
3.oracle 安装 OPatch
# mv /u01/app/oracle/product/19.0.0/dbhome_1/OPatch /u01/app/oracle/product/19.0.0/dbhome_1/OPatch_bak_20201010
#unzip /software/p6880880_190000_Linux-x86-64.zip -d /u01/app/oracle/product/19.0.0/dbhome_1
#chown -R oracle:oinstall /u01/app/oracle/product/19.0.0/dbhome_1/OPatch
#/u01/app/oracle/product/19.0.0/dbhome_1/OPatch/opatch version
4. 解压补丁包
#unzip /software/p31326369_190000_Linux-x86-64.zip -d /tmp
#chown -R grid:oinstall /tmp/31326369
#chmod -R 777 /tmp/31326369
5. 验证补丁包
#/u01/app/19.0.0/grid/OPatch/opatch version
#/u01/app/oracle/product/19.0.0/dbhome_1/OPatch/opatch version
#/u01/app/19.0.0/grid/OPatch/opatch lsinventory -detail -oh /u01/app/19.0.0/grid/
#su - grid
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /tmp/31326369/31305339/31281355
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /tmp/31326369/31305339/31305087
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /tmp/31326369/31305339/31304218
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir
$vi /tmp/patch_list_gihome.txt
/tmp/31326369/31305339/31281355
/tmp/31326369/31305339/31305087
/tmp/31326369/31305339/31304218
/tmp/31326369/31305339/31335188
$ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_gihome.txt
su - oracle
$vi /tmp/patch_list_dbhome.txt
/tmp/31326369/31305339/31281355
/tmp/31326369/31305339/31305087
$ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_dbhome.txt
6. 安装 GI 和 DB 补丁包（所有节点，滚动发车不能并行）
su - root
#/u01/app/19.0.0/grid/OPatch/opatchauto apply /tmp/31326369/31305339 -oh /u01/app/19.0.0/grid
OPatchauto session is initiated at Sat Oct 10 22:37:02 2020
System initialization log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-10_10-37-06PM.log.
Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-10_10-37-33PM.log
The id for this session is 2K2Q
Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.0.0/grid
Patch applicability verified successfully on home /u01/app/19.0.0/grid
Bringing down CRS service on home /u01/app/19.0.0/grid
CRS service brought down successfully on home /u01/app/19.0.0/grid
Start applying binary patch on home /u01/app/19.0.0/grid
Binary patch applied successfully on home /u01/app/19.0.0/grid
Starting CRS service on home /u01/app/19.0.0/grid
CRS service started successfully on home /u01/app/19.0.0/grid
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:yxdb01
CRS Home:/u01/app/19.0.0/grid
Version:19.0.0.0.0
Summary:
==Following patches were SUCCESSFULLY applied:
Patch: /tmp/31326369/31219897
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-10_22-42-06PM_1.log
OPatchauto session completed at Sat Oct 10 22:47:48 2020
Time taken to complete the session 10 minutes, 46 seconds
su - grid
$/u01/app/19.0.0/grid/OPatch/opatch lspatches
31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)
31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
31281355;Database Release Update : 19.8.0.0.200714 (31281355)
这里的 TOMCAT 可以不甩
#/u01/app/oracle/product/19.0.0/dbhome_1/OPatch/opatchauto apply /tmp/31326369/31305339 -oh /u01/app/oracle/product/19.0.0/dbhome_1
OPatchauto session is initiated at Sun Oct 11 00:23:03 2020
System initialization log file is /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchautodb/systemconfig2020-10-11_12-23-07AM.log.
Session log file is /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto2020-10-11_12-23-24AM.log
The id for this session is XD76
Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0/dbhome_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/dbhome_1
Verifying SQL patch applicability on home /u01/app/oracle/product/19.0.0/dbhome_1
"/bin/sh -c 'cd /u01/app/oracle/product/19.0.0/dbhome_1; ORACLE_HOME=/u01/app/oracle/product/19.0.0/dbhome_1 ORACLE_SID=cdbyx2 /u01/app/oracle/product/19.0.0/dbhome_1/OPatch/datapatch -prereq -verbose'" command failed with errors. Please refer to logs for more details. SQL changes, if any, can be analyzed by manually retrying the same command.
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/dbhome_1
Preparing to bring down database service on home /u01/app/oracle/product/19.0.0/dbhome_1
Successfully prepared home /u01/app/oracle/product/19.0.0/dbhome_1 to bring down database service
Bringing down database service on home /u01/app/oracle/product/19.0.0/dbhome_1
Following database(s) and/or service(s) are stopped and will be restarted later during the session: cdbyx
Database service successfully brought down on home /u01/app/oracle/product/19.0.0/dbhome_1
Performing prepatch operation on home /u01/app/oracle/product/19.0.0/dbhome_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/dbhome_1
Start applying binary patch on home /u01/app/oracle/product/19.0.0/dbhome_1
Binary patch applied successfully on home /u01/app/oracle/product/19.0.0/dbhome_1
Performing postpatch operation on home /u01/app/oracle/product/19.0.0/dbhome_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/dbhome_1
Starting database service on home /u01/app/oracle/product/19.0.0/dbhome_1
Database service successfully started on home /u01/app/oracle/product/19.0.0/dbhome_1
Preparing home /u01/app/oracle/product/19.0.0/dbhome_1 after database service restarted
No step execution required.........
Trying to apply SQL patch on home /u01/app/oracle/product/19.0.0/dbhome_1
SQL patch applied successfully on home /u01/app/oracle/product/19.0.0/dbhome_1
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:yxdb02
RAC Home:/u01/app/oracle/product/19.0.0/dbhome_1
Version:19.0.0.0.0
Summary:
==Following patches were SKIPPED:
Patch: /tmp/31326369/31305339/31304218
Reason: This patch is not applicable to this specified target type - "rac_database"
Patch: /tmp/31326369/31305339/31335188
Reason: This patch is not applicable to this specified target type - "rac_database"
==Following patches were SUCCESSFULLY applied:
Patch: /tmp/31326369/31305339/31281355
Log: /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-11_00-27-26AM_1.log
Patch: /tmp/31326369/31305339/31305087
Log: /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-11_00-27-26AM_1.log
说明：黄色的两个是 GI 的所以在 oracle 下是没有 apply 的，这是正常的。
OPatchauto session completed at Sun Oct 11 00:36:31 2020
Time taken to complete the session 13 minutes, 28 seconds
su - oracle
$/u01/app/oracle/product/19.0.0/dbhome_1/OPatch/opatch lspatches
$ /u01/app/oracle/product/19.0.0/dbhome_1/OPatch/opatch lspatches
31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
31281355;Database Release Update : 19.8.0.0.200714 (31281355)
30128191;OJVM RELEASE UPDATE: 19.5.0.0.191015 (30128191)
上面看到 OJVM 还没有 UPDATE ，此时可以不管，到第二节点进行 GI 和 DB 的补丁包更新。
7. 所有节点都完成 GI 和 DB 的补丁之后在任意节点注册补丁
sqlplus /nolog
Connect / as sysdba
startup
alter pluggable database all open;
cd $ORACLE_HOME/OPatch
[oracle@yxdb01 OPatch]$./datapatch -verbose
SQL Patching tool version 19.8.0.0.0 Production on Sun Oct 11 01:02:15 2020
Copyright (c) 2012, 2020, Oracle. All rights reserved.
Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_86605_2020_10_11_01_02_15/sqlpatch_invocation.log
Connecting to database...OK
Gathering database info...done
Note: Datapatch will only apply or rollback SQL fixes for PDBs
that are in an open state, no patches will be applied to closed PDBs.
Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
(Doc ID 1585822.1)
Bootstrapping registry and package to current versions...done
Determining current state...done
Current state of interim SQL patches:
Interim patch 30128191 (OJVM RELEASE UPDATE: 19.5.0.0.191015 (30128191)):
Binary registry: Not installed on nodes yxdb01
PDB CDB$ROOT: Applied successfully on 06-JAN-20 06.04.44.845179 PM
PDB PDB$SEED: Applied successfully on 06-JAN-20 06.14.30.418488 PM
PDB PDBTEST: Applied successfully on 06-JAN-20 06.14.30.418488 PM
PDB PDBYX: Applied successfully on 06-JAN-20 06.14.30.418488 PM
PDB PDBYXBAK: Applied successfully on 06-JAN-20 06.14.30.418488 PM
Current state of release update SQL patches:
Binary registry:
19.8.0.0.0 Release_Update 200703031501: Installed
PDB CDB$ROOT:
Applied 19.8.0.0.0 Release_Update 200703031501 successfully on 11-OCT-20 12.02.38.348777 AM
PDB PDB$SEED:
Applied 19.8.0.0.0 Release_Update 200703031501 successfully on 11-OCT-20 12.02.39.978390 AM
PDB PDBTEST:
Applied 19.8.0.0.0 Release_Update 200703031501 successfully on 11-OCT-20 12.02.44.884611 AM
PDB PDBYX:
Applied 19.8.0.0.0 Release_Update 200703031501 successfully on 11-OCT-20 12.02.41.616556 AM
PDB PDBYXBAK:
Applied 19.8.0.0.0 Release_Update 200703031501 successfully on 11-OCT-20 12.02.43.255669 AM
Adding patches to installation queue and performing prereq checks...done
Installation queue:
For the following PDBs: CDB$ROOT PDB$SEED PDBYX PDBYXBAK PDBTEST
No interim patches need to be rolled back
No release update patches need to be installed
No interim patches need to be applied
SQL Patching tool complete on Sun Oct 11 01:04:16 2020
8. 跑 utlrp.sql
cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
CONNECT / AS SYSDBA
@utlrp.sql
9. 查看组件状态和失效对象
SELECT COMP_ID,COMP_NAME,VERSION,STATUS from DBA_REGISTRY;
SELECT COUNT(*) FROM DBA_OBJECTS WHERE STATUS= ’ INVALID ’ ;
10.OJVM 补丁（所有节点）
正常操作：
su - oracle
$cd /tmp/31326369/31219897
$/u01/app/oracle/product/19.0.0/dbhome_1/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./
$cd /tmp/31326369/31219897
$/u01/app/oracle/product/19.0.0/dbhome_1/OPatch/opatch apply
野路子：
/u01/app/oracle/product/19.0.0/dbhome_1/OPatch/opatchauto apply /tmp/31326369/31219897 -oh /u01/app/oracle/product/19.0.0/dbhome_1
说明： README 里面是正常操作，野路子在 19.5 上升 19.8 也是成功了。
11. 附回滚操作
#/u01/app/19.3.0/grid/OPatch/opatchauto rollback -id 31142749
#/u01/app/oracle/product/19.3.0/db_1/OPatch/opatchauto rollback -id 31142749
二、打补丁过程中遇到的错误
1. 预检查
[grid@yxdb01 OPatch]$ ./opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /tmp/31326369/31305339
Oracle Interim Patch Installer version 12.2.0.1.21
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /u01/app/19.0.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/19.0.0/grid/oraInst.loc
OPatch version : 12.2.0.1.21
OUI version : 12.2.0.7.0
Log file location : /u01/app/19.0.0/grid/cfgtoollogs/opatch/opatch2020-10-10_21-48-41PM_1.log
This command doesn't support System Patch.
OPatch failed with error code 21
原因： 31305339 是 OJVM 的补丁，需要在 oracle 用户下
2.GI APPLY
# /u01/app/19.0.0/grid/OPatch/opatchauto apply /tmp/31326369/31219897/
OPatchauto session is initiated at Sat Oct 10 22:22:25 2020
System initialization log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-10_10-22-32PM.log.
Clusterware is either not running or not configured. You have the following 2 options:
1. Configure and start the Clusterware on this node and re-run the tool
2. Run the tool with '-oh <GI_HOME>' to first patch the Grid Home, then invoke tool with '-database <oracle database name>' or '-oh <RAC_HOME>' to patch the RAC home
OPATCHAUTO-72029: CLusterware home not configured.
OPATCHAUTO-72029: Clusterware is either not running or not configured or cluster is software only GI
OPATCHAUTO-72029: If only Grid Infrastructure software is installed, please run opatchauto with '-oh' option . Alternatively configure and start the Clusterware before running opatchauto.
OPatchauto session completed at Sat Oct 10 22:22:38 2020
Time taken to complete the session 0 minute, 13 seconds
Topology creation failed.
这里有两个错误：
1 ） 31219897 是 OJVM 的补丁，要在 oracle 下面打
2 ）语句不全，应为
#/u01/app/19.0.0/grid/OPatch/opatchauto apply /tmp/31326369/31305339/ -oh /u01/app/19.0.0/grid
3. 集群相关
#/u01/app/19.0.0/grid/OPatch/opatchauto apply /tmp/31326369/31219897/ -oh /u01/app/19.0.0/grid
OPatchauto session is initiated at Sat Oct 10 22:29:13 2020
System initialization log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-10-10_10-29-16PM.log.
Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-10-10_10-29-21PM.log
The id for this session is YJ4N
OPATCHAUTO-72132: Grid is not running on the local host.
OPATCHAUTO-72132: Cannot start a new apply or rollback session when the local grid is not running.
OPATCHAUTO-72132: Please start grid service on the local host to start patching.
OPatchAuto failed.
OPatchauto session completed at Sat Oct 10 22:29:28 2020
Time taken to complete the session 0 minute, 16 seconds
opatchauto failed with error code 42
原因：这里是因为在备份 /u01 的时候，集群停掉没有启动的原因（在打补丁的过程中会自动停集群）
解决方法： #./crsctl start crs
4. 补丁冲突
# /u01/app/oracle/product/19.0.0/dbhome_1/OPatch/opatchauto apply /tmp/31326369/31305339 -oh /u01/app/oracle/product/19.0.0/dbhome_1
OPatchauto session is initiated at Sat Oct 10 23:15:47 2020
System initialization log file is /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchautodb/systemconfig2020-10-10_11-15-50PM.log.
Session log file is /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto2020-10-10_11-16-26PM.log
The id for this session is 8U49
Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0/dbhome_1
Patch applicability verification failed on home /u01/app/oracle/product/19.0.0/dbhome_1
Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : yxdb01->/u01/app/oracle/product/19.0.0/dbhome_1 Type[rac]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/oracle/product/19.0.0/dbhome_1, host: yxdb01.
Command failed: /u01/app/oracle/product/19.0.0/dbhome_1/OPatch/opatchauto apply /tmp/31326369/31305339 -oh /u01/app/oracle/product/19.0.0/dbhome_1 -target_type rac_database -binary -invPtrLoc /u01/app/oracle/product/19.0.0/dbhome_1/oraInst.loc -jre /u01/app/oracle/product/19.0.0/dbhome_1/OPatch/jre -persistresult /u01/app/oracle/product/19.0.0/dbhome_1/opatchautocfg/db/sessioninfo/sessionresult_analyze_yxdb01_rac_2.ser -analyze -online -prepare_home
Command failure output:
==Following patches FAILED in analysis for apply:
Patch: /tmp/31326369/31305339/31281355
Log: /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-10_23-16-40PM_1.log
Reason: Failed during Analysis: CheckConflictAgainstOracleHome Failed, [ Prerequisite Status: FAILED, Prerequisite output:
The details are:
Please rebuild the superset patch [31281355] to make sure it supersedes all the relevant patch(es) [30125133,30321076].
The rebuild patch should contain bug fix [29332763, 30169254, 29766207, 29947145, 28917080, 30834110, 29502561, 29198913, 29968085, 29293072, 29351044, 29376346, 29516727, 29483723, 28774416, 30790441, 29844275, 30539519, 30443393, 29766679, 30147 ...
After fixing the cause of failure Run opatchauto resume
]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.
OPatchauto session completed at Sat Oct 10 23:16:54 2020
Time taken to complete the session 1 minute, 8 seconds
opatchauto failed with error code 42
原因：因为之前这个节点上打过一个 30321076 的补丁，在检查验证的时候并没有报错，这里是由于安装的时候检测到节点 2 并没有安装这个补丁包，导致两边节点补丁包不一致，所以报错。
解决方法：在节点 1 回滚这个补丁包。
# /u01/app/oracle/product/19.3.0/db_1/OPatch/opatch lspatches
#/u01/app/oracle/product/19.3.0/db_1/OPatch/opatchauto rollback -id 30321076
5. 由于存在失效对象、组件 INVALID ， upgrade 遇到的问题
SQL> startup upgrade
ORACLE instance started.
Total System Global Area 2.1475E+11 bytes
Fixed Size 30389024 bytes
Variable Size 1.5730E+11 bytes
Database Buffers 5.6908E+10 bytes
Redo Buffers 506478592 bytes
Database mounted.
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-39701: database must be mounted EXCLUSIVE for UPGRADE or DOWNGRADE
Process ID: 230057
Session ID: 1122 Serial number: 12093
解决方法：
On only one node, execute the following steps to start an Oracle RAC database in startup upgrade mode:
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP
SQL> alter system set cluster_database=false scope=spfile;
$ORACLE_HOME/bin/srvctl stop database -d <dbname>
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP UPGRADE
来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/30654353/viewspace-2726147/，如需转载，请注明出处，否则将追究法律责任。
