1. 备份
　1.1 备份Oracle Inventory, CRS_HOME, ORACLE_HOME.　
　　　使用tar命令备份。Inventory的位置记录在文件/etc/oraInst.loc
  1.2 备份数据库
2. 升级CRS
  1.1 安装CRS升级软件
  1.2 CRS rolling upgrade
    用OUI升级CRS软件后，有如下提示：
    To complete the installation of this patch set, perform. the following tasks on each node:
　　1.Log in as the root user and enter the following command to shut down the Oracle Clusterware:
　　      # CRS_home/bin/crsctl stop crs　
　　2.Run the root102.sh script. to automatically start the Oracle Clusterware on the patched node:
      # CRS_home/install/root102.sh
      
  1.2.1 在节点1停止dbconsole/instance/asm/nodeapps/crs，然后执行root102.sh
    [root@10grac1 ~]# /app/oracle/product/crs/install/root102.sh
    Creating pre-patch directory for saving pre-patch clusterware files
    Completed patching clusterware files to /app/oracle/product/crs
    Relinking some shared libraries.
    Relinking of patched files is complete.
   
    WARNING: directory '/app/oracle/product' is not owned by root
    WARNING: directory '/app/oracle' is not owned by root
    Preparing to recopy patched init and RC scripts.
    Recopying init and RC scripts.
    Startup will be queued to init within 30 seconds.
    Starting up the CRS daemons.
    Waiting for the patched CRS daemons to start.
      This may take a while on some systems.
    .
    .
    .
    .
    10204 patch successfully applied.
    clscfg: EXISTING configuration version 3 detected.
    clscfg: version 3 is 10G Release 2.
    Successfully accumulated necessary OCR keys.
    Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
    node :
    node 1: 10grac1 10grac1_prv 10grac1
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    clscfg -upgrade completed successfully
  节点1升级成功后检查crs
    [root@10grac1 ~]# crsctl check crs
    CSS appears healthy
    CRS appears healthy
    EVM appears healthy
  在节点1上检查crs的版本：
    [root@10grac1 ~]# crsctl query crs softwareversion
    CRS software version on node [10grac1] is [10.2.0.4.0]
    [root@10grac1 ~]# crsctl query crs activeversion
    CRS active version on the cluster is [10.2.0.1.0]　(activeversion版本仍为10.2.0.1)
  检查节点1上的资源：(执行root102.sh后,CRS及实例均重启动了)
    [root@10grac1 ~]#  crs_stat -t -c 10grac1
    Name           Type           Target    State     Host
    ------------------------------------------------------------
    ora....ac1.vip application    ONLINE    ONLINE    10grac1
    ora....C1.lsnr application    ONLINE    ONLINE    10grac1
    ora....SM1.asm application    ONLINE    ONLINE    10grac1
    ora....ac1.gsd application    ONLINE    ONLINE    10grac1
    ora....ac1.ons application    ONLINE    ONLINE    10grac1
    ora.ergdb.db   application    ONLINE    ONLINE    10grac1
    ora....b1.inst application    ONLINE    ONLINE    10grac1

  1.2.2　在节点2停止instance/asm/nodeapps/crs，然后执行root102.sh                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               [root@10grac2 ~]# /app/oracle/product/crs/install/root102.sh
    Creating pre-patch directory for saving pre-patch clusterware files
    Completed patching clusterware files to /app/oracle/product/crs
    Relinking some shared libraries.
    Relinking of patched files is complete.
    WARNING: directory '/app/oracle/product' is not owned by root
    WARNING: directory '/app/oracle' is not owned by root
    Preparing to recopy patched init and RC scripts.
    Recopying init and RC scripts.
    Startup will be queued to init within 30 seconds.
    Starting up the CRS daemons.
    Waiting for the patched CRS daemons to start.
      This may take a while on some systems.
    .
    .
    10204 patch successfully applied.
    clscfg: EXISTING configuration version 3 detected.
    clscfg: version 3 is 10G Release 2.
    Successfully accumulated necessary OCR keys.
    Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
    node :
    node 2: 10grac2 10grac2_prv 10grac2
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    clscfg -upgrade completed successfully
   
    [root@10grac2 ~]# crsctl check crs
    CSS appears healthy
    CRS appears healthy
    EVM appears healthy
   
    [root@10grac2 ~]# crsctl query crs softwareversion
    CRS software version on node [10grac2] is [10.2.0.4.0]
    [root@10grac2 ~]# crsctl query crs activeversion
    CRS active version on the cluster is [10.2.0.4.0](节点2升级后activeversion变为10.2.0.4)
    [root@10grac1 ~]# crsctl query crs activeversion
    CRS active version on the cluster is [10.2.0.4.0](节点1的activeversion也变为10.2.0.4)
   
    [root@10grac2 ~]# crs_stat -t -c 10grac2
    Name           Type           Target    State     Host
    ------------------------------------------------------------
    ora....SM2.asm application    ONLINE    ONLINE    10grac2
    ora....C2.lsnr application    ONLINE    ONLINE    10grac2
    ora....ac2.gsd application    ONLINE    ONLINE    10grac2
    ora....ac2.ons application    ONLINE    ONLINE    10grac2
    ora....ac2.vip application    ONLINE    ONLINE    10grac2
    ora....b2.inst application    ONLINE    ONLINE    10grac2

3. 升级数据库
　3.1 安装Oracle升级软件,安装前停止所有dbconsole/instance/asm/nodeapps
　3.2 升级数据库
　　在节点1上运行Pre-Upgrade Information Tool
　　$ sqlplus "/as sysdba"
    SQL> STARTUP NOMOUNT
    SQL> ALTER SYSTEM SET CLUSTER_DATABASE=FALSE SCOPE=spfile;
    SQL> SHUTDOWN
    SQL> STARTUP UPGRADE
    SQL> SPOOL upgrade_info.log 　
　　SQL> @?/rdbms/admin/utlu102i.sql
　　SQL> SPOOL OFF
　　然后检查upgrade_info.log.
    在节点1上升级数据库：
    SQL> SPOOL patch.log
　　SQL> @?/rdbms/admin/catupgrd.sql
    SQL> SPOOL OFF
    SQL> SHUTDOWN IMMEDIATE
    SQL> STARTUP
    SQL> @?/rdbms/admin/utlrp.sql
    检查升级后的状态：
    SQL> select comp_name, version, status from sys.dba_registry;
    改为cluster方式：
    SQL> ALTER SYSTEM SET CLUSTER_DATABASE=TRUE SCOPE=spfile;
    SQL> SHUTDOWN
　　SQL> STARTUP
  3.3　配置Enterprise Manager
  　保证database和listener是启动的,然后执行emca -upgrade db -cluster配置
  　[oracle@10grac1 ~]$ emca -upgrade db -cluster
    STARTED EMCA at Nov 30, 2008 12:23:34 AM
    EM Configuration Assistant, Version 10.2.0.1.0 Production
    Copyright (c) 2003, 2005, Oracle.  All rights reserved.
   
    Enter the following information:
    ORACLE_HOME for the database to be upgraded: /app/oracle/product/10g
    Database unique name: ergdb
    Listener port number: 1521
   
    Do you wish to continue? [yes(Y)/no(N)]: y
    Nov 30, 2008 12:24:20 AM oracle.sysman.emcp.EMConfig perform
    INFO: This operation is being logged at /app/oracle/product/10g/cfgtoollogs/emca/ergdb/emca_2008-11-30_12-23-34-AM.log.
    Nov 30, 2008 12:24:39 AM oracle.sysman.emcp.util.DBControlUtil stopOMS
    INFO: Stopping Database Control (this may take a while) ...
    Nov 30, 2008 12:25:11 AM oracle.sysman.emcp.EMDBCConfig instantiateOC4JConfigFiles
    INFO: Propagating /app/oracle/product/10g/oc4j/j2ee/OC4J_DBConsole_10grac1_ergdb1 to remote nodes ...
    Nov 30, 2008 12:25:13 AM oracle.sysman.emcp.EMDBCConfig instantiateOC4JConfigFiles
    INFO: Propagating /app/oracle/product/10g/oc4j/j2ee/OC4J_DBConsole_10grac2_ergdb2 to remote nodes ...
   
    Nov 30, 2008 12:28:41 AM oracle.sysman.emcp.util.DBControlUtil secureDBConsole
    INFO: Securing Database Control (this may take a while) ...
    Nov 30, 2008 12:30:28 AM oracle.sysman.emcp.util.DBControlUtil startOMS
    INFO: Starting Database Control (this may take a while) ...
    Nov 30, 2008 12:33:38 AM oracle.sysman.emcp.EMDBPostConfig performUpgrade
    INFO: Database Control started successfully
    Nov 30, 2008 12:33:38 AM oracle.sysman.emcp.EMDBPostConfig performUpgrade
    INFO: >>>>>>>>>>> The Database Control URL is https://10grac1:1158/em <<<<<<<<<<<
    Nov 30, 2008 12:33:40 AM oracle.sysman.emcp.EMDBPostConfig showClusterDBCAgentMessage
    INFO:
    ****************  Current Configuration  ****************
     INSTANCE            NODE           DBCONTROL_UPLOAD_HOST
    ----------        ----------        ---------------------
   
    ergdb1            10grac1           10grac1
    ergdb2            10grac2           10grac1
   
   
    Enterprise Manager configuration completed successfully
    FINISHED EMCA at Nov 30, 2008 12:33:40 AM

4. 数据库降级
　4.1 关闭所有节点的数据库
  4.2 在节点1上执行：
　　$ sqlplus "/as sysdba"
    SQL> STARTUP NOMOUNT
    SQL> ALTER SYSTEM SET CLUSTER_DATABASE=FALSE SCOPE=spfile;
    SQL> SHUTDOWN
  4.3 备份以下文件(10.2.0.4版本)
  $ ORACLE_HOME/rdbms/admin/catrelod.sql
　$ ORACLE_HOME/network/admin/tnsnames.ora
　4.4 在节点1上执行：
　$ sqlplus / as sysdba
  SQL> STARTUP DOWNGRADE
  SQL> SPOOL catdwgrd.out
  SQL> @?/rdbms/admin/catdwgrd.sql
  SQL> SPOOL OFF
  SQL> SHUTDOWN IMMEDIATE
  SQL> EXIT
  4.5 恢复备份的Oracle Inventory和Oracle Home.   
  4.6 把10.2.0.4版本的catrelod.sql和tnsnames.ora文件copy回原处。
  4.7 在节点1上执行：
  $ sqlplus / as sysdba               
  SQL> STARTUP DOWNGRADE           
  SQL> SPOOL catrelod.out          
  SQL> @?/rdbms/admin/catrelod.sql 
  SQL> SPOOL OFF                   
  SQL> SHUTDOWN IMMEDIATE
  SQL> STARTUP
  SQL> @?/rdbms/admin/utlrp.sql
  4.8  改为cluster模式
  SQL> ALTER SYSTEM SET CLUSTER_DATABASE=TRUE SCOPE=spfile;
　SQL> SHUTDOWN
　SQL> EXIT
  4.9 在所有节点重启实例
5. CRS降级
　10.2.0.4不支持CRS降级,如果确实需要降级，以下方式应该是可行的：
　1. 恢复备份的Oracle Inventory和Oracle CRS_HOME.
  2. 恢复备份的OCR和VOTE DISK
  3. 恢复备份的相关文件(升级前最好备份一下)
   linux应该包括以下内容：
   /etc/oracle
   /etc/init.d/init.cssd
   /etc/init.d/init.crs
   /etc/init.d/init.crsd
   /etc/init.d/init.evmd
   /etc/inittab