SUSE 11.1 安装ORACLE 11.2.0.3 ASM非RAC双机过程纪要

增加用户和组：
/usr/sbin/groupadd -g 501 oinstall
/usr/sbin/groupadd -g 502 dba
/usr/sbin/groupadd -g 503 oper
/usr/sbin/groupadd -g 504 asmadmin
/usr/sbin/groupadd -g 505 asmoper
/usr/sbin/groupadd -g 506 asmdba
useradd -u 502 -g oinstall -G dba,asmdba,oper oracle
useradd -u 501 -g oinstall -G asmadmin,asmdba,asmoper,oper,dba grid

建立目录：
mkdir –p /home/app/
chown -R grid:oinstall /home/app/
chmod -R 775 /home/app/
mkdir -p /home/app/oraInventory
chown -R grid:oinstall /home/app/oraInventory
chmod -R 775 /home/app/oraInventory
mkdir -p /home/app/grid
mkdir -p /home/app/oracle
chown -R grid:oinstall /home/app/grid
chown -R oracle:oinstall /home/app/oracle
chmod -R 775 /home/app/oracle
chmod -R 775 /home/app/grid

修改GRID环境变量
ORACLE_SID=+ASM; export ORACLE_SID
ORACLE_BASE=/home/app; export ORACLE_BASE
ORACLE_HOME=/home/app/grid; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$PATH; export PATH
export ORA_CRS_HOME=/home/app/grid   ---这一行SUSE下面必须加

需要安装的包
 ksh-93t libaio-0.3.104 libaio-32bit-0.3.104 libaio-devel-0.3.104 libaio-devel-32bit-0.3.104 libstdc++33-3.3.3 libstdc++33-32bit-3.3.3 libstdc++43-4.3.3_20081022 libstdc++43-32bit-4.3.3_20081022 libstdc++43-devel-4.3.3_20081022 libstdc++43-devel-32bit-4.3.3_20081022 libgcc43-4.3.3_20081022 libstdc++-devel-4.3 make-3.81 sysstat-8.1.5

rpm -q binutils ksh  gcc gcc-c++ glibc-32bit glibc- glibc-devel-  glibc-devel-32bit  libaio- libaio-32bit- libelf- libgcc- libstdc++ libstdc++-devel make- sysstat- --qf '%{name}.%{arch} '|sort

添加裸设备
/dev/raw
raw1:sdb
raw2:sdc
raw3:sdd
raw4:sde
raw5:sdf
raw6:sdg
raw7:sdh
raw8:sdi
raw9:sdj
raw10:sdk
raw11:sdl

rcraw start 绑定设备。
然后chkconfig raw   on 开启开机自动绑定。
设置裸设备权限
11.1和10.3 50-udev-default.rules文件位置不一样，如下：
/lib/udev/rules.d/50-udev-default.rules
改动
KERNEL=="rawctl",               NAME="raw/%k", GROUP="disk"
SUBSYSTEM=="raw", KERNEL=="raw[0-9]*", NAME="raw/%k", GROUP="dba", MODE="660", OWNER="grid"
修改限制
/etc/security/limits.conf
oracle  soft    nproc   2047
oracle  hard    nproc   16384
oracle  soft    nofile  1024
oracle  hard    nofile  65536
grid  soft    nproc   2047
grid  hard    nproc   16384
grid  soft    nofile  1024
grid  hard    nofile  65536

修改用户限制
/etc/profile
if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
        if [ $SHELL = "/bin/ksh" ]; then
              ulimit -u 16384
              ulimit -n 65536
        else
              ulimit -u 16384 -n 65536
        fi
        umask 022
fi

增加shm 
/etc/fstab 加入一行
tmpfs                /dev/shm    tmpfs   defaults,size=11264m 0 0
mount -o remount /dev/shm
重新MOUNT一下

修改SYSCTL.CONF核心参数如下：
kernel.shmall = 4194304
kernel.shmmax = 17179869184
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6815744
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 4194304
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
fs.aio-max-nr = 1048576

需要加入
/etc/pam.d/login
session    required     pam_limits.so

规划DG
/DATA1 4T
/DATA2 4T
/ARCH  3T

根据文档进行建立How To Setup ASM (11.2) On An Active/Passive Cluster (Non-RAC). (Doc ID 1296124.1)

安装ORACLE软件

ORACLE_SID=ismpdb; export ORACLE_SID
ORACLE_BASE=/home/app; export ORACLE_BASE
ORACLE_HOME=/home/app/oracle; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$PATH; export PATH
export ORA_CRS_HOME=/home/app/grid  ----这一行SUSE必须添加否者报错

PRCR-1006 : 无法添加 ismpdb 的资源 ora.ismpdb.db
PRCR-1118 : 无法生成资源的资源权限
PRKH-1013 : Oracle 主目录 /opt/oracle/db/product/11.2.0/crs 不存在。

安装过程中遇到错误
Installing 11gR2 Fails With [INS-30060] Check For Group Existence Failed (Doc ID 1268391.1)
原因是他们以前安装过ORACLE软件其/TMP下的CUV目录权限和用户组不对

srvctl add database -d ismpdb -n ismpdb -o /home/app/oracle -p +DATA1/ISMPDB/spfileismpdb.ora -s OPEN -y AUTOMATIC -a DATA1,DATA2,ARCH -t IMMEDIATE
启动数据库srvctl start database -d ismpdb
报错
PRCR-1079 : Failed to start resource ora.t11gr2.db
CRS-5010: Update of configuration file "/hostname/app/oracle/product/11.2.0/server_2/srvm/admin/oratab.bak.orcldevbal882" failed: 
details at "(:CLSN00011:)" in "/orcldevbal882/app/grid/product/11.2.0.2/grid/log/orcldevbal882/agent/ohasd/oraagent_grid/oraagent_grid.log"
CRS-5010: Update of configuration file "/hostname/app/oracle/product/11.2.0/server_2/dbs/initt11gR2.ora" failed: 
details at "(:CLSN00014:)" in "/orcldevbal882/app/grid/product/11.2.0.2/grid/log/orcldevbal882/agent/ohasd/oraagent_grid/oraagent_grid.log"
CRS-5017: The resource action "ora.t11gr2.db start" encountered the following error:
CRS-5010: Update of configuration file "/hostname/app/oracle/product/11.2.0/server_2/dbs/initt11gR2.ora" failed: 
details at "(:CLSN00014:)" in "/orcldevbal882/app/grid/product/11.2.0.2/grid/log/orcldevbal882/agent/ohasd/oraagent_grid/oraagent_grid.log"
解决
http://hi.baidu.com/wuzhanyin/item/c5bbcdfa7ecfd6d742c36aec
其实就是权限问题。

启动LISTENER报错
TNS-12555: TNS:permission denied
TNS-12560: TNS:protocol adapter error
  TNS-00525: Insufficient privilege for operation
   Linux Error: 1: Operation not permitted

 Listener Fails To Start With IPC Permission Errors (TNS-12546 TNS-12555 TNS-00516 TNS-00525) (Doc ID 434062.1)
 启动LISTENER报了一个权限不足的错误，这个由文章可以看出可能是由于不同的用户启动了LISTENER导致/TMP/.ORACLE里面的运行文件权限不对。
 文档建议删除，但是同时这个操作会导致RAC 的cluster 以及单实例ASM的环境崩溃，只能重启服务器
 所以建议如果出现这样的问题一定要关闭数据库，关闭CRS HAS后再删除，然后重启服务器，不能冒昧的直接删除。

配置浮动IP的时候注意BUG，如果添加一个IP

192.168.190.8是我手动加入的
LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
    )
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = tasm)(PORT = 1521))
      (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.190.8)(PORT = 1521))
    )
  )
报错：
TNS-12542: TNS:address already in use
TNS-12560: TNS:protocol adapter error
TNS-00512: Address already in use
Linux Error: 98: Address already in use
需要修改tasm主机名为IP
LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
    )
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.190.4)(PORT = 1521))
      (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.190.8)(PORT = 1521))
    )
  )

参考：
Listener Fails to Start on Linux, Errors with TNS-12542, Linux Error: 98: Address already in use (Doc ID 301014.1)

安装GRID AND DATABASE PSU
下载p17272731_112030_Linux-x86-64.zip，这个文件包含了GID PSU和DATABASE PSU 我这个时候是11.2.0.3.8

按照要求下载最新的OPatch 6880880 
解压
然后使用命令
export JAVA_HOME=$ORACLE_HOME/jdk  -grid
/home/grid/OPatch/ocm/bin/emocmrsp  -grid
生成文件ocm.rsp 
验证GRID和DATABASE 的HOME目录可用

/home/grid/OPatch/opatch lsinventory -detail -oh /home/app/grid -grid 
chmod 775 /home/app/oracle/cfgtoollogs  注意这个目录需要赋权，不然不能执行 -root
/home/grid/OPatch/opatch lsinventory -detail -oh /home/app/oracle  -grid

使用root用户解压
 unzip p17272731_112030_.zip
停止所有ORCALE进程CRS DATABASE
 然后使用root用户
 opatch auto -ocmrf
/home/grid/OPatch/opatch auto /soft/gidpsu  -ocmrf /home/grid/ocm.rsp
我这里报错
Using configuration parameter file: /opt/app/11.2.0/grid/crs/install/crsconfig_params
Either does not exist or is not readable
Make sure the file exists and it has read and execute access
Clusterware home location does not exist
说明
opatch auto fails: Clusterware home location does not exist (Doc ID 1567205.1)
At the time of this writing, the bug is still being worked by Development.
The workaround is to apply the patch manually.
The manual instruction is part of patch readme, alternatively refer to the following for general instruction:
note 1494646.1 - Readme - Patch Installation and Deinstallation For 11.2.0.3.x GI PSU
只能手动进行升级安装
参考
Readme - Patch Installation and Deinstallation For 11.2.0.3.x GI PSU (Doc ID 1494646.1)
第一步停止有所的ORACLE进程
第二步 root用户
/home/app/grid/crs/install/roothas.pl -unlock  
第三步 grid用户
 /OPatch/opatch napply -oh -local /
 /home/grid/OPatch/opatch napply -oh  /home/app/grid -local /soft/gidpsu/17076717 
 报错
 Copy Action: Source File "/soft/gidpsu/17076717/files/crs/demo/demoActionScript" does not exists or is not readable
'oracle.crs, 11.2.0.3.0': 无法将文件从 'demoActionScript' 复制到 '/home/app/grid/crs/demo/demoActionScript'
注意一下权限 chmod 777 -R 即可
第四步 grid用户
/OPatch/opatch apply -oh -local /
 /home/grid/OPatch/opatch apply -oh  /home/app/grid -local /soft/gidpsu/16902043
第五步 oracle用户
$ //custom/server//custom/scripts/prepatch.sh -dbhome
/soft/gidpsu/17076717/custom/server/17076717/custom/scripts/prepatch.sh -dbhome /home/app/oracle
第六步 oracle用户
$ /OPatch/opatch napply -oh -local //custom/server/
/home/oracle/OPatch/opatch napply -oh  /home/app/oracle -local /soft/gidpsu/17076717/custom/server/17076717
这里要注意一些权限问题，以及以前升级的HISTROY文件问题。
第七步 oracle用户
$ /OPatch/opatch apply -oh -local /
/home/oracle/OPatch/opatch apply -oh  /home/app/oracle -local /soft/gidpsu/16902043
第八步 oracle用户
//custom/server//custom/scripts/postpatch.sh -dbhome
/soft/gidpsu/17076717/custom/server/17076717/custom/scripts/postpatch.sh -dbhome /home/app/oracle
第九步骤 root用户
/rdbms/install/rootadd_rdbms.sh
/home/app/grid/rdbms/install/rootadd_rdbms.sh
/crs/install/roothas.pl -patch
/home/app/grid/crs/install/roothas.pl -patch

NO09-ISMP-LXDB-01:/home/app/oracle/cfgtoollogs # /home/app/grid/crs/install/roothas.pl -patch
Using configuration parameter file: /home/app/grid/crs/install/crsconfig_params
CRS-4123: Oracle High Availability Services has been started.

第十步 将patch应用到数据库
cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP
SQL> @catbundle.sql psu apply
SQL> QUIT

整个升级刚过程报错基本都是权限问题，因为有grid和oracle两个用户，注意一下报错后查看权限修改即可，不过
这种升级过程比较复杂。