Deploy Grid (11.2.0.4) on VMware Workstation with Oracle Linux 6.6

Environment:

Software:

VMware Workstation 10

Oracle Linux 6.6 64bit

Oracle GRID 11.2.0.4

Plan:

Two oracle grid nodes + One Linux server as iSCSI / DNS / NTP server

Deployment steps for GRID support server:

1. Build new virtualization machine with Oracle Linux OS type

1.1 Install Oracle Linux 6.6 with software development option (or other option you like, it is only influenced on linux software package installation and would be fix with yum resources).

1.1.1 Bind a fixed IP address on NIC and open ONBOOT option

In /etc/sysconfig/network-scripts/ifcfg-eth0 file (eth0 would in the same subnet with RAC public address), modify ONBOOT=yes, use service network restart or ifdown / ifup commands to check.

1.1.2 Turn off firewall and NetworkManager service

iptables -F ; service iptables save ; chkconfig iptables off

service NetworkManager stop ; chkconfig NetworkManager off

1.2 Configurations of YUM

Modify original YUM file in /etc/yum.repo.d/*.repo or add a new .repo file in same directory.

In the .repo file (suppose we'd like use official disk image) modify or add:

'--------------------------------------------------------------------------------------------

[public_ol6_latest]

name=Oracle Linux $releasever Latest ($basearch)

baseurl=file:///media/OL6.6%20x86_64%20Disc%201%2020141018/

gpgcheck=0

enabled=1

--------------------------------------------------------------------------------------------'

Please note there have four '%20' in baseurl option, it is because of the .repo file would resolve the option in URL format, in this way the '\' symbol would not work in front of a 'space' symbol, so we use '%20'.

You may use yum repolist or yum list commands to verify it.

1.3 Prepare shared disk for GRID deployment with iSCSI

1.3.1 Add second disk file to Linux VM.

1.3.2 Make partitions (suppose new disk is /dev/sdb)

vgcreate crsvote /dev/sdb                              

vgdisplay crsvote                                            

lvcreate -l 511 crsvote -n crsvote1                  

lvcreate -l 511 crsvote -n crsvote2                    divided by PE number

lvcreate -l 511 crsvote -n crsvote3                    on VG named as crsvoteN>

1.3.3 Install iSCSI service

yum install -y scsi-target-utils

1.3.4 Modify configuration file shared three LVs on public network

In /etc/tgt/targets.conf file, add following line:

'--------------------------------------------------------------------------------------------

    backing-store /dev/crsvote/crsvote1

    initiator-address 192.168.100.201

    initiator-address 192.168.100.202

    backing-store /dev/crsvote/crsvote2

    initiator-address 192.168.100.201

    initiator-address 192.168.100.202

    backing-store /dev/crsvote/crsvote3

    initiator-address 192.168.100.201

    initiator-address 192.168.100.202

--------------------------------------------------------------------------------------------'

service tgtd start ; chkconfig tgtd on

So the iSCSI device shared on public network for two RAC nodes.

1.4 Use shared iSCSI disks on clients (nodes)

1.4.1 Find and connect iSCSI device

iscsiadm -m discovery -t sendtargets -p 192.168.100.254           

service iscsi start ; chkconfig iscsi on                                          

udevadm trigger ; udevadm trigger                                            

1.4.2 Bind iSCSI device to RAW device

udevadm info --export-db | grep -A 20 sdc                                

In /etc/udev/rules.d/99-user.rules file or 60-raw.rules file add following line:

'--------------------------------------------------------------------------------------------

ACTION=="add",SUBSYSTEM=="block",ENV{ID_PATH}=="ip-192.168.100.254:3260-iscsi-iqn.2014-12.com.rac:nodesupport:crsvote1-lun-1",RUN+="/bin/raw /dev/raw/raw1 %M %m"

ACTION=="add",SUBSYSTEM=="block",ENV{ID_PATH}=="ip-192.168.100.254:3260-iscsi-iqn.2014-12.com.rac:nodesupport:crsvote2-lun-1",RUN+="/bin/raw /dev/raw/raw2 %M %m"

ACTION=="add",SUBSYSTEM=="block",ENV{ID_PATH}=="ip-192.168.100.254:3260-iscsi-iqn.2014-12.com.rac:nodesupport:crsvote3-lun-1",RUN+="/bin/raw /dev/raw/raw3 %M %m"

KERNEL=="raw[1-3]", OWNER="grid", GROUP="asmadmin", MODE="660"

--------------------------------------------------------------------------------------------'

start_udev ; ll /dev/raw/                                                      

1.5 Prepare DNS resolver for GRID deployment

1.5.1 Install DNS service

yum install -y bind bind-chroot ; service named restart        

1.5.2 Configuration DNS main file

In /var/named/chroot/etc/named.conf file (because we use bind-chroot otherwise the file would located in /etc) keep and modify following line:

'--------------------------------------------------------------------------------------------

options

{

        directory               "/var/named";

        dump-file               "data/cache_dump.db";

        statistics-file         "data/named_stats.txt";

        memstatistics-file      "data/named_mem_stats.txt";

        listen-on port 53       { any; };

        allow-query             { any; };

        allow-query-cache       { any; };

        recursion no;

};

view "public"

{

        match-clients           { any; };

        include "/etc/named.public.zones";

};

--------------------------------------------------------------------------------------------'

1.5.3 Configuration DNS resolver files

In /var/named/chroot/etc/named.public.zones file (also because of bind-chroot) add line:

'--------------------------------------------------------------------------------------------

zone "rac.com" IN {

        type master;

        file "rac.com.public.zone";

        allow-update { none; };

};

--------------------------------------------------------------------------------------------'

In /var/named/chroot/ var/named/rac.com.public.zone file (also because of bind-chroot) add line:

'--------------------------------------------------------------------------------------------

$TTL 86400

@       IN SOA  @ nodesupport.rac.com. (

                                        0       ; serial

                                        1D      ; refresh

                                        1H      ; retry

                                        1W      ; expire

                                        3H )    ; minimum

IN      NS      @

IN      A       127.0.0.1

nodesupport.rac.com.              IN      A       192.168.100.254

node1.rac.com.                          IN      A       192.168.100.201

node2.rac.com.                          IN      A       192.168.100.202

node1-vip.rac.com.                   IN      A       192.168.100.203

node2-vip.rac.com.                   IN      A       192.168.100.204

node1-priv.rac.com.                  IN      A       192.168.200.101

node2-priv.rac.com.                  IN      A       192.168.200.102

node-cluster.rac.com.               IN      A       192.168.100.251

node-cluster-scan.rac.com.     IN      A       192.168.200.251

--------------------------------------------------------------------------------------------'

In /etc/resolv.conf file (on server also all node) add line:

'--------------------------------------------------------------------------------------------

search rac.com

nameserver 192.168.100.254

--------------------------------------------------------------------------------------------'

service named restart ; chkconfig named on

Can use nslookup [name] command to check name resolve, and if there have resolve speed problem you may also edit the /etc/hosts file on all nodes.

1.6 Configuration NTP

If there have third-part time source, all nodes can use it directly. Or can build self-adjustment time server as install ntp package and add self-adjustment options, Or just use Oracle time sync service directly by ignore all ntp configurations on all nodes.

1.6.1 Self-adjustment time server:

yum install -y ntp

In /etc/ntp.conf delete all public server option or make comments on all server options, add line:

'--------------------------------------------------------------------------------------------

server  127.127.1.0

fudge   127.127.1.0 stratum 5

--------------------------------------------------------------------------------------------'

You can use SYNC_HWCLOCK=yes option in /etc/sysconfig/ntpd or use VMware tools to adjustment time with hardware BIOS.

1.6.2 Oracle time sync service:

Move /etc/ntp.conf on all nodes, add an option '-x' in /etc/sysconfig/ntpd file in OPTIONS="-u -x ntp:ntp -p /var/run/ntpd.pid -g" line. Then Grid installer will ignore standard ntp service on linux to use Oracle time sync service.

2. Prepare environment to install Grid

2.1 The network planning

# Public Address

192.168.100.201 node1 node1.rac.com                                  

192.168.100.202 node2 node2.rac.com                                  

# Virtual Address

192.168.100.203 node1-vip node1-vip.rac.com                       

192.168.100.204 node2-vip node2-vip.rac.com                       

# Private Address

192.168.200.101 node1-priv node1-priv.rac.com                    

192.168.200.102 node2-priv node2-priv.rac.com                     

# SCAN Address

# Should resolved by DNS                                                    <192.168.100.251>

# RAC support

192.168.100.254 nodesupport nodesupport.rac.com                

2.2 OS package

Follow the official documents:

yum install -y

compat-libcap1

ibstdc++-33.x86_64

compat-libstdc++-33.i686

glibc-devel.i686

ksh

libstdc++.i686

libstdc++-devel.i686

libaio.i686

libaio-devel.i686

libaio-devel.x86_64

compat-libstdc++-33.i686

compat-libstdc++-33.x86_64

oracleasm-support

oracle-rdbms-server-11gR2-preinstall

ocfs2-tools

In GRID software package: ../grid/rpm

rpm -iv cvuqdisk-1.0.9-1.rpm

2.3 Kernel parameter

oracle-rdbms-server-11gR2-preinstall will fix all or Check Oracle official Documents

2.4 Other system level configuration

In /etc/security/limits.conf file:

oracle   soft   nofile    1024

oracle   hard   nofile    65536

oracle   soft   nproc    16384

oracle   hard   nproc    16384

oracle   soft   stack    10240

oracle   hard   stack    32768

grid   soft   nofile    1024

grid   hard   nofile    65536

grid   soft   nproc    16384

grid   hard   nproc    16384

grid   soft   stack    10240

grid   hard   stack    32768

In /etc/pam.d/login file:

session    required     pam_limits.so

2.5 User and Groups

Add groups

groupadd -g 8000 oinstall

groupadd -g 8001 dba

groupadd -g 8002 oper

groupadd -g 9000 asmadmin

groupadd -g 9001 asmdba

groupadd -g 9002 asmoper

Add users and password

useradd -g oinstall -G dba,asmadmin,asmdba,asmoper -u 11000 grid ; echo oracle | passwd --stdin grid

useradd -g oinstall -G dba,oper -u 12000 oracle ; echo oracle | passwd --stdin oracle

2.6 User profile

For grid user edit ~/.bash_profile:

ORACLE_SID=+ASM1

export ORACLE_SID

ORACLE_BASE=/u01/app/oracle

export ORACLE_BASE

ORACLE_HOME=/u01/app/grid

export ORACLE_HOME

TMP=/u01/tmp

export TMP

TMPDIR=$TMP

export TMPDIR

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib

export LD_LIBRARY_PATH

CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

export CLASSPATH

PATH=$PATH:$HOME/bin:$ORACLE_HOME/bin

export PATH

if [ $USER = "grid" ]; then

  if [ $SHELL = "/bin/ksh" ]; then

    ulimit -p 16384

    ulimit -n 65536

  else

    ulimit -u 16384 -n 65536

  fi

fi

umask 022

For oracle user edit ~/.bash_profile:

ORACLE_SID=rac

export ORACLE_SID

ORACLE_BASE=/u01/app/oracle

export ORACLE_BASE

ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1

export ORACLE_HOME

TMP=/u01/tmp

export TMP

TMPDIR=$TMP

export TMPDIR

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib

export LD_LIBRARY_PATH

CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

export CLASSPATH

PATH=$PATH:$HOME/bin:$ORACLE_HOME/bin

export PATH

if [ $USER = "oracle" ]; then

  if [ $SHELL = "/bin/ksh" ]; then

    ulimit -p 16384

    ulimit -n 65536

  else

    ulimit -u 16384 -n 65536

  fi

fi

umask 022

2.7 Directory planning

mkdir -p /u01/app/grid

mkdir -p /u01/app/oracle/product/11.2.0/db_1

mkdir -p /u01/tmp

mkdie -p /u01/app/oraInventory

chown -R grid:oinstall /u01

chown -R oracle:oinstall /u01/app/oracle

chmod -R 775 /u01

2.8 SSH connective

For grid user and oracle on all node:

ssh-keygen -t rsa

ssh-copy-id node1 ; ssh-copy-id node2

ssh node1.rac.com ; ssh node2.rac.com

ssh node1-priv ; ssh node2-priv

ssh node1-priv.rac.com ; ssh node2-priv.rac.com

User grid can access all node with ssh without password, user oracle can access all node with ssh without password.

2.9 Pre-Install script

Run script:

./runcluvfy.sh stage -pre crsinst -n node1,node2 -fixup –verbose

3. Install GRID on GUI

 

 

Node1:

/u01/app/oraInventory/orainstRoot.sh

/u01/app/grid/root.sh

Node2:

/u01/app/oraInventory/orainstRoot.sh

/u01/app/grid/root.sh

 

 

Use crs_stat -t check service startup on all nodes.

 

4. Storage option (VMware)

4.1 Create Shared Disks

vmware-vdiskmanager -c -a lsilogic -s 2GB -t 4 [Path/Disk Image(N)]

In VMware GUI add disks, and Change disk SCSI bus number to separate with main OS disk bus

4.2 Modify VMware configuration file

Edit .vmx file for all nodes, add scsi[N].SharedBus = "VIRTUAL" and disk.locking = "FALSE" parameters.

4.3 Bind Shared Disks to all nodes

Use udevadm info --export-db command to check udev device information.

Edit /etc/udev/rules.d/60-raw.rules or /etc/udev/rules.d/99-user.rules file to bind raw device.

'--------------------------------------------------------------------------------------------

ACTION=="add", KERNEL=="sdc", RUN+="/bin/raw /dev/raw/raw1 %N"

ACTION=="add", KERNEL=="sdd", RUN+="/bin/raw /dev/raw/raw2 %N"

ACTION=="add", KERNEL=="sde", RUN+="/bin/raw /dev/raw/raw3 %N"

KERNEL=="raw[1-3]", OWNER="grid", GROUP="asmadmin", MODE="660"

--------------------------------------------------------------------------------------------'

Finally, use dd command to initialize all raw disks.

dd if=/dev/zero of=/dev/raw/raw1 bs=1M count=2048

dd if=/dev/zero of=/dev/raw/raw2 bs=1M count=2048

dd if=/dev/zero of=/dev/raw/raw3 bs=1M count=2048

 

5. Shutdown & Startup

5.1 Shutdown

With grid user privilege, execute csr_stop -all shutdown all services on all nodes. Can use crs_stop --help check more options.

5.2 Startup

Generally with nodes startup, CRS services on all nodes will startup automatically. If not and with grid privilege csr_start -all returned "CRS-0184: Cannot communicate with the CRS daemon." message, then with root privilege execute ./crsctl start crs first. Or with OS boot set crsctl enable crs and crsctl check crs