9.Oracle Clusterware Administration

Oracle Clusterware: Overview

• Portable cluster infrastructure that provides HA to RAC databases and/or other applications:

– Monitors applications’ health - 监控应用运行情况

– Restarts applications on failure - 重启失败的应用

– Can fail over applications on node failure - 将应用failover到其他节点

Oracle Clusterware Run-Time View

在Unix平台上，clusterware stack的启动是根据/etc/inittab文件中定义的respawn条目

A basic description of each process:

• Cluster Synchronization Services Daemon (OCSSD) - 通过network interconnect和voting disk实现节点间的健康监控，一旦发现异常，则重启问题节点以避免发生data corruption或split brain。由oracle用户运行，vendor或nonvendor clusterware情况下都使用。

• Process Monitor Daemon (OPROCD) - nonvendor情况下生成，以root运行，用于检测机器硬件或驱动层面的freeze。Linux和windows上使用kernel driver（如hangcheck-timer）来取代这个进程的作用。

• Cluster Ready Services Daemon (CRSD) - Oracle Clusterware高可用的核心，负责管理注册应用的start、stop、monitor和failover。CRS为每个实例起一个专用的RACGIMON进程负责监控各应用的运行情况，CRSD进行负责管理OCR中的各资源的配置及状态。以root用户运行，在进程失败后自动重启。此外，CRSD还可临时生成以下进程来执行某些操作：

- racgeut (Execute Under Timer), to kill actions that do not complete after a certain amount of time

- racgmdb (Manage Database), to start/stop/check instances

- racgchsn (Change Service Name), to add/delete/check service names for instances

- racgons, to add/remove ONS configuration to OCR

- racgvip, to start/stop/check instance virtual IP

• Event Management Daemon (EVMD) - 负责事件发生时event的发送，会生成一个evmlogger的永久进行，并在需要时生成racgevtf进程来执行callouts。以oracle用户运行，进程失败后自动重启。

Manually Control Oracle Clusterware Stack

# crsctl stop crs

# crsctl start crs

# crsctl disable crs

# crsctl enable crs

以root身份运行，用于一些计划停机场合。为使节点重启时不自动启动crs，必须首先disable掉crs。Disable命令并不停止当前运行的crs。

CRS Resources

CRS管理的应用被成为resource。它的profile及相关属性存放在OCR中。Profile中的属性包含Check interval、Action script、Dependencies、Failure policies、Privileges等。每个resource必须拥有action script，其中必须包含startstopcheck操作。

一个resource的生命周期一般包含：

创建或修改profile（crs_profile） -> 将resource注册到OCR中（crs_register）->启动（crs_start） ->查看状态（crs_stat）->重新分配到其他节点（crs_relocate）->停止（crs_stop）->取消注册（crs_unregister）

RAC Resources

可使用crs_stat –t来查看各resource的运行情况。在默认情况下，Oracle Clusterware管理的resouce包括databases、database and asm instances、VIP/ONS/GSD/Listener（统称为nodeapps）、services和service members等。

当某个resouce的state为unknown时，可使用crs_stop –f resoucename命令强行终止。

可使用crs_stat –p resourcename来查看某个resouce的各项属性。

Main Voting Disk Function

CSS使用private network和voting disk来管理节点的驱逐策略。各节点间通过彼此的心跳来检测各个节点是否处于正常状态。当节点间的网络连接出现问题时，各节点通过voting disk来交换状态信息，从而决定哪个节点需要被驱逐。被驱逐的节点从voting disk接收到驱逐信息后自行重启。

在数据库各实例间也有类似机制。有时在各节点间网络正常的情况下也需要驱逐某个实例（被称为Instance Membership Reconfiguration (IMR)），这一功能通过控制文件来实现。

Important CSS Parameters

• MISSCOUNT: - 网络心跳超时以及reconfiguration时的磁盘I/O超时

– Represents network heartbeat timeouts

– Determines disk I/O timeouts during reconfiguration

– Defaults to 30 seconds (60 for Linux)

– Defaults to 600 when using vendor (non-Oracle) clusterware

– Should not be changed

• DISKTIMEOUT: - 非reconfiguration时的磁盘I/O超时，可临时修改

– Represents disk I/O timeouts outside reconfiguration

– Defaults to 200 seconds

– Can be temporarily changed when experiencing very long I/O latencies to voting disks:

1. Shut down Oracle Clusterware on all nodes but one.

2. As root on available node, use: crsctl set css disktimeout M+1

3. Reboot available node.

4. Restart all other nodes.

Multiplexing Voting Disks

Voting disk如要作冗余，则至少要三份。

使用以下公式决定冗余度：v=f*2+1 - v代表voting disk的个数，f代表可以容忍的问题磁盘个数。

Change Voting Disk Configuration

# crsctl add css votedisk

# crsctl delete css votedisk

# crsctl add css votedisk -force

# crsctl delete css votedisk -force

OU教材上说可以动态增加或删除voting disk的副本，但在linux上实验没有成功。实验中只能通过加force选项完成。命令执行前必须停止clusterware。

Back Up and Recover Your Voting Disks

由于voting disk的路径配置在OCR中，难以直接修改，因此推荐使用link文件，使得当原始路径不可用时能较容易地恢复。

• Back up one voting disk by using the dd command. 使用dd命令进行备份，可联机备份。

– After Oracle Clusterware installation

– After node addition or deletion

– Can be done online

$ crsctl query css votedisk

$ dd if= of= bs=4k

OCR Architecture

OCR记录了集群的配置信息。在集群中的每个节点上，都有一份OCR的拷贝保留在内存中，同时有一个CRSD进程对OCR cache进行访问。但只有一个CRSD进程实际地读取和修改OCR文件。该进行负责刷新所在节点以及其他节点上的OCR cache。当本地可户端需要读取OCR的信息时，直接和本地的CRS进程进行交互并读取。当客户端需要修改OCR时，它们通过本地的CRS进程，与负责修改OCR的CRS进程进行通信并完成修改。

主要的客户端包括OUI、srvctl、EM、DBCA、DBUA、NetCA以及VIPCA等。

OCR记录并维护resource间的依赖关系以及状态信息。在安装cluserware时，OUI会提示是否需要对OCR作镜象。在UNIX平台，/etc/oracle/ocr.loc文件中记录了OCR以及其镜象文件的所在路径。

OCR Contents and Organization

OCR中的信息以key-value的形式按照树状结构存储。

The main branches composing the OCR structure:

• The SYSTEM keys - 存储Oracle Clusterware进程（CSSDCRSDEVMD等）相关的数据。

• The DATABASE keys - 存储RAC db（instancenodeappsservices等）相关数据。

• The CRS keys - 存储其他注册应用的相关数据。

Managing OCR Files and Locations: Overview

使用ocrconfig命令对OCR进行维护：

• 使用-export/import导出或导入逻辑备份

• Upgrade/downgrade OCR

• 使用-showbackup查看系统自动生成的备份，使用-backuploc修改自动备份生成的路径，使用-restore恢复备份

• 使用-replace ocr/ocrmirror来增加、修改或删除OCR及OCR镜象

• 在Oracle support的支持下，在一个或多个节点由于OCR corruption无法启动时，使用-overwrite命令修改OCR保护机制

• 使用-repair修改OCR或OCR镜象的相关参数

ocrcheck - 验证OCR及镜象的完整性

ocrdump - 将OCR内容dump成文本或XML文件

Automatic OCR Backups

OCR文件自动备份频率（无法修改）：

每四小时一次（保留最近三份备份）

每天结束时（保留最近两份备份）

每周结束时（保留最近两份备份）

修改默认备份路径：# ocrconfig –backuploc /shared/bak

Back Up OCR Manually

• Daily backups of your automatic OCR backups to a different storage device:

– Use your favorite backup tool.

• Logical backups of your OCR before and after making significant changes:

# ocrconfig –export file name

• Make sure that you restore OCR backups that match your current system configuration.

Recover OCR Using Physical Backups

1. Locate a physical backup:

$ ocrconfig –showbackup

2. Review its contents:

# ocrdump –backupfile file_name

3. Stop Oracle Clusterware on all nodes:

# crsctl stop crs

4. Restore the physical OCR backup:

# ocrconfig –restore /cdata/jfv_clus/day.ocr

5. Restart Oracle Clusterware on all nodes:

# crsctl start crs

6. Check OCR integrity:

$ cluvfy comp ocr -n all

Recover OCR Using Logical Backups

1. Locate a logical backup created using an OCR export.

2. Stop Oracle Clusterware on all nodes.

3. Restore the logical OCR backup:

# ocrconfig –import /shared/export/ocrback.dmp

4. Restart Oracle Clusterware on all nodes.

5. Check OCR integrity:

$ cluvfy comp ocr -n all

Replace an OCR Mirror: Example

# ocrconfig –replace ocrmirror /oradata/OCR2

/etc/oracle/ocr.loc文件同步修改，经实验，文件必须存在才能增加OCR或镜象。可用于新增、修改、删除OCR及镜象文件。

Repair OCR Configuration: Example

当不同节点间对OCR配置信息存在不一致时（一般发生在某个节点未起的情况下进行了OCR/ocrmirror的配置修改），需要使用-repair命令来进行修复。Repair操作必须在crs停止的情况下进行。该命令仅修改OCR配置信息（ocr.loc文件），不对OCR 本身进行修复。

1. Stop Oracle Clusterware on Node2:

# crsctl stop crs

2. Add OCR mirror from Node1:

# ocrconfig –replace ocrmirror /OCRMirror

3. Repair OCR mirror location on Node2:

# ocrconfig –repair ocrmirror /OCRMirror

4. Start Oracle Clusterware on Node2:

# crsctl start crs

OCR Considerations

• If using raw devices to store OCR files, make sure they exist before add or replace operations.

• You must be the root user to be able to add, replace, or remove an OCR file while using ocrconfig.

• While adding or replacing an OCR file, its mirror needs to be online.

• If you remove a primary OCR file, the mirror OCR file becomes primary.

• Never remove the last remaining OCR file.

Change VIP Addresses

1. Determine the interface used to support your VIP:

$ ifconfig -a

2. Stop all resources depending on the VIP:

$ srvctl stop instance -d DB -i DB1

$ srvctl stop asm -n node1

# srvctl stop nodeapps -n node1

3. Verify that the VIP is no longer running:

$ ifconfig -a + $ crs_stat

4. Change IP in /etc/hosts and DNS.

5. Modify your VIP address using srvctl:

# srvctl modify nodeapps -n node1 –A 192.168.2.125/255.255.255.0/eth0

6. Start nodeapps and all resources depending on it:

# srvctl start nodeapps -n node1

7. Repeat from step 1 for the next node.

如修改了虚拟主机名。则相应的listener以及tnsnames也需要修改。

Change Public/Interconnect IP Subnet Configuration: Example

1. You get the current interfaces information by using the getif option.

$ /bin/oifcfg getif

eth0 139.2.156.0 global public

eth1 192.168.0.0 global cluster_interconnect

2. You delete the entry corresponding to public interface first by using the delif option, and

then enter the correct information by using the setif option.

$ oifcfg delif -global eth0

$ oifcfg setif –global eth0/139.2.166.0:public

3. You do the same for your private interconnect.

$ oifcfg delif –global eth1

$ oifcfg setif –global eth1/192.168.1.0:cluster_interconnect

4. You check that the new information is correct.

$ oifcfg getif

eth0 139.2.166.0 global public

eth1 192.168.1.0 global cluster_interconnect

Third-Party Application Protection: Overview

• High Availability framework: - 将应用纳入CRS以提供高可用性

– Command-line tools to register applications with CRS

– Calls control application agents to manage applications

– OCR used to describe CRS attributes for the applications

• High Availability C API:

– Modify directly CRS attributes in OCR

– Modify CRS attributes on the fly

• Application VIPs: - 应用虚拟IP，提供failover功能（VIP与应用同时failover）

– Used for applications accessed by network means

– NIC redundancy

– NIC failover

• OCFS: - 为应用提供共享存储

– Store application configuration files

– Share files between cluster nodes

Application VIP and RAC VIP Differences

• RAC VIP is mainly used in case of node down events: - resource无须failover，RAC VIP被接管后，并不再接受连接，而只是立刻发送失败信息以便于客户端更换地址重连。

– VIP is failed over to a surviving node.

– From there it returns NAK to clients forcing them to reconnect.

– There is no need to fail over resources associated to the VIP.

• Application VIP is mainly used in case of application down events:

– VIP is failed over to another node together with the application(s).

– From there, clients can still connect through the VIP.

– Although not recommended, one VIP can serve many applications.

Use CRS Framework: Overview

1. Create an application VIP, if necessary:

a) Create a profile: Network data + usrvip predefined script

b) Register the application VIP.

c) Set user permissions on the application VIP.

d) Start the application VIP by using crs_start.

2. Write an application action script that accepts three parameters:

• start: Script should start the application.

• check: Script should confirm that the application is up.

• stop: Script should stop the application.

3. Create an application profile:

• Action script location

• Check interval

• Failover policies

• Application VIP, if necessary

4. Set permissions on your application.

5. Register the profile with Oracle Clusterware.

6. Start your application by using crs_start.

Prevent Automatic Instance Restarts

将as值改为2可disable掉自动重启（1-自动重启，0-保持重启前的状态）

# crs_register ora.….inst -update -o as=2,ra=1,ut=7d

10.Diagnosing Oracle Clusterware and RAC Components

The One Golden Rule in RAC Debugging

• Always make sure that your nodes have exactly the same system time to:

– Facilitate log information analysis

– Ensure accurate results when reading GV$ views

– Avoid untimely instance evictions

• The best recommendation is to synchronize nodes using Network Time Protocol.

避免节点间时钟不同步引发的问题

Oracle Clusterware Main Log Files

The main directories used by Oracle Clusterware to store its log files:

• CRS日志在$ORA_CRS_HOME/log//crsd/下，每10MB一归档

• CSS 日志在$ORA_CRS_HOME/log//cssd/下，每20MB一归档

• EVM 日志在$ORA_CRS_HOME/log//evmd下

• 各个resource的日志，格式为imon_.log，每10MB一归档，在

$ORA_CRS_HOME/log//racg 及$ORACLE_HOME/log//racg下

RACG下各可执行命令的日志在该路径下与命令同名的子目录中。

• SRVM (srvctl) 和OCR (ocrdump, ocrconfig, ocrcheck) 等客户端工具的日志在

$ORA_CRS_HOME/log//client/及$ORACLE_HOME/log//client/下

• Oracle Clusterware的重要日志在$ORA_CRS_HOME/log/下，文件名为alert.log

Diagnostics Collection Script

• Script to collect all important log files:

– Must be executed as root

– Is located in $ORA_CRS_HOME/bin/

– Is called diagcollection.pl

• Generates the following files in the local directory:

– basData_.tar.gz

– crsData _. tar.gz

– ocrData _. tar.gz

– oraData _. tar.gz

Cluster Verify: Overview

cluvfy是一种提供各项集群验证功能的工具，在10.2版本中包含该工具。Cluvfy命令可分为两种：

• Stage commands - 在软件安装、数据库创建、配置项更改等各场景前后进行系统环境检查

• Component commands - 对单个组件的状态及完整性等进行验证

Cluster Verify Stages

• -post hwos: Postcheck for hardware and operating system

• -pre cfs: Precheck for CFS setup

• -post cfs: Postcheck for CFS setup

• -pre crsinst: Precheck for CRS installation

• -post crsinst: Postcheck for CRS installation

• -pre dbinst: Precheck for database installation

• -pre dbcfg: Precheck for database configuration

Cluster Verify Components

• nodereach: Checks reachability between nodes

• nodecon: Checks node connectivity

• cfs: Checks Oracle Cluster File System integrity

• ssa: Checks shared storage accessibility

• space: Checks space availability

• sys: Checks minimum system requirements

• clu: Checks cluster integrity

• clumgr: Checks cluster manager integrity

• ocr: Checks OCR integrity

• crs: Checks CRS integrity

• nodeapp: Checks node applications existence

• admprv: Checks administrative privileges

• peer: Compares properties with peers