本文是分享 OBCE V3 培训的理论和实验实践的第八篇。OBCE V3 实验的数据迁移主要就是 OMS 的部署的 MySQL和ORACLE（可选）的迁移。这个在此前文章《OB 企业版数据同步 OMS 部署实践》里分享过部署，基本差不多就不重复了。今天接着分享 OB 高可用的主备集群实验。

1. 理论部分

OB 的高可用理论涉及到 OBProxy 和 OB集群的高可用方案，以及 OBProxy 和 OB 之间的链路探测等。本文只围绕 OB 集群的高可用方案。理解 OB 集群高可用方案首先从 ORACLE 高可用方案说起。

1.1 ORACLE 高可用方案回顾

ORACLE 客户常用的高可用方案是 RAC + DG 。RAC 依赖共享存储，提供计算节点的高可用能力。当共享存储故障时就需要切换到 DG 实例。DG 实例可以有多个，其中部分 DG 实例可以打开供业务查询用。ORACLE DG 还可以开启闪回功能，然后直接激活提供读写用于业务测试验证，验证完后闪回到过去状态，继续转为备库角色。这个功能的使用也有一些具体的限制。如闪回空间和闪回时间等。ORACLE 的 DG 可以是单实例也可以是 RAC ，跟主库类型无直接关系。

1.2 OB 主集群多副本同步原理

OB 的高可用方案首先是集群的高可用。生产环境主集群通常是三副本或五副本，集群可以分布在一个机房或多个机房，可以跨两个或三个城市。集群的高可用能力前提是故障的副本数不超过集群副本成员的一半（三副本集群能最多坏一个副本，五副本集群能最多坏两个副本）。

生产环境通常是 switchover 演练检验备集群的能力，真正 failover 的机会很少。因为 OB 主集群的可用性比传统 ORACLE 的单实例或 RAC 实例还是高很多。这得益于 OB 集群的多副本架构是无共享架构（shared-nothing）并且多副本事务日志的同步使用的是 MultiPaxos 协议。OBCE 培训关于这部分的篇幅挺多，主要是 Paxos 相对 Raft 协议的优势。Paxos 的算法原理比较复杂，实际运维中只需要关注多数派通信的性能即可。OB 3.x 以后未提交事务可以转储，在事务进行过程中事务日志就生成并开始同步（需要多数派确认），这个事务日志落盘跟 DML 是解耦的，所以不影响 DML 性能。但是在事务提交的时候只有所有事务日志都落盘成功 COMMIT 才会成功。

这个图需要调整一下，事务开始后的 DML 应该也会在备副本中同时进行。

1.3 OB 备集群部署

如果故障的副本数超过一半成员，高可用也依赖OB的DG（备集群）。这个跟ORACLE DG类似。OB 备集群的搭建命令非常简单，当主备集群建立同步关系时，备集群会自动从主集群拉取最近的全量合并生成的快照版本以及此后的事务日志，不需要 DBA 专门去做备份恢复。OB 的备集群的副本数跟主集群没有直接关系，不过资源能力也不能相差太多。备集群里的租户跟主集群的租户是一致的，不同的是备集群的租户的资源规格可以跟主集群不一样。

在部署备集群的时候，默认逻辑是按照主集群的资源规格创建租户。如果备集群资源不足，则会创建失败。此时需要DBA介入调整备集群已有资源使用情况。在使用 OCP 自动部署备集群的时候很容易碰到这个问题。如果是命令行部署备集群，则可以在备集群初始化成功后创建一个默认资源模板 standby_unit_config_template，确保所有租户使用这个模板的时候能从备集群中创建出业务租户。

1.4 OB 主备集群的应用连接

OB 主备集群的集群名（cluster_name）是一样的，这是识别主备集群关系的必要条件。不一样的是集群ID（cluster_id)，这个会体现在集群的OBSERVER 进程初始化参数里。OB 的连接方式里用户名决定了是连接到哪个集群哪个租户，用户名里的集群名不带数字（:集群ID）表示连接的是主集群。即使后边主备集群切换了，它也会连接到新的主集群里。如果要连接备集群，则可以在集群名后面带上备集群 ID 。如格式：

obclient -h10.0.0.61 -P2883 -uSYS@oboracle#obdemo:1680747946 -paaAA11__

不过，这个只是一个 OBProxy 跟OCP绑定使用关联了多个 OB集群（主和备）才要求带上这个集群 ID。如果OBProxy是直接跟 OB 备集群绑定的，那连接的时候不需要带上备集群的 ID 。

1.5 OB 主备同步模式

OB 主备集群同步模式支持最大性能、最大可用和最大保护，这点跟 ORACLE 基本一样。在使用最大保护的时候意味着主集群的事务提交成功的标准除了要看主集群里多数派副本的事务日志要成功，还要看备集群里这笔事务日志落盘成功。而备集群里这个事务日志落盘成功的标准就是备集群的多数派成员都成功。所以，如果备集群也是三副本的时候，主备集群最大保护的同步模式性能相比最大性能同步模式下的性能理论上会差很多。

1.6 OB 主备切换场景

OB 主备集群切换也有 switchover 和 failover 两种。

switchover 命令跟ORACLE 一样，在主备之间可以反复多次切换。

failover 又分为无损切换和有损切换。当备集群的同步模式是最大保护的时候，主集群发生故障后，备集群可以直接 failover 为主集群。当老的主集群故障修复后，还可以转换为备集群角色，减少重搭备库的次数。如果故障前主备集群是运行在最大性能模式并且还有不一致数据，故障发生后可以激活备集群为主集群。老的主集群后面修复故障后可以转换为只读集群，用于读取数据，手动判断丢失数据并到新集群上补齐。这个只读集群后面不能转换为备集群，只能删除重建。

1.7 备集群的读问题

OB 备集群肯定是不能写的，读默认也不行，需要开启弱一致性读特性。这个可以在会话层面或者 SQL 层面（用 HINT）。如果应用不想改，那也可以在 OBProxy 里设置参数 obproxy_read_consistency 为 1 实现。

(SYS@10.0.0.62:2883) [SYS]> select count(*) from tpcc.bmsql_warehouse;ORA-16000: standby cluster support weak read only

也就是说如果业务应用想在 OB 主集群和被集群之间做读写分离，目前是需要单独维护一个连接到备集群的数据库连接。通常用一个独立的 OBProxy 最方便。

2. 实验部分

OBCE 的高可用实验就是主备库。包括部署备集群、主备集群的 switchover 和 failover 操作（分无损和有损两种）。阿里云的实验环境练习时间是 4 小时，还是很充分的。

2.1 OB 备集群部署

部署备集群之前一样要对备集群节点做初始化。相关参数设置，NTP 时间同步都要做。如果节点内存资源不高的时候，在节点进程 OBSERVER 启动时要指定一些内存相和 CPU 相关的参数。其中关键的是参数 -n 的名字跟主集群的集群名保持一致，而 -c 参数跟的集群 ID 不能跟主集群一样。

下面是主备集群进程启动参数示例。

cd /home/admin/oceanbase && bin/observer -r '172.17.0.3:2882:2881' -p 2881 -P 2882 -z zone1 -n obdemo -c 2 -d /home/admin/oceanbase/store -i eth0 -l WARN -o "__min_full_resource_pool_memory=1073741824,memory_limit=9G,system_memory=1G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,major_freeze_duty_time=Disable,minor_freeze_times=100,enable_separate_sys_clog=False,enable_merge_by_turn=False,datafile_size=20G,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=10"
cd /home/admin/oceanbase && bin/observer -r '172.17.0.4:2882:2881' -p 2881 -P 2882 -z zone1 -n obdemo -c 1 -d /home/admin/oceanbase/store -i eth0 -l WARN -o "__min_full_resource_pool_memory=1073741824,memory_limit=9G,system_memory=1G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,major_freeze_duty_time=Disable,minor_freeze_times=100,enable_separate_sys_clog=False,enable_merge_by_turn=False,datafile_size=20G,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=10"

备集群多个节点进程都启动后，一样也要进行初始化 BOOTSTRAP 操作。这一步跟主集群的 BOOTSTAP 命令稍微有点不一样。下面我将主备集群的 BOOTSTRAP 命令放在一起方便对比理解。

set session ob_query_timeout=10000000000;alter system bootstrap ZONE 'zone1' SERVER '172.17.0.3:2882' ;
set session ob_query_timeout=10000000000;ALTER SYSTEM BOOTSTRAP CLUSTER STANDBY ZONE 'zone1' SERVER '172.17.0.4:2882' PRIMARY_CLUSTER_ID 2 PRIMARY_ROOTSERVICE_LIST '172.17.0.3:2882' ;

从命令里看出备集群的 OBSERVER 进程认主集群看的是主集群的 rootservice_list 。

备集群初始化成功后 DBA 在备集群创建一个默认的资源规格模板。资源大小根据备集群节点的资源能力定。

CREATE resource unit standby_unit_config_template max_cpu=2,min_cpu=2,max_memory='1G',min_memory='1G',max_iops=1024,max_disk_size=1024000000,max_session_Num=100;

这个在后面备集群同步主集群的租户时会用到这个模板。通常是在备集群资源跟主集群资源不对等（创建不出来主集群的租户）时才用这个。不过 OCP 自动化搭建备集群的步骤里没有给机会创建这个默认的资源规格模板，所以 OCP 里部署备集群可能会报错（OBCE 考试里会碰到）。

当主备集群都初始化后，还需要在主集群上添加一个备集群信息。这个时候主集群认备集群就是通过集群 ID （备集群初始化的时候注册到主集群上的）。

obclient [oceanbase]> ALTER SYSTEM ADD CLUSTER VERIFY;Query OK, 0 rows affected (0.013 sec)
obclient [oceanbase]> ALTER SYSTEM ADD CLUSTER obdemo CLUSTER_ID 1;Query OK, 0 rows affected (0.012 sec)

可以通过备集群的事件视图（__all_rootservice_event_history) 了解备集群添加的细节步骤。

SELECT gmt_create ,module,event ,name1 ,value1 ,name2,value2,name3,value3,name4,value4 FROM `__all_rootservice_event_history` WHERE 1=1  AND module NOT IN ('schema_recycler')ORDER BY gmt_create DESC LIMIT 10;

2.2 OB 主备集群信息查看

查看主备集群信息语句。

查询视图 v$ob_all_clusters 可以看到所有主备集群部分信息。这个在主备集群查看都是一样的。这个在主备集群看到的都是一样的。

obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, protection_mode,protection_level,redo_transport_options FROM v$ob_cluster;       +------------+--------------+--------------+----------------+---------------------+---------------------+------------------------------+| cluster_id | cluster_name | cluster_role | cluster_status | protection_mode     | protection_level    | redo_transport_options       |+------------+--------------+--------------+----------------+---------------------+---------------------+------------------------------+|          2 | obdemo       | PRIMARY      | VALID          | MAXIMUM PERFORMANCE | MAXIMUM PERFORMANCE | ASYNC NET_TIMEOUT = 30000000 |+------------+--------------+--------------+----------------+---------------------+---------------------+------------------------------+1 row in set (0.003 sec)

如果想看当前集群是主集群还是备集群，查看视图 v$ob_cluster 。

obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status,"switchover#",protection_mode,protection_level,redo_transport_options FROM v$ob_cluster;+------------+--------------+--------------+----------------+-------------+---------------------+---------------------+-----------------------------+| cluster_id | cluster_name | cluster_role | cluster_status | switchover# | protection_mode     | protection_level    | redo_transport_options      |+------------+--------------+--------------+----------------+-------------+---------------------+---------------------+-----------------------------+|          2 | obdemo       | PRIMARY      | VALID          | switchover# | MAXIMUM PERFORMANCE | MAXIMUM PERFORMANCE | SYNC NET_TIMEOUT = 10000000 |+------------+--------------+--------------+----------------+-------------+---------------------+---------------------+-----------------------------+1 row in set (0.003 sec)

上面这个就是主集群的，下面这个是备集群的。

obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, protection_mode,protection_level,redo_transport_options FROM v$ob_cluster;       +------------+--------------+------------------+----------------+---------------------+---------------------+------------------------------+| cluster_id | cluster_name | cluster_role     | cluster_status | protection_mode     | protection_level    | redo_transport_options       |+------------+--------------+------------------+----------------+---------------------+---------------------+------------------------------+|          1 | obdemo       | PHYSICAL STANDBY | VALID          | MAXIMUM PERFORMANCE | MAXIMUM PERFORMANCE | ASYNC NET_TIMEOUT = 30000000 |+------------+--------------+------------------+----------------+---------------------+---------------------+------------------------------+1 row in set (0.002 sec)

主集群还可以通过视图 V$OB_STANDBY_STATUS 查看备集群信息，备集群这个视图内容为空。

obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role, cluster_status, rootservice_list,redo_transport_options,protection_level,synchronization_status FROM V$OB_STANDBY_STATUS\G*************************** 1. row ***************************            cluster_id: 1          cluster_name: obdemo          cluster_role: PHYSICAL STANDBY        cluster_status: VALID      rootservice_list: 172.17.0.4:2882:2881redo_transport_options: ASYNC NET_TIMEOUT = 30000000      protection_level: MAXIMUM PERFORMANCEsynchronization_status: OK1 row in set (0.004 sec)

2.3 OB 主备集群 switchover

切换之前要检查主备集群的状态，并且这些状态会随着切换进度而变化。

主集群切换前状态，必须是 TO STANDBY

obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, switchover_status, switchover_info from v$ob_cluster;+------------+--------------+--------------+----------------+-------------------+----------------------------+| cluster_id | cluster_name | cluster_role | cluster_status | switchover_status | switchover_info            |+------------+--------------+--------------+----------------+-------------------+----------------------------+|          2 | obdemo       | PRIMARY      | VALID          | TO STANDBY        | SYNCED STANDBY CLUSTERS: 1 |+------------+--------------+--------------+----------------+-------------------+----------------------------+1 row in set (0.038 sec)

备集群切换前状态，这个 NOT ALLOWED 是正常的。

obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, switchover_status, switchover_info from v$ob_cluster;+------------+--------------+------------------+----------------+-------------------+-------------------------+| cluster_id | cluster_name | cluster_role     | cluster_status | switchover_status | switchover_info         |+------------+--------------+------------------+----------------+-------------------+-------------------------+|          1 | obdemo       | PHYSICAL STANDBY | VALID          | NOT ALLOWED       | NO CLUSTER IN SWITCHING |+------------+--------------+------------------+----------------+-------------------+-------------------------+1 row in set (0.005 sec)

主集群开始切换，集群角色变更为 PHYSICAL STANDBY 。

obclient [oceanbase]> alter system commit to switchover to physical standby;Query OK, 0 rows affected (0.069 sec)
obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, switchover_status, switchover_info from v$ob_cluster;+------------+--------------+------------------+----------------+----------------------+-----------------+| cluster_id | cluster_name | cluster_role     | cluster_status | switchover_status    | switchover_info |+------------+--------------+------------------+----------------+----------------------+-----------------+|          2 | obdemo       | PHYSICAL STANDBY | VALID          | SWITCHOVER SWITCHING |                 |+------------+--------------+------------------+----------------+----------------------+-----------------+1 row in set (0.002 sec)

备集群开始切换前，状态为 TO PRIMARY 。集群角色变更为 PRIMARY 。

obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, switchover_status, switchover_info from v$ob_cluster;+------------+--------------+------------------+----------------+-------------------+-----------------+| cluster_id | cluster_name | cluster_role     | cluster_status | switchover_status | switchover_info |+------------+--------------+------------------+----------------+-------------------+-----------------+|          1 | obdemo       | PHYSICAL STANDBY | VALID          | TO PRIMARY        |                 |+------------+--------------+------------------+----------------+-------------------+-----------------+1 row in set (0.017 sec)
obclient [oceanbase]> alter system commit to switchover to primary;Query OK, 0 rows affected (0.041 sec)
obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, switchover_status, switchover_info from v$ob_cluster;+------------+--------------+--------------+----------------+-------------------+----------------------------+| cluster_id | cluster_name | cluster_role | cluster_status | switchover_status | switchover_info            |+------------+--------------+--------------+----------------+-------------------+----------------------------+|          1 | obdemo       | PRIMARY      | VALID          | TO STANDBY        | SYNCED STANDBY CLUSTERS: 2 |+------------+--------------+--------------+----------------+-------------------+----------------------------+1 row in set (0.038 sec)
obclient [oceanbase]> select * from v$ob_all_clusters;+------------+--------------+------------------+----------------+----------------------+------------------------------+| cluster_id | cluster_name | cluster_role     | cluster_status | rootservice_list     | redo_transport_options       |+------------+--------------+------------------+----------------+----------------------+------------------------------+|          1 | obdemo       | PRIMARY          | VALID          | 172.17.0.4:2882:2881 | ASYNC NET_TIMEOUT = 30000000 ||          2 | obdemo       | PHYSICAL STANDBY | VALID          | 172.17.0.3:2882:2881 | ASYNC NET_TIMEOUT = 30000000 |+------------+--------------+------------------+----------------+----------------------+------------------------------+2 rows in set (0.019 sec)

2.4 主备集群 failover 测试

先看看上面的集群如果主集群 CRASH 会怎么样。KILL 掉主集群（cluster_id=1)

查看备集群里视图。

obclient [oceanbase]> select * from v$ob_all_clusters;+------------+--------------+------------------+----------------+----------------------+------------------------------+| cluster_id | cluster_name | cluster_role     | cluster_status | rootservice_list     | redo_transport_options       |+------------+--------------+------------------+----------------+----------------------+------------------------------+|          1 | obdemo       | PRIMARY          | VALID          | 172.17.0.4:2882:2881 | ASYNC NET_TIMEOUT = 30000000 ||          2 | obdemo       | PHYSICAL STANDBY | VALID          | 172.17.0.3:2882:2881 | ASYNC NET_TIMEOUT = 30000000 |+------------+--------------+------------------+----------------+----------------------+------------------------------+2 rows in set (0.001 sec)

这个视图 v$ob_all_clusters 不能反应主集群状态，所以不能依此判断。

obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, switchover_status, switchover_info,protection_mode, protection_level from v$ob_cluster;+------------+--------------+------------------+----------------+-------------------+-----------------+---------------------+---------------------+| cluster_id | cluster_name | cluster_role     | cluster_status | switchover_status | switchover_info | protection_mode     | protection_level    |+------------+--------------+------------------+----------------+-------------------+-----------------+---------------------+---------------------+|          2 | obdemo       | PHYSICAL STANDBY | VALID          | NOT ALLOWED       |                 | MAXIMUM PERFORMANCE | MAXIMUM PERFORMANCE |+------------+--------------+------------------+----------------+-------------------+-----------------+---------------------+---------------------+1 row in set (0.007 sec)

确认当前集群是备集群，最大性能模式，日志同步时异步的。看看能不能直接 failover 。

obclient [oceanbase]> alter system failover to 'obdemo' cluster_id=2;ERROR 4708 (HY000): Failover is not allowed when protection level not in MAXIMUM PROTECTION or AVAILABILITYobclient [oceanbase]>

failover 失败，因为当前备集群不是最大同步模式。如果是生产环境，此时悔之晚矣。

将主集群拉起，重新设置备集群的同步模式。首先在主集群确认当前集群的同步模式。

obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, switchover_status, switchover_info,protection_mode, protection_level, redo_transport_options from v$ob_cluster \G*************************** 1. row ***************************            cluster_id: 1          cluster_name: obdemo          cluster_role: PRIMARY        cluster_status: VALID     switchover_status: TO STANDBY       switchover_info: SYNCED STANDBY CLUSTERS: 2       protection_mode: MAXIMUM PERFORMANCE      protection_level: MAXIMUM PERFORMANCEredo_transport_options: ASYNC NET_TIMEOUT = 300000001 row in set (0.044 sec)

然后看看能不能直接修改为最大保护模式。

obclient [oceanbase]> alter system set standby cluster to maximize protection;ERROR 4179 (HY000): set maximum protection mode with zero or more than one cluster in sync mode not allowedobclient [oceanbase]> alter system modify cluster 'obdemo' cluster_id 2 set redo_transport_options = 'sync net_timeout=10000000';Query OK, 0 rows affected (0.005 sec)
obclient [oceanbase]> alter system set standby cluster to maximize protection;Query OK, 0 rows affected (0.074 sec)
obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, switchover_status, switchover_info,protection_mode, protection_level, redo_transport_options from v$ob_cluster \G*************************** 1. row ***************************            cluster_id: 1          cluster_name: obdemo          cluster_role: PRIMARY        cluster_status: VALID     switchover_status: NOT ALLOWED       switchover_info: PRIMARY CLUSTER NOT IN SYNC MODE       protection_mode: MAXIMUM PROTECTION      protection_level: MAXIMUM PROTECTIONredo_transport_options: ASYNC NET_TIMEOUT = 300000001 row in set (0.010 sec)
obclient [oceanbase]>

修改同步模式为最大保护模式时，必须将 REDO 传输模式改为 SYNC 模式。

OB 主备集群改为最大保护模式后，如果备集群 CRASH 了，会发生什么？

杀掉备集群（cluster_id=2)。查看主集群的视图。

obclient [oceanbase]> select * from v$ob_all_clusters;+------------+--------------+------------------+----------------+----------------------+------------------------------+| cluster_id | cluster_name | cluster_role     | cluster_status | rootservice_list     | redo_transport_options       |+------------+--------------+------------------+----------------+----------------------+------------------------------+|          1 | obdemo       | PRIMARY          | VALID          | 172.17.0.4:2882:2881 | ASYNC NET_TIMEOUT = 30000000 ||          2 | obdemo       | PHYSICAL STANDBY | VALID          | 172.17.0.3:2882:2881 | SYNC NET_TIMEOUT = 10000000  |+------------+--------------+------------------+----------------+----------------------+------------------------------+2 rows in set (0.002 sec)
obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, switchover_status, switchover_info,protection_mode, protection_level, redo_transport_options from v$ob_cluster\G*************************** 1. row ***************************            cluster_id: 1          cluster_name: obdemo          cluster_role: PRIMARY        cluster_status: VALID     switchover_status: NOT ALLOWED       switchover_info: CAN NOT ACCESS CLUSTERS: 2       protection_mode: MAXIMUM PROTECTION      protection_level: MAXIMUM PROTECTIONredo_transport_options: ASYNC NET_TIMEOUT = 300000001 row in set (0.011 sec)

视图 v$ob_cluster 提示备集群连不上，不能切换。此时尝试在主集群的业务租户写入数据，会 HANG 住，直到 SQL 语句执行超时。

obclient [TPCC]> insert into t1 (id,c1) values(5,'D');ORA-00600: internal error code, arguments: -4012, Timeoutobclient [TPCC]>

可见 OB 主备集群同步模式为最大保护的时候，备集群不可用时，主集群也会停写。并且也不能读。

obclient [TPCC]> select * from t1;ORA-00600: internal error code, arguments: -4012, Timeoutobclient [TPCC]> select /*+ read_consistency(weak) */ * from t1;ORA-00600: internal error code, arguments: -6212, Statement is timeoutobclient [TPCC]>

两个超时错误代码居然还不一样，这个后面再研究。恢复备集群，主集群会话重连后可以读写了。继续 failover 测试。

此时要确认一下备集群上的视图信息变为最大保护模式。

obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, switchover_status, switchover_info,protection_mode, protection_level from v$ob_cluster;+------------+--------------+------------------+----------------+-------------------+-------------------------+--------------------+--------------------+| cluster_id | cluster_name | cluster_role     | cluster_status | switchover_status | switchover_info         | protection_mode    | protection_level   |+------------+--------------+------------------+----------------+-------------------+-------------------------+--------------------+--------------------+|          2 | obdemo       | PHYSICAL STANDBY | VALID          | NOT ALLOWED       | NO CLUSTER IN SWITCHING | MAXIMUM PROTECTION | MAXIMUM PROTECTION |+------------+--------------+------------------+----------------+-------------------+-------------------------+--------------------+--------------------+1 row in set (0.005 sec)

主集群上这个视图信息的切换信息依然提示不允许切换，意思是需要同步。

obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, switchover_status, switchover_info,protection_mode, protection_level, redo_transport_options from v$ob_cluster\G*************************** 1. row ***************************            cluster_id: 1          cluster_name: obdemo          cluster_role: PRIMARY        cluster_status: VALID     switchover_status: NOT ALLOWED       switchover_info: PRIMARY CLUSTER NOT IN SYNC MODE       protection_mode: MAXIMUM PROTECTION      protection_level: MAXIMUM PROTECTIONredo_transport_options: ASYNC NET_TIMEOUT = 300000001 row in set (0.013 sec)

杀掉主集群（cluster_id=1) OBSERVER 进程。备集群 failover 到主集群。

obclient [oceanbase]> alter system failover to 'obdemo' cluster_id=2;Query OK, 0 rows affected (2.451 sec)
obclient [oceanbase]> select * from v$ob_all_clusters;+------------+--------------+------------------+----------------+----------------------+------------------------------+| cluster_id | cluster_name | cluster_role     | cluster_status | rootservice_list     | redo_transport_options       |+------------+--------------+------------------+----------------+----------------------+------------------------------+|          1 | obdemo       | PHYSICAL STANDBY | DISABLED       | 172.17.0.4:2882:2881 | ASYNC NET_TIMEOUT = 30000000 ||          2 | obdemo       | PRIMARY          | VALID          | 172.17.0.3:2882:2881 | SYNC NET_TIMEOUT = 10000000  |+------------+--------------+------------------+----------------+----------------------+------------------------------+2 rows in set (0.001 sec)
obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, switchover_status, switchover_info,protection_mode, protection_level from v$ob_cluster\G*************************** 1. row ***************************       cluster_id: 2     cluster_name: obdemo     cluster_role: PRIMARY   cluster_status: VALIDswitchover_status: NOT ALLOWED  switchover_info: NONE SYNCED STANDBY CLUSTER  protection_mode: MAXIMUM PERFORMANCE protection_level: MAXIMUM PERFORMANCE1 row in set (0.005 sec)

备集群 failover 到主集群成功，同步模式变为最大性能模式。

2.5 OB 主备集群 failover 后的主集群修复

上面备集群 failover 成功后，写入几笔业务数据。然后恢复原主集群（cluster_id=1) 看看，主备集群同步还能修复不？

obclient [TPCC]> insert into t1 (id,c1) values(10,'AA');Query OK, 1 row affected (0.037 sec)
obclient [TPCC]> select * from t1;+----+------+-----------+| ID | C1   | C2        |+----+------+-----------+|  5 | D    | 24-NOV-23 || 10 | AA   | 24-NOV-23 |+----+------+-----------+2 rows in set (0.022 sec)
obclient [TPCC]> commit;Query OK, 0 rows affected (0.024 sec)

恢复老的主集群（cluster_id=1)。不过此时要特别注意，不能用原来的方式启动 OBSERVER 进程（bin/observer) ，而要所有节点 OBSERVER 进程带参数 -m disabled_cluster 启动。否则，启动后的集群继续写入数据就会导致双主集群，这个时候估计再也修复不了（没有测试）。

[admin@server02 oceanbase]$ pwd/home/admin/oceanbase[admin@server02 oceanbase]$ bin/observer -m disabled_clusterbin/observer -m disabled_clusterserver mode: disabled_cluster

老的主集群启动后，查看自身集群信息。

obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, switchover_status, switchover_info,protection_mode, protection_level, redo_transport_options from v$ob_cluster\G*************************** 1. row ***************************            cluster_id: 1          cluster_name: obdemo          cluster_role: PRIMARY        cluster_status: DISABLED     switchover_status: NOT ALLOWED       switchover_info: NO CLUSTER IN SWITCHING       protection_mode: MAXIMUM PROTECTION      protection_level: MAXIMUM PROTECTIONredo_transport_options: ASYNC NET_TIMEOUT = 300000001 row in set (0.012 sec)
obclient [oceanbase]> select * from v$ob_all_clusters;+------------+--------------+------------------+----------------+----------------------+------------------------------+| cluster_id | cluster_name | cluster_role     | cluster_status | rootservice_list     | redo_transport_options       |+------------+--------------+------------------+----------------+----------------------+------------------------------+|          1 | obdemo       | PRIMARY          | VALID          | 172.17.0.4:2882:2881 | ASYNC NET_TIMEOUT = 30000000 ||          2 | obdemo       | PHYSICAL STANDBY | VALID          | 172.17.0.3:2882:2881 | SYNC NET_TIMEOUT = 10000000  |+------------+--------------+------------------+----------------+----------------------+------------------------------+2 rows in set (0.002 sec)

老主集群还认为自己是主，另外一个集群是备。再次说明视图 v$ob_all_clusters 信息不可靠。不过视图 v$ob_cluster 还是可靠的，集群状态为 DISABLED（禁用）状态。

此时可以老的主集群可以转换（CONVERT）到备集群。注意不是切换（SWITCHOVER 或 FAILOVER）。

obclient [oceanbase]> alter system convert to physical standby;Query OK, 0 rows affected (0.030 sec)
obclient [oceanbase]> select * from v$ob_all_clusters;+------------+--------------+------------------+----------------+----------------------+------------------------------+| cluster_id | cluster_name | cluster_role     | cluster_status | rootservice_list     | redo_transport_options       |+------------+--------------+------------------+----------------+----------------------+------------------------------+|          1 | obdemo       | PHYSICAL STANDBY | DISABLED       | 172.17.0.4:2882:2881 | ASYNC NET_TIMEOUT = 30000000 ||          2 | obdemo       | PRIMARY          | VALID          | 172.17.0.3:2882:2881 | SYNC NET_TIMEOUT = 10000000  |+------------+--------------+------------------+----------------+----------------------+------------------------------+2 rows in set (0.001 sec)
obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, switchover_status, switchover_info,protection_mode, protection_level, redo_transport_options from v$ob_cluster\G*************************** 1. row ***************************            cluster_id: 1          cluster_name: obdemo          cluster_role: PHYSICAL STANDBY        cluster_status: DISABLED     switchover_status: NOT ALLOWED       switchover_info:       protection_mode: MAXIMUM PROTECTION      protection_level: MAXIMUM PROTECTIONredo_transport_options: ASYNC NET_TIMEOUT = 300000001 row in set (0.004 sec)
obclient [oceanbase]>

这个时候视图内容有发生变化，集群角色对了，但是集群状态还是禁用状态。

此时还需要到新集群（cluster_id=2）上去激活主备同步。

obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, switchover_status, switchover_info,protection_mode, protection_level from v$ob_cluster\G                                                                                                                                                              *************************** 1. row ***************************                                                                                                              cluster_id: 2     cluster_name: obdemo     cluster_role: PRIMARY   cluster_status: VALIDswitchover_status: NOT ALLOWED  switchover_info: NONE SYNCED STANDBY CLUSTER  protection_mode: MAXIMUM PERFORMANCE protection_level: MAXIMUM PERFORMANCE1 row in set (0.005 sec)
obclient [oceanbase]> alter system enable cluster synchronization 'obdemo' cluster_id=1;Query OK, 0 rows affected (0.047 sec)
obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, switchover_status, switchover_info,protection_mode, protection_level from v$ob_cluster\G*************************** 1. row ***************************       cluster_id: 2     cluster_name: obdemo     cluster_role: PRIMARY   cluster_status: VALIDswitchover_status: NOT ALLOWED  switchover_info: NONE SYNCED STANDBY CLUSTER  protection_mode: MAXIMUM PERFORMANCE protection_level: MAXIMUM PERFORMANCE1 row in set (0.019 sec)

等同步赶上，切换状态会再次变化。

obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, switchover_status, switchover_info,protection_mode, protection_level from v$ob_cluster\G*************************** 1. row ***************************       cluster_id: 2     cluster_name: obdemo     cluster_role: PRIMARY   cluster_status: VALIDswitchover_status: TO STANDBY  switchover_info: SYNCED STANDBY CLUSTERS: 1  protection_mode: MAXIMUM PERFORMANCE protection_level: MAXIMUM PERFORMANCE1 row in set (0.047 sec)

此时这个备集群的数据也可以验证一下。

obclient [TPCC]> select * from t1;ORA-16000: standby cluster support weak read onlyobclient [TPCC]> select /*+ read_consistency(weak) */ * from t1;+----+------+-----------+| ID | C1   | C2        |+----+------+-----------+|  5 | D    | 24-NOV-23 || 10 | AA   | 24-NOV-23 |+----+------+-----------+2 rows in set (0.002 sec)

由于数据没有丢失，这种场景的 failover 又叫无损 failover 。

2.6 OB 主备有损 failover 测试

无损 failover 的前提条件是主备同步用最大保护模式，但估计开启的人不会很多（在 ORACLE 数据库里，就没有多少客户会真的开启最大保护模式）。默认都是以最大性能同步模式运行。当主集群故障的时候，就要进行强制切换了。

先看如果主集群可用的情况下，能不能直接激活备集群（cluster_id=1)。

obclient [oceanbase]> ALTER SYSTEM ACTIVATE PHYSICAL STANDBY CLUSTER;ERROR 4708 (HY000): Failover is not allowed when primary cluster [cluster_id=2] is still aliveobclient [oceanbase]>

从上面结果可见 OB 激活备集群的时候有对主集群进行判断。这样可以防止运维上的误操作。继续杀掉主集群进程。

obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, switchover_status, switchover_info,protection_mode, protection_level, redo_transport_options from v$ob_cluster\G*************************** 1. row ***************************            cluster_id: 1          cluster_name: obdemo          cluster_role: PHYSICAL STANDBY        cluster_status: VALID     switchover_status: NOT ALLOWED       switchover_info:       protection_mode: MAXIMUM PERFORMANCE      protection_level: MAXIMUM PERFORMANCEredo_transport_options: ASYNC NET_TIMEOUT = 300000001 row in set (0.007 sec)

备集群上的集群视图切换信息变为空了。继续激活备集群。

obclient [oceanbase]> ALTER SYSTEM ACTIVATE PHYSICAL STANDBY CLUSTER;                                                                                                ERROR 4012 (HY000): Timeoutobclient [oceanbase]> set session ob_query_timeout=100000000000000;Query OK, 0 rows affected (0.001 sec)
obclient [oceanbase]> ALTER SYSTEM ACTIVATE PHYSICAL STANDBY CLUSTER;Query OK, 0 rows affected (49.186 sec)
obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, switchover_status, switchover_info,protection_mode, protection_level, redo_transport_options from v$ob_cluster\G*************************** 1. row ***************************            cluster_id: 1          cluster_name: obdemo          cluster_role: PRIMARY        cluster_status: VALID     switchover_status: NOT ALLOWED       switchover_info: NONE SYNCED STANDBY CLUSTER       protection_mode: MAXIMUM PERFORMANCE      protection_level: MAXIMUM PERFORMANCEredo_transport_options: ASYNC NET_TIMEOUT = 300000001 row in set (0.005 sec)

激活备集群估计内部操作非常多，默认情况下会语句超时。

这种情况下理论上新主集群会丢失一部分数据。OBCE 实验练习里会通过插入大量数据以及断网来人为导致主备集群在无损 failover 之前的不一致。这里我就不演示了。我们直到理论上会丢失数据即可，所以这种切换也叫有损 failover 。

2.7 OB 主备有损 failover 后的老主集群处理

老的主集群我们试试以默认方式启动看看（bin/observer）。

obclient [oceanbase]> SELECT cluster_id, cluster_name, cluster_role,cluster_status, switchover_status, switchover_info,protection_mode, protection_level from v$ob_cluster\G*************************** 1. row ***************************       cluster_id: 2     cluster_name: obdemo     cluster_role: PRIMARY   cluster_status: VALIDswitchover_status: NOT ALLOWED  switchover_info: NONE SYNCED STANDBY CLUSTER  protection_mode: MAXIMUM PERFORMANCE protection_level: MAXIMUM PERFORMANCE1 row in set (0.024 sec)
obclient [oceanbase]> select * from v$ob_all_clusters;+------------+--------------+------------------+----------------+----------------------+------------------------------+| cluster_id | cluster_name | cluster_role     | cluster_status | rootservice_list     | redo_transport_options       |+------------+--------------+------------------+----------------+----------------------+------------------------------+|          1 | obdemo       | PHYSICAL STANDBY | VALID          | 172.17.0.4:2882:2881 | ASYNC NET_TIMEOUT = 30000000 ||          2 | obdemo       | PRIMARY          | VALID          | 172.17.0.3:2882:2881 | SYNC NET_TIMEOUT = 10000000  |+------------+--------------+------------------+----------------+----------------------+------------------------------+2 rows in set (0.002 sec)
obclient [oceanbase]>‍

显然它还不知道自己已经不是主集群了。尝试读取一下业务数据，也是可以的。并且也能继续写入。

obclient [TPCC]> select * from t1;+----+------+-----------+| ID | C1   | C2        |+----+------+-----------+|  5 | D    | 24-NOV-23 || 10 | AA   | 24-NOV-23 |+----+------+-----------+2 rows in set (0.004 sec)
obclient [TPCC]> insert into t1 (id,c1) values(10,'AA');Query OK, 1 row affected (0.021 sec)

这种老主集群恢复后可以继续写入容易导致业务问题。所以 DBA 在恢复老的主集群时不能这样启动，而应该以只读模式启动，允许业务连上来读取故障前的数据，以判断有损 failover 丢失多少数据。

[admin@server01 oceanbase]$ kill -9 `pidof observer`[admin@server01 oceanbase]$ bin/observer -m disabled_with_readonly_clusterbin/observer -m disabled_with_readonly_clusterserver mode: disabled_with_readonly_cluster

此时可以读，但是不能写入。

obclient [TPCC]> select * from t1;+----+------+-----------+| ID | C1   | C2        |+----+------+-----------+|  5 | D    | 24-NOV-23 || 10 | AA   | 24-NOV-23 |+----+------+-----------+2 rows in set (0.008 sec)
obclient [TPCC]> insert into t1 (id,c1) values(10,'AA');ORA-16000: database open for read-only access

这个集群用完后就可以直接删除了。因为此时是绝无可能再修复主备同步了。只能重新搭建备集群。

总结

本文总结了 OB 主备集群部署的注意事项，详细探索了主备集群的同步模式、各种场景下的切换方式等等。总体来说，OB 的主备集群的部署相比 ORACLE 的部署方法还是要简单好用的多。

关于 OBCE 的实验还有运维监控，主要是锁的排查。以前已经分享过类似文章《https://mp.weixin.qq.com/s/QsPbF0RqpasXR24wWv3UMQ》，就不再重复了。关于实验部分的总结就到这里。下一篇分享 OBCE 机考心得。欢迎继续关注！

更多阅读请参考

ITPub博客

OBCE V3 培训实验：OB 主备集群

最新文章