1. 背景概述

问题描述：晚上ADG 发现有delay问题，同时伴随ORA-16055: FAL request rejected 的错误信息发出

2. 故障分析

2.1.   故障现象

当时报错如下

Mon May 10 02:16:10 2021

ORA-16055: FAL request rejected

Mon May 10 02:16:10 2021

Errors in file /u01/app/oracle/app/diag/rdbms/cbsobdb/cbsobdb1/trace/cbsobdb1_arc1_25558.trc:

ORA-16055: FAL request rejected

Mon May 10 02:21:11 2021

ARC1 (PID:25558): FAL archive failed with error 272 (see trace for details)

ARC1 (PID:25558): FAL archive failed, archiver continuing

Mon May 10 02:21:11 2021

ORACLE Instance cbsobdb1, archival error, archiver continuing

Mon May 10 02:21:11 2021

ORA-16055: FAL request rejected

Mon May 10 02:21:11 2021

Errors in file /u01/app/oracle/app/diag/rdbms/cbsobdb/cbsobdb1/trace/cbsobdb1_arc1_25558.trc:

ORA-16055: FAL request rejected

Mon May 10 02:26:12 2021

ARC1 (PID:25558): SRL selected for T-2.S-8229 for LAD:2

ARC1 (PID:25558): Error 272 archiving LNO:0 to 'CBSOBDB_DG'

ARC1 (PID:25558): FAL archive failed with error 272 (see trace for details)

ARC1 (PID:25558): FAL archive failed, archiver continuing

Mon May 10 02:26:12 2021

ORACLE Instance cbsobdb1, archival error, archiver continuing

分析trc 例如如下

Trace file /u01/app/oracle/app/diag/rdbms/cbsobdb/cbsobdb1/trace/cbsobdb1_arc1_25558.trc

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.7.0.0.0

Build label:    RDBMS_19.7.0.0.0DBRU_LINUX.X64_200403

ORACLE_HOME:    /u01/app/oracle/product/19.0.0/dbhome_1

System name:       Linux

Node name:   gldb01

Release:  3.10.0-1062.el7.x86_64

Version: #1 SMP Thu Jul 18 20:25:13 UTC 2019

Machine:       x86_64

Instance name: cbsobdb1

Redo thread mounted by this instance: 1

Oracle process number: 91

Unix process pid: 25558, image: oracle@gldb01 (ARC1)

*** 2021-05-10 02:16:09.822

*** SESSION ID:(14288.46084) 2023-05-10 02:16:09.822

*** CLIENT ID:() 2023-05-10 02:16:09.822

*** SERVICE NAME:(SYS$BACKGROUND) 2023-05-10 02:16:09.822

*** MODULE NAME:() 2023-05-10 02:16:09.822

*** ACTION NAME:() 2023-05-10 02:16:09.822

*** CLIENT DRIVER:() 2023-05-10 02:16:09.822

*** TRACE FILE RECREATED AFTER BEING REMOVED ***

krsu_upi_status: RFS network connection lost at host 'CBSOBDB_DG', error=3135

krsu_upi_status: Error 3135 detaching RFS from standby instance at host 'CBSOBDB_DG'

krsi_verify_network: Ignoring detach error 3135

*** 2023-05-10 02:16:10.639

*** 2023-05-10 02:16:10.639000 [krsb.c:4584]

krsb_stream_dispatch: Error 272 during streaming operation to remote target:2

*** 2023-05-10 02:16:10.640404 [krsh.c:6348]

Error 272 archiving LNO:0 to 'CBSOBDB_DG'

*** 2023-05-10 02:16:10.644604 [krsi.c:8743]

krsi_dst_fail_caller: LAD:2 err:272 force:0 blast:1

krsf_fal_worker: Encountered error status: 272

<error barrier> at 0x7ffc524a79e0 placed krsf.c@4167

ORA-16055: FAL request rejected

krsf_fal_worker: Encountered error status: 272

2.2.   故障根源

首先看下delay 原因(delay 后可自愈, 不可自愈就要另查原因了) ，最主要的就是1. 网络问题，2. 切换太频繁了。网络问题只能和网络工程师一起排查，切换问题查一下( 检查后发现切换几分钟才有一个不会造成异常) 。

要想解决ORA-16055 首先解决trc 中的 Error 3135 detaching RFS  Error 272

考虑到这个问题Doc ID 739522.1 看到这个mos

 Mainly ORA-3135 occurs when the connection is broken because of underlying network issues.

In  this case, an intermediate firewall between primary and secondary server is altering the data inside the sqlnet packet.

3. 根本解决方案及建议

结合本次故障及发现的问题，给出以下建议

查了mos

Doc ID 1212204.1  -- 写出解决方法

ASA(config)#class-map sqlnet-port
ASA(config-cmap)#match port tcp eq <PORT_NUMBER>
ASA(config-cmap)#exit
ASA(config)#policy-map sqlnet_policy
ASA(config-pmap)#class sqlnet-port
ASA(config-pmap-c)#no inspect sqlnet
ASA(config-pmap-c)#exit
ASA(config)#service-policy sqlnet_policy interface outside

最后网络一查交换机确实没设，设置好后 delay 问题就解决了

之后就是处理 ORA-16055 了

我们的架构是 主 —> 备 1—> 备 2

处理方法是手动在备1 库执行了
ALTER SYSTEM SET log_archive_dest_state_3='DEFER' SCOPE=BOTH;
alter system switch logfile;
ALTER SYSTEM SET log_archive_dest_state_3='ENABLE' SCOPE=BOTH;

但这个肯定不是解决问题的主要原因

分析应该和之前设的，*.fal_client  和*.fal_server 有关，之前写的*.fal_server=' 备1 ，备2 ，当DG delay 的时候他会飘的去备2 这台上面找归档，但当时CBSOBBJ 这个上面归档是没传过来的，所以他会报错ORA-16055 之后传过来他就恢复了