1 数据库报错，30496归档到ADG报错

Wed Jun 02 15:16:18 2021

ARC3: Archive log rejected (thread 1 sequence 30452) at host 'testadg'

FAL[server, ARC3]: FAL archive failed, see trace file.

ARCH: FAL archive failed. Archiver continuing

ORACLE Instance primary1 - Archival Error. Archiver continuing.

2  在15:18时查看ADG 应用日志情况，发现实例1目前已经应用到30459， 即证明30452redo日志已经应用完成。

从此处可以认为此错误可以忽略。

select process,CLIENT_PROCESS,thread#,sequence#,status,DELAY_MINS from v$managed_standby;SQL> 

3 查看ADG历史应用日志情况，发现在报错时间点，有5分钟的延迟

SQL>   SELECT * FROM V$STANDBY_EVENT_HISTOGRAM order by 5 ;

NAME          TIME UNIT                COUNT LAST_TIME_UPDATED

------------------ -------------- ---------- -----------------------------

apply lag        0 seconds          19713768 06/02/2021 14:38:58

apply lag        1 seconds           1743900 06/02/2021 14:39:00

apply lag        2 seconds             25809 06/02/2021 14:39:02

apply lag        3 seconds             19377 06/02/2021 14:39:04

apply lag        4 seconds             18169 06/02/2021 14:39:06

apply lag        5 seconds             15032 06/02/2021 14:39:07

apply lag        6 seconds             10071 06/02/2021 14:39:08

apply lag        7 seconds              6186 06/02/2021 14:39:11

apply lag        8 seconds              3760 06/02/2021 14:39:12

apply lag        9 seconds              2437 06/02/2021 14:39:13

apply lag       10 seconds              1706 06/02/2021 14:39:15

apply lag       11 seconds              1233 06/02/2021 14:39:16

apply lag       12 seconds               957 06/02/2021 14:39:18

apply lag       13 seconds               821 06/02/2021 14:39:19

apply lag       14 seconds               688 06/02/2021 14:39:20

apply lag       15 seconds               643 06/02/2021 14:39:21

apply lag       16 seconds               550 06/02/2021 14:39:24

apply lag       17 seconds               518 06/02/2021 14:39:25

apply lag       18 seconds               520 06/02/2021 14:39:26

apply lag       19 seconds               524 06/02/2021 14:39:29

apply lag       20 seconds               551 06/02/2021 14:39:30

apply lag       21 seconds               600 06/02/2021 14:39:31

apply lag       22 seconds               541 06/02/2021 14:39:33

apply lag       23 seconds               548 06/02/2021 14:39:34

apply lag       24 seconds               553 06/02/2021 14:39:35

apply lag       25 seconds               553 06/02/2021 14:39:36

apply lag       26 seconds               568 06/02/2021 14:39:37

apply lag       27 seconds               577 06/02/2021 14:39:38

apply lag       28 seconds               543 06/02/2021 14:39:39

apply lag       29 seconds               583 06/02/2021 14:39:43

apply lag       30 seconds               559 06/02/2021 14:39:44

apply lag       31 seconds               606 06/02/2021 14:39:46

apply lag       32 seconds               620 06/02/2021 14:39:47

apply lag       33 seconds               596 06/02/2021 14:39:48

apply lag       34 seconds               613 06/02/2021 14:39:50

apply lag       35 seconds               612 06/02/2021 14:39:51

apply lag       36 seconds               563 06/02/2021 14:39:52

apply lag       37 seconds               551 06/02/2021 14:39:53

apply lag       38 seconds               570 06/02/2021 14:39:54

apply lag       39 seconds               525 06/02/2021 14:39:56

apply lag       40 seconds               536 06/02/2021 14:39:57

apply lag       41 seconds               499 06/02/2021 14:39:59

apply lag       42 seconds               583 06/02/2021 14:40:00

apply lag       43 seconds               568 06/02/2021 14:40:02

apply lag       44 seconds               560 06/02/2021 14:40:03

apply lag       45 seconds               584 06/02/2021 14:40:04

apply lag       46 seconds               590 06/02/2021 14:40:06

apply lag       47 seconds               580 06/02/2021 14:40:07

apply lag       48 seconds               607 06/02/2021 14:40:08

apply lag       49 seconds               647 06/02/2021 14:40:09

apply lag       50 seconds               716 06/02/2021 14:40:11

apply lag       51 seconds               698 06/02/2021 14:40:12

apply lag       52 seconds               678 06/02/2021 14:40:13

apply lag       53 seconds               690 06/02/2021 14:40:15

apply lag       54 seconds               678 06/02/2021 14:40:16

apply lag       55 seconds               751 06/02/2021 14:40:17

apply lag       56 seconds               729 06/02/2021 14:40:18

apply lag       57 seconds               731 06/02/2021 14:40:20

apply lag       58 seconds               762 06/02/2021 14:40:21

apply lag       59 seconds               683 06/02/2021 14:40:23

apply lag        5 minutes              1125 06/02/2021 14:59:23

apply lag        4 minutes              5964 06/02/2021 15:46:23

apply lag        2 minutes             44187 06/02/2021 15:53:35

apply lag        3 minutes             26792 06/02/2021 15:55:26

123 rows selected.

4 查看数据库生成的trace文件

-rw-r----- 1 oracle asmadmin     42035 Jun  2 16:05 primary1_lgwr_48970.trm

-rw-r----- 1 oracle asmadmin    152518 Jun  2 16:05 primary1_lgwr_48970.trc

-rw-r----- 1 oracle asmadmin      2032 Jun  2 16:06 primary1_arc0_49532.trm

-rw-r----- 1 oracle asmadmin     13728 Jun  2 16:06 primary1_arc0_49532.trc

-rw-r----- 1 oracle asmadmin      1538 Jun  2 16:06 primary1_arc1_49534.trm

-rw-r----- 1 oracle asmadmin      9047 Jun  2 16:06 primary1_arc1_49534.trc

-rw-r----- 1 oracle asmadmin  48311632 Jun  2 16:06 alert_primary1.log

5 查看trace文件的内容，发现报ORA-16401错误

[oracle@primarydb1 trace]$ more primary1_arc1_49534.trc

Trace file /u01/app/oracle/diag/rdbms/primary/primary1/trace/primary1_arc1_49534.trc

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining and Real Application Testing options

ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1

System name:    Linux

Node name:      primarydb1

Release:        3.10.0-862.el7.x86_64

Version:        #1 SMP Wed Mar 21 18:14:51 EDT 2018

Machine:        x86_64

Instance name: primary1

Redo thread mounted by this instance: 1

Oracle process number: 51

Unix process pid: 49534, image: oracle@primarydb1 (ARC1)

*** 2021-05-02 00:05:00.093

*** SESSION ID:(1633.21) 2021-05-02 00:05:00.093

*** CLIENT ID:() 2021-05-02 00:05:00.093

*** SERVICE NAME:(SYS$BACKGROUND) 2021-05-02 00:05:00.093

*** MODULE NAME:() 2021-05-02 00:05:00.093

*** ACTION NAME:() 2021-05-02 00:05:00.093

*** TRACE FILE RECREATED AFTER BEING REMOVED ***

Error 16401 creating standby archive log file at host 'testadg'

kcrrwkx: unknown error:16401

Error 16401 creating standby archive log file at host 'testadg'

kcrrwkx: unknown error:16401

Error 16401 creating standby archive log file at host 'testadg'

*** 2021-05-02 00:18:37.836

kcrrwkx: unknown error:16401

Error 16401 creating standby archive log file at host 'testadg'

kcrrwkx: unknown error:16401

*** 2021-05-02 00:22:16.469

Error 16401 creating standby archive log file at host 'testadg'

kcrrwkx: unknown error:16401

6 查看ORA-16401错误，Oracle认为此错误可以忽略

[oracle@primarydb1 trace]$ oerr ora 16401

16401, 00000, "archive log rejected by Remote File Server (RFS)"

// *Cause:  An attempt was made to re-archive an existing archive log.

// *Action: See alert log and trace file for more details.

//          No action is necessary. This is an informational statement provided

//          to record the event for diagnostic purposes.

7 查看日志切换，发现日志切换太频繁，20秒一个，每个日志组大小为2G

8 根据Oracle官网的文档 Doc ID 1243177.1 认为此报错可以忽略,或增加redo log日志组的大小

ORA-16401 and ORA-16055 reported in primary alert.log when redolog switch is over frequently (Doc ID 1243177.1)

CAUSE  --原因

The Problem here is that the Primary Database is switching Logs too frequently.  --日志切换太频繁

Using ARCH to send the archives, every time there's a log switch the Primary has to send the Archivelog to the Standby,  --主库切换日志并发送redo日志到备库

meanwhile another Log Switch occurred on the Primary which causes also another Archivelog to be sent to the Standby,     --同时意味着主库切换到下一个日志组，这会导致另一个日志组再次发送到备库。

but the first one has not finished yet, a GAP is formed and detected by the Standby.    --这时由于前一个redo还没有应用完成，有间隙被诊断出来

At this Time the first Archivelog is also sent as FAL Request,  --前一个redo正在发送请求

but this one will fail because the first one is still being archiving, locked, so the second one fails.  --故导致第二个请求被拒绝，应用失败，导致告警

Also check for the case explained in below note,

Ora-16401: Archivelog Rejected By Rfs (Doc ID 1183143.1)

SOLUTION  --解决方法

1 Ignore these Messages as long as the Standby Database keeps synchronized with the Primary  --只要数据进行同步，此告警可以忽略

2 Database Increase the Size of the Online Redologs to reduce Redolog Switch Frequency       --增加redo日志组的大小，减少切换频率

3 Increase Network Bandwith between the Primary and Standby Database                         --增加主备库之间的带宽