今天早上到公司发现集群数据库有一台机器down 机，集群件 crsctl status res -t 看都是online 只是instance down
查看alter_xxx.log
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/mes/mes1/trace/mes1_p001_30837.trc:
ORA-10388: parallel query server interrupt (failure)
ORA-00600: internal error code, arguments: [kcbzwfcro_2], [89074], [1], [32768], [0], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/mes/mes1/trace/mes1_p001_30837.trc:
ORA-10388: parallel query server interrupt (failure)
ORA-00600: internal error code, arguments: [kcbzwfcro_2], [89074], [1], [32768], [0], [], [], [], [], [], [], []
查看：
/u01/app/oracle/diag/rdbms/mes/mes1/trace/mes1_p001_30837.trc:

fatal error occurred while scanning redo
ORA-10388: parallel query server interrupt (failure)
ORA-00600: internal error code, arguments: [kcbzwfcro_2], [89074], [1], [32768], [0], [], [], [], [], [], [], []
Error 10388 occurred while dumping log
ORA-10388: parallel query server interrupt (failure)
ORA-00600: internal error code, arguments: [kcbzwfcro_2], [89074], [1], [32768], [0], [], [], [], [], [], [], []
Parallel Transaction recovery server caught exception 10388

DDE: Problem Key 'ORA 600 [kcbzwfcro_2]' was flood controlled (0x2) (incident: 360988)
ORA-00600: internal error code, arguments: [kcbzwfcro_2], [89074], [1], [32768], [0], [], [], [], [], [], [], []
Potentially stale force-CR-override buffer found before OBJD MISMATCH check.
This issue should be investigated by both cache fusion and space layer.
BH (0x21dd5a498) file#: 13 rdba: 0x03444484 (13/279684) class: 1 ba: 0x21c16a000
set: 72 pool: 3 bsz: 8192 bsi: 0 sflg: 1 pwc: 0,25
dbwrid: 2 obj: 89074 objn: 89074 tsn: 8 afn: 13 hint: f
hash: [0x2ec5dab18,0xa7d8c670] lru: [0xb1d7a088,0x1c1d65028]
lru-flags: hot_buffer
ckptq: [NULL] fileq: [NULL] objq: [0xc9d9e190,0x27419ee38] objaq: [0xb3d6ba00,0x211e61680]
st: XCURRENT md: NULL fpin: 'kdgwh05: kdglfe' tch: 1 le: 0x247f78cd8
flags: block_written_once redo_since_read remote_transfered
force_cr_override
LRBA: [0x0.0.0] LSCN: [0x0.0] HSCN: [0xffff.ffffffff] HSUB: [190]
GLOBAL CACHE ELEMENT DUMP (address: 0x247f78cd8):

上OTN

ORA-600:[kcbzwfcro_2] Reported in Alert.log (文档 ID 2085507.1)

Oracle Database - Enterprise Edition - Version 11.2.0.4 and later
Information in this document applies to any platform.

Alert log reports errs like:

Errors in file /u01/app/oracle/diag/rdbms/piaprod/piaprod2/trace/piaprod2_p029_2980.trc:
ORA-10388: parallel query server interrupt (failure)
ORA-00600: internal error code, arguments: [kcbzwfcro_2], [187984], [1], [32768], [0], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/piaprod/piaprod2/trace/piaprod2_p029_2980.trc:
ORA-10388: parallel query server interrupt (failure)
ORA-00600: internal error code, arguments: [kcbzwfcro_2], [187984], [1], [32768], [0], [], [], [], [], [], [], []

The trace file contains no current sql and a call stack with functions like:

... ksucin krr_thread_read krr_read_buffer krr_parse_redo kcra_scan_redo kcra_dump_redo kcra_dump_redo_internal kcbzwfcro kcbgcur ktbgcur
kturGetCurPinForUndo kdiulk kcoubk ktprundo_intra ktprits ktprmai kxfprdp opirip opidrv sou2o opimai_real ssthrdmain main

At the beginning of the trace file also lines are reported like:

========= Dump for incident 231769 (ORA 600 [kcbzwfcro_2]) ========
----- Beginning of Customized Incident Dump(s) -----
Potentially stale force-CR-override buffer found before OBJD MISMATCH check.
This issue should be investigated by both cache fusion and space layer.
BH (0x57f89cc98) file#: 7 rdba: 0x0006048b (1024/394379) class: 1 ba: 0x57c3ea000
set: 48 pool: 3 bsz: 8192 bsi: 0 sflg: 1 pwc: 0,5
dbwrid: 1 obj: 187984 objn: 187984 tsn: 7 afn: 7 hint: f
hash: [0xc12258060,0x2e7778a30] lru: [0x8f79d14a8,0x6f787e608]
ckptq: [NULL] fileq: [NULL] objq: [0x6af974790,0x3e7a932f0] objaq: [0x9ef72d000,0x977872820]
st: XCURRENT md: NULL fpin: 'kdgwh05: kdglfe' tch: 4 le: 0x7b7e39ef0
flags: block_written_once redo_since_read remote_transfered
force_cr_override
...

CAUSE

The issue seems related to a stale buffer in buffer cache.

Bug reports like below have been filed for a similar situation but have been suspended due to lack of information:
Bug 19141040 - ORA-600 [KCBZWFCRO_2], [821514], [1], [32768]
Bug 19435745 - ORA-600 [KCBZWFCRO_2], [107470], [1], [32768], [0] IN PQ SLAVES

However, the bug reports mention a possible workaround a Flush buffer cache OR to restart the instance and check.

SOLUTION（解决办法）

Verify if next workarounds can be used for your situation:

Flush buffer cache OR to restart the instance

This is because the error will continue whilst the stale buffer is in cache. We can try flushing the buffer cache to invalidate this.
If that doesn't help the instance will need to be restarted to clear the cache.

- OR -
Set "_gc_override_force_cr=false" to avoid this particular issue

There should not be any noticeable performance impact.

If the issue still persists even after testing above options, then we possibly may need to file a new bug report.

看到bug Bug 19435745

描述：

- Exadata 11.2.04 + BP#6
- parallel execution slaves died with :

ORA-600: [kcbzwfcro_2], [107470], [1], [32768], [0]

- For example:

Sun Aug 10 10:16:55 2014
Errors in file
/u01/app/oracle/diag/rdbms/cwlv/cwlv1/trace/cwlv1_p049_67791.trc
(incident=235514):
ORA-600: internal error code, arguments: [kcbzwfcro_2], [107470], [1],
[32768], [0], [], [], [], [], [], [], []
Incident details in:
/u01/app/oracle/diag/rdbms/cwlv/cwlv1/incident/incdir_235514/cwlv1_p049_67791_
i235514.trc

DIAGNOSTIC ANALYSIS:
--------------------

- Seems to have happened during parallel transaction recovery.

- cwlv1_p049_67791.trc :

*** NAME:() 2014-08-10 10:11:41.887

Incident 235513 created, dump file:
/u01/app/oracle/diag/rdbms/cwlv/cwlv1/incident/incdir_235513/cwlv1_p049_67791_
i235513.trc
ORA-600: internal error code, arguments: [kcbzwfcro_2], [107470], [1],
[32768], [0], [], [], [], [], [], [], []

fatal error occurred while scanning redo
ORA-10388: parallel query server interrupt (failure)
ORA-600: internal error code, arguments: [kcbzwfcro_2], [107470], [1],
[32768], [0], [], [], [], [], [], [], []
Error 10388 occurred while dumping log
ORA-10388: parallel query server interrupt (failure)
ORA-600: internal error code, arguments: [kcbzwfcro_2], [107470], [1],
[32768], [0], [], [], [], [], [], [], []
Parallel Transaction recovery server caught exception 10388

- cwlv1_p029_67736_i235355.trc :

*** NAME:() 2014-08-10 10:11:41.887

Dump continued from file:
/u01/app/oracle/diag/rdbms/cwlv/cwlv1/trace/cwlv1_p049_67791.trc
ORA-600: internal error code, arguments: [kcbzwfcro_2], [107470], [1],
[32768], [0], [], [], [], [], [], [], []

========= Dump for incident 235513 (ORA-600 [kcbzwfcro_2]) ========
----- Beginning of Customized Incident Dump(s) -----
Potentially stale force-CR-override buffer found before OBJD MISMATCH check.
This issue should be investigated by both cache fusion and space layer.
BH (0xe3ea5ff58) file#: 15 rdba: 0x03e9bc65 (15/2735205) class: 1 ba:
0xe300c6000
...

ITPub博客

ORA-600:[kcbzwfcro_2] Reported in Alert.log (文档 ID 2085507.1)

CAUSE

SOLUTION（解决办法）

最新文章