生产AIX系统上Oracle 10.2.0.3数据库近日出现ORA-600错误，同时数据库实例被异常终止，以下为alert日志内容：

 

Thu Sep  8 23:01:49 2011

Errors in file /oracle/product/10.2.0/admin/UC/udump/uc_ora_278916.trc:

ORA-00600: internal error code, arguments: [kcbget_37], [1], [], [], [], [], [], []

Thu Sep  8 23:01:59 2011

Errors in file /oracle/product/10.2.0/admin/UC/udump/uc_ora_278916.trc:

ORA-00600: internal error code, arguments: [kcbso1_1], [], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [kcbget_37], [1], [], [], [], [], [], []

Thu Sep  8 23:02:00 2011

Errors in file /oracle/product/10.2.0/admin/UC/udump/uc_ora_278916.trc:

ORA-00600: internal error code, arguments: [kcbso1_1], [], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [kcbso1_1], [], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [kcbget_37], [1], [], [], [], [], [], []

Thu Sep  8 23:02:01 2011

Errors in file /oracle/product/10.2.0/admin/UC/udump/uc_ora_278916.trc:

ORA-00600: internal error code, arguments: [kcbso1_1], [], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [kcbso1_1], [], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [kcbget_37], [1], [], [], [], [], [], []

Thu Sep  8 23:02:05 2011

Errors in file /oracle/product/10.2.0/admin/UC/bdump/uc_pmon_172250.trc:

ORA-00600: internal error code, arguments: [kcbso1_1], [], [], [], [], [], [], []

Thu Sep  8 23:02:06 2011

Errors in file /oracle/product/10.2.0/admin/UC/bdump/uc_pmon_172250.trc:

ORA-00600: internal error code, arguments: [kcbso1_1], [], [], [], [], [], [], []

Thu Sep  8 23:02:06 2011

PMON: terminating instance due to error 472

Instance terminated by PMON, pid = 172250

 

 

可以看到，由于出现ORA-600报错，实例被pmon进程直接终止。

 

部分uc_ora_278916.trc内容：

 

Dump file /oracle/product/10.2.0/admin/UC/udump/uc_ora_278916.trc

Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production

With the Partitioning, OLAP and Data Mining options

ORACLE_HOME = /oracle/product/10.2.0

System name: AIX

Node name:    EM_AUTH

Release:   3

Version:   5

Machine: 0002D8D2D900

Instance name: UC

Redo thread mounted by this instance: 1

Oracle process number: 28

Unix process pid: 278916, image: oracle@EM_AUTH (TNS V1-V3)

 

*** 2011-09-08 23:01:49.583

*** ACTION NAME:() 2011-09-08 23:01:49.574

*** MODULE NAME:(Post-40034-splexcd-splexUC) 2011-09-08 23:01:49.574

*** SERVICE NAME:(SYS$USERS) 2011-09-08 23:01:49.574

*** SESSION ID:(859.6834) 2011-09-08 23:01:49.574

BH (700000135fdbac8) file#: 44 rdba: 0x0b20f500 (44/2159872) class: 1 ba: 700000135bf6000

  set: 19 blksize: 8192 bsi: 0 set-flg: 0 pwbcnt: 0

  dbwrid: 0 obj: 15193 objn: 13689 tsn: 5 afn: 44

  hash: [70000018c949da0,70000018c949da0] lru: [70000012dfdba38,7000000dafb7948]

  lru-flags: hot_buffer

  obj-flags: object_ckpt_list

  ckptq: [70000008ffc0978,700000042f8e308] fileq: [700000074fd5618,70000008efd3c98] objq: [700000023f9aeb8,70000005bf9ba68]

  st: XCURRENT md: NULL tch: 176

  flags: buffer_dirty gotten_in_current_mode block_written_once

          redo_since_read

  LRBA: [0x4ad.2c9b3.0] HSCN: [0x8.bd49ec9c] HSUB: [2]

Dump of buffer cache at level 10 for tsn=5, rdba=186709248

BH (700000135fdbac8) file#: 44 rdba: 0x0b20f500 (44/2159872) class: 1 ba: 700000135bf6000

  set: 19 blksize: 8192 bsi: 0 set-flg: 0 pwbcnt: 0

  dbwrid: 0 obj: 15193 objn: 13689 tsn: 5 afn: 44

  hash: [70000018c949da0,70000018c949da0] lru: [70000012dfdba38,7000000dafb7948]

  lru-flags: hot_buffer

  obj-flags: object_ckpt_list

  ckptq: [70000008ffc0978,700000042f8e308] fileq: [700000074fd5618,70000008efd3c98] objq: [700000023f9aeb8,70000005bf9ba68]

  st: XCURRENT md: NULL tch: 176

  flags: buffer_dirty gotten_in_current_mode block_written_once

          redo_since_read

  LRBA: [0x4ad.2c9b3.0] HSCN: [0x8.bd49ec9c] HSUB: [2]

  buffer tsn: 5 rdba: 0x0b20f500 (44/2159872)

  scn: 0x0008.bd49ec9c seq: 0x01 flg: 0x00 tail: 0xec9c0601

frmt: 0x02 chkval: 0x0000 type: 0x06=trans data

 

*** 2011-09-08 23:01:49.609

ksedmp: internal or fatal error

ORA-00600: internal error code, arguments: [kcbget_37], [1], [], [], [], [], [], []

Current SQL statement for this session:

insert into "LINC"."VIPDB_ACCGL" ("GLB_DTIME","XACCOUNT","BANK","CARD_BIN","CURR_NUM","REL_DAY","SAVING_FLG","BAL_CMPINT","BAL_FREE","BAL_INT","BAL_MP","BAL_NOINT","BAL_ORINT","BUSINESS","PRODUCT","REV_IND","STM_BALFRE","STM_BALINT","STM_BALMP","STM_BALORI","STM_NOINT","TODAY_BAL","BAL_CMPFEE") values (:V001,:V002,:V003,:V004,:V005,:V006,:V007,:V008,:V009,:V00A,:V00B,:V00C,:V00D,:V00E,:V00F,:V010,:V011,:V012,:V013,:V014,:V015,:V016,:V017)

 

----- Call Stack Trace -----

calling              call     entry                argument values in hex     

location             type     point                (? means dubious value)    

-------------------- -------- -------------------- ----------------------------

ksedst+001c          bl       ksedst1              000000000 ? FFFFFFFFFFE4F44 ?

ksedmp+0290          bl       ksedst               1048DFFC0 ?

ksfdmp+0018          bl       03F5B014            

kgerinv+00dc         bl       _ptrgl              

kgeasnmierr+004c     bl       kgerinv              11011FCD0 ? 700000135FDBB28 ?

                                                   0FFFFFFFF ? 1054CBF58 ?

                                                   70000018B95BF20 ?

kcbassertb1+0084     bl       kgeasnmierr          110190110 ? 110480040 ?

                                                   1054CCD70 ? 100000001 ?

                                                   000000000 ? 000000001 ?

                                                   000000013 ? 022404224 ?

kcbget+0704          bl       kcbassertb1          1054CCA18 ? 1054CCA44 ?

                                                   0FFFFF7FF ?

ktbxchg+0110         bl       kcbget               7000000F51F4000 ? 000000000 ?

                                                   FFFFFFFFFFEB000 ? 1101B7590 ?

kdifind+0f50         bl       ktbxchg              FFFFFFFFFFE5828 ?

                                                   FFFFFFFFFFE5818 ?

                                                   FFF00FFFFFE5640 ? 000000000 ?

kdiins0+0cc4         bl       kdifind              14001EC70874CA0 ?

                                                   70000017C67C260 ?

                                                   FFFFFFFFFFEB000 ?

                                                   FFFFFFFFFFE6730 ?

                                                   FFFFFFFFFFE6280 ? 0FFFEB000 ?

                                                   000000000 ? 000000000 ?

kdiinsp+0070         bl       kdiins0              70000017C67C258 ? 104FA1578 ?

                                                   FFFFFFFFFFE8E10 ?

                                                   70000017252D988 ? 000000000 ?

                                                   000000000 ? 000000000 ?

                                                   2000000000020 ?

kauxsin+2bbc         bl       kdiinsp              700000129B6B7FC ?

                                                   700000129B6B7D0 ?

                                                   700000129B6B7D0 ?

                                                   700000189EF1828 ? 000000001 ?

                                                   70000018AEF7588 ? 000000000 ?

                                                   FFFFFFFFFFE9072 ?

insidx+09f4          bl       kauxsin              70000017C67C2C8 ? 000000000 ?

                                                   1104AC730 ? 1104AC868 ?

                                                   1104AC928 ? 1104AC958 ?

                                                   1104AC828 ? 000000000 ?

insrowFastPath+0120  bl       insidx               1104AC740 ?

insdrvFastPath+029c  bl       insrowFastPath       FFFFFFFFFFF7800 ?

                                                   70000017FA7BA70 ?

                                                   FFFFFFFFFFF7800 ?

inscovexe+0294       bl       insdrvFastPath       1104AC740 ?

insExecStmtExecIniE  bl       01FC695C            

ngine+005c                                         

insexe+025c          bl       insExecStmtExecIniE  000000017 ? 000000010 ?

                              ngine                000000000 ?

opiexe+26a8          bl       insexe               1105F3DC8 ? FFFFFFFFFFF8A60 ?

opiodr+0adc          bl       _ptrgl              

ttcpip+1020          bl       _ptrgl              

opitsk+10b0          bl       ttcpip               1100C94D8 ? 000000000 ?

                                                   900000000155874 ? 000000000 ?

                                                   FFFFFFFFFFFB5C0 ?

                                                   FFFFFFFFFFFB698 ?

                                                   90000000006AA28 ?

                                                   CCCCCCCCCCCCCCCD ?

opiino+0990          bl       opitsk               1E00000000 ? 000000000 ?

opiodr+0adc          bl       _ptrgl              

opidrv+0474          bl       opiodr               3C10296C90 ? 4102985B0 ?

                                                   FFFFFFFFFFFF550 ? 0102989D8 ?

sou2o+0090           bl       opidrv               3C02AB393C ? 440656000 ?

                                                   FFFFFFFFFFFF550 ?

opimai_real+01bc     bl       01FC4EF4            

main+0098            bl       opimai_real          000000000 ? 000000000 ?

__start+0098         bl       main                 000000000 ? 000000000 ?

 

 

部分uc_pmon_172250.trc内容：

 

Dump file /oracle/product/10.2.0/admin/UC/bdump/uc_pmon_172250.trc

Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production

With the Partitioning, OLAP and Data Mining options

ORACLE_HOME = /oracle/product/10.2.0

System name: AIX

Node name:    EM_AUTH

Release:   3

Version:   5

Machine: 0002D8D2D900

Instance name: UC

Redo thread mounted by this instance: 1

Oracle process number: 2

Unix process pid: 172250, image: oracle@EM_AUTH (PMON)

 

*** 2011-09-08 23:02:04.989

*** SERVICE NAME:(SYS$BACKGROUND) 2011-09-08 23:02:04.977

*** SESSION ID:(885.1) 2011-09-08 23:02:04.977

BH (700000135fdbac8) file#: 44 rdba: 0x0b20f500 (44/2159872) class: 1 ba: 700000135bf6000

  set: 19 blksize: 8192 bsi: 0 set-flg: 0 pwbcnt: 0

  dbwrid: 0 obj: 15193 objn: 13689 tsn: 5 afn: 44

  hash: [70000018c949da0,70000018c949da0] lru: [70000012dfdba38,7000000dafb7948]

  lru-flags: hot_buffer

  obj-flags: object_ckpt_list

  ckptq: [70000008ffc0978,700000042f8e308] fileq: [700000074fd5618,70000008efd3c98] objq: [700000023f9aeb8,70000005bf9ba68]

  st: XCURRENT md: NULL tch: 176

  flags: buffer_dirty gotten_in_current_mode block_written_once

          redo_since_read

  LRBA: [0x4ad.2c9b3.0] HSCN: [0x8.bd49ec9c] HSUB: [2]

Dump of buffer cache at level 10 for tsn=5, rdba=186709248

BH (700000135fdbac8) file#: 44 rdba: 0x0b20f500 (44/2159872) class: 1 ba: 700000135bf6000

  set: 19 blksize: 8192 bsi: 0 set-flg: 0 pwbcnt: 0

  dbwrid: 0 obj: 15193 objn: 13689 tsn: 5 afn: 44

  hash: [70000018c949da0,70000018c949da0] lru: [70000012dfdba38,7000000dafb7948]

  lru-flags: hot_buffer

  obj-flags: object_ckpt_list

  ckptq: [70000008ffc0978,700000042f8e308] fileq: [700000074fd5618,70000008efd3c98] objq: [700000023f9aeb8,70000005bf9ba68]

  st: XCURRENT md: NULL tch: 176

  flags: buffer_dirty gotten_in_current_mode block_written_once

          redo_since_read

  LRBA: [0x4ad.2c9b3.0] HSCN: [0x8.bd49ec9c] HSUB: [2]

  buffer tsn: 5 rdba: 0x0b20f500 (44/2159872)

  scn: 0x0008.bd49ec9c seq: 0x01 flg: 0x00 tail: 0xec9c0601

frmt: 0x02 chkval: 0x0000 type: 0x06=trans data

 

 

*** 2011-09-08 23:02:05.008

ksedmp: internal or fatal error

ORA-00600: internal error code, arguments: [kcbso1_1], [], [], [], [], [], [], []

 

----- Call Stack Trace -----

calling              call     entry                argument values in hex     

location             type     point                (? means dubious value)    

-------------------- -------- -------------------- ----------------------------

ksedst+001c          bl       ksedst1              1052DEDC0 ? 000000002 ?

ksedmp+0290          bl       ksedst               1048DFFC0 ?

ksfdmp+0018          bl       03F5B014            

kgerinv+00dc         bl       _ptrgl              

kgeasnmierr+004c     bl       kgerinv              700000010008000 ? 000000003 ?

                                                   000002000 ? 000000000 ?

                                                   70000018C949B08 ?

kcbassertb+0074      bl       kgeasnmierr          110190110 ? 1104515B8 ?

                                                   1054CC6F0 ? 000000000 ?

                                                   000007FFF ? 000001F5F ?

                                                   000000013 ? 022484224 ?

kcbso1+0170          bl       kcbassertb           1109A3B70 ? 11099D810 ?

kcbpsod+04bc         bl       kcbso1               000000000 ? 110992930 ?

                                                   4000000000 ? 110296CD0 ?

kcbsod+00a4          bl       kcbpsod              000000000 ? 000000001 ?

kssxdl+030c          bl       _ptrgl              

kssdch_stage+05c4    bl       kssxdl               70000018B95BF20 ? 3049A6068 ?

kssdch+0014          bl       kssdch_stage         FFFFFFFFFFFCAB0 ? 110190110 ?

                                                   FFFFFFFFFFFCC10 ?

ksudlc+0158          bl       kssdch               FFFFFFFFFFFCC10 ? 110993AB0 ?

kssxdl+030c          bl       _ptrgl              

ksudlp+019c          bl       kssxdl               70000017DBD8D28 ? 31045A3A8 ?

kssxdl+030c          bl       _ptrgl              

ksuxdl+0274          bl       kssxdl               70000018F99CA88 ? 3FFFFD6B0 ?

ksuxda+027c          bl       ksuxdl               70000018F99CA88 ? 0A2F4589E ?

ksucln+0814          bl       ksuxda              

ksbrdp+0408          bl       _ptrgl              

opirip+03fc          bl       03F59B54            

opidrv+0448          bl       opirip               110296CD0 ? 4102985F0 ?

                                                   FFFFFFFFFFFF800 ?

sou2o+0090           bl       opidrv               3202AB393C ? 440656000 ?

                                                   FFFFFFFFFFFF800 ?

opimai_real+0150     bl       01FC4EF4            

main+0098            bl       opimai_real          000000000 ? 000000000 ?

__start+0098         bl       main                 000000000 ? 000000000 ?

 

 

从trace文件来看，报Ora-00600 [kcbget_37] 时，当时的call stack trace如下： 

kcbassertb1 kcbget ktbxchg kdifind kdiins0 kdiinsp kauxsin insidx insrow

此时pmon的call stack trace为：

kcbassertb kcbso1 kcbpsod kcbsod kssxdl

 

以上现象符合oracle bug 6110331，此报错在10.2.0.4以后版本被修复。

 

当出现此bug时，用户进程可能由于等待cache buffer chains latch而spin或者hang住，甚至导致pmon进程被异常终止，数据库crash。