环境：HP-UX Itanium 11.31
数据库版本：10.2.0.4
是否RAC：是


事件描述：
db alert日志中报ORA-600,应用无反馈影响业务


1、巡检发现以下ORA-600，查看DB alert日志如下：
Errors in file /oracle/product/admin/dbrac/udump/dbrac1_ora_19083.trc: 
ORA-00600: internal error code, arguments: [17087], [0xC000000371DB1210], [], [], [], [], [], [] 




2、继续分析事件TRACE日志dbrac1_ora_19083.trc 
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production 
With the Partitioning, Real Application Clusters, OLAP, Data Mining 
and Real Application Testing options 
ORACLE_HOME = /oracle/product/10.2.0/db 
System name: HP-UX 
Node name: dbrac1 
Release: B.11.31 
Version: U 
Machine: ia64 
Instance name: dbrac1 
Redo thread mounted by this instance: 1 
Oracle process number: 137 
Unix process pid: 19083, image: oracle@dbrac1 


*** ACTION NAME:() 2014-04-04 12:36:58.301 
*** MODULE NAME:(test@test1 (TNS V1-V3)) 2014-04-04 12:36:58.301 
*** SERVICE NAME:(dbrac) 2014-04-04 12:36:58.301 
*** SESSION ID:(2782.33812) 2014-04-04 12:36:58.301 
*** 2014-04-04 12:36:58.301 
ksedmp: internal or fatal error 
ORA-00600: internal error code, arguments: [17087], [0xC000000371DB1210], [], [], [], [], [], [] 
No current SQL statement being executed. 


----- Call Stack Trace ----- 
ksedst <- ksedmp <- ksfdmp <- kgeriv <- kgeasi 
<- $cold_kglivr <- kksfbc <- kkspsc0 <- kksParseCursor <- opiosq0 
<- kpooprx <- kpoal8 <- opiodr <- ttcpip <- opitsk 
<- opiino <- opiodr <- opidrv <- sou2o <- opimai_real 
<- main <- main_opd_entry 


PROCESS STATE 
------------- 
Process global information: 
process: c0000003fb173890, call: c0000003ce5f0fe0, xact: 0000000000000000, curses: c0000003fab76a68, usrses: c0000003fab76a68 
---------------------------------------- 
SO: c0000003fb173890, type: 2, owner: 0000000000000000, flag: INIT/-/-/0x00 
(process) Oracle pid=137, calls cur/top: c0000003ce5f0fe0/c0000003ce5f0fe0, flag: (0) - 
int error: 0, call error: 0, sess error: 0, txn error 0 
(post info) last post received: 4614190 0 4 
last post received-location: kslpsr 
last process to post me: c0000003fa145500 1 6 
last post sent: 0 0 117 
last post sent-location: kcbzww 
last process posted by me: c0000003fa158a90 2 0 
(latch info) wait_event=0 bits=20 
holding (efd=5) c0000003e6f7e6c8 Child library cache level=5 child#=2 
Location from where latch is held: kglivr: child: child 
Context saved from call: 9 
state=busy, wlstate=free 
waiters [orapid (seconds since: put on list, posted, alive check)]: 
156 (0, 1396586218, 0) 
83 (0, 1396586218, 0) 
86 (0, 1396586218, 0) 
105 (0, 1396586218, 0) 
85 (0, 1396586218, 0) 
...... 
waiter count=102 
Process Group: DEFAULT, pseudo proc: c0000003fb43fce0 
O/S info: user: oracle, term: UNKNOWN, ospid: 19083 
OSD pid info: Unix process pid: 19083, image: oracle@dbrac1 


SO: c0000003fab76a68, type: 4, owner: c0000003fb173890, flag: INIT/-/-/0x00 
(session) sid: 2782 trans: 0000000000000000, creator: c0000003fb173890, flag: (100041) USR/- BSY/-/-/-/-/- 
DID: 0001-0089-00000AFD, short-term DID: 0001-0089-000007FF 
txn branch: 0000000000000000 
oct: 47, prv: 0, sql: c0000003713c5c28, psql: c0000003c8dc2e78, user: 83/IVRSYS 
service name: dbrac 
O/S info: user: icd, term: , ospid: 29768, machine: test1 
program: test@test1 (TNS V1-V3) 
client info: logon 10.154.92.45 
application name: test@test1 (TNS V1-V3), hash value=1161610004 
last wait for 'SQL*Net message from client' blocking sess=0x0000000000000000 seq=26434 wait_time=2655 seconds since wait started=0 
driver id=54435000, #bytes=1, =0 
Dumping Session Wait History 
for 'SQL*Net message from client' count=1 wait_time=2655 
driver id=54435000, #bytes=1, =0 
for 'SQL*Net message to client' count=1 wait_time=0 
driver id=54435000, #bytes=1, =0 


Current cursor: 40, pgadep: 0 


Cursor#40(9fffffffbf3c26e8) state=NULL curiob=9fffffffbf3e1918 
curflg=44 fl2=0 par=0000000000000000 ses=c0000003fab76a68 
sqltxt(c0000003713c5c28)=begin Sp_Ivr_Rec_Report (:1, :2, :3, :4, :5, :6, :7, :8, :9, :10, :11, :12, :13, :14, :15, :16, :17, :18, :19, :20, :21, :22); end; <<<<< 
hash=12bc8f2afed600159d162df35af0319f 
parent=c00000037c5319c8 maxchild=04 plk=c0000003c6d0c0f0 ppn=n 
cursor instantiation=9fffffffbf3e1918 used=1396586218 
child#0(c000000371db1210) pcs=c00000037c5315d8 
clk=c0000003c67f0a28 ci=c000000399c48e50 pn=c00000039c759468 ctx=0000000000000000 
kgsccflg=0 llk[9fffffffbf3e1920,9fffffffbf3e1920] idx=0 
xscflg=28 fl2=0 fl3=2022000 fl4=100 
sharing failure(s)=4000 
kkscoacd 
Bind#0 
oacdty=96 mxl=128(00) mxlc=00 mal=00 scl=00 pre=00 
oacflg=03 fl2=1000000 frm=01 csi=852 siz=2608 off=0 
No bind buffers allocated 
...... 
Bind#21 
oacdty=02 mxl=22(00) mxlc=00 mal=00 scl=00 pre=00 
oacflg=01 fl2=1000000 frm=00 csi=00 siz=0 off=2584 
No bind buffers allocated 


3、MOS了一把，比对了CALL STACK TRACE，命中Bug 14265450。
Bug 7706062 - OERI [17087] following concurrent hard parses on same cursor ( Doc ID 7706062.8 ) 


Affects: 
Versions confirmed as being affected 10.2.0.4 


Description 


   A library cache lock can get into an inconsistent state 
   if a large number of concurrent hard parses occur against 
   the same cursor. This can lead to ORA-600 [17087] errors. 


Workaround 
    Try reducing the number of hard parses. 


Search on: kglivr <- kksfbc <- kkspsc0 <- kksParseCursor 
Found: 
ORA-07445 [kksumc] And ORA-600 [17087] During Insert Into Table ( Doc ID 580347.1 ) 


解决方案：
打Patch 7706062
补丁地址：
https://updates.oracle.com/download/7706062.html 




官档：
Bug 7706062 - OERI [17087] following concurrent hard parses on same cursor ( Doc ID 7706062.8 ) 
ORA-07445 [kksumc] And ORA-600 [17087] During Insert Into Table ( Doc ID 580347.1 )