ITPub博客

首页 > Linux操作系统 > Linux操作系统 > 长时间latch free等待——记一次系统异常的诊断过程

长时间latch free等待——记一次系统异常的诊断过程

原创 Linux操作系统 作者:kvsion 时间:2009-08-27 22:18:27 0 删除 编辑

出自:http://yangtingkun.itpub.net/post/468/273340

今天发现一个报表数据库中SQL运行异常,简单记录一下问题的诊断和解决过程。


问题是在检查ALERT文件时发现的,一个过程运行时间太长而出现了ORA-1555错误。

错误信息:

ORA-01555 caused by SQL statement below (Query Duration=38751 sec, SCN: 0x0000.fe5b584a):
INSERT INTO MAN_ORDER_ITEM (
ID,
REQUEST_QTY,
SALER_ID,
PRODUCT_ID,
UNIT_PRICE,
CREATE_DATE,
ANSWER_DATE,
BUYER_ID
)
SELECT
A.RECORD_ID,
A.REQUEST_QTY,
A.SALER_ORGID,
A.PRODUCT_ID,
A.UNIT_PRICE,
A.CREATE_DATE,
NULL,
A.BUYER_ORGID
FROM ORD_ORDER_ITEM A
WHERE A.CREATE_DATE >= TO_DATE('2004-01-01 0:0:0', 'YYYY-MM-DD HH24:MI:SS')
AND A.CREATE_DATE < TRUNC(SYSDATE)
AND EXISTS (SELECT 1 FROM MAN_PRODUCT WHERE ID = A.PRODUCT_ID)
AND EXISTS (SELECT 1 FROM MAN_DEALER WHERE ID = A.SALER_ORGID)
AND EXISTS (SELECT 1 FROM MAN_BUYER WHERE ID = A.BUYER_ORGID)

由于这是个JOB调用,在失败后JOB会自动重试,于是从DBA_JOBS_RUNNING中查看相关的JOBSESSION信息。

SQL> SELECT SID, JOB FROM DBA_JOBS_RUNNING;

SID JOB
---------- ----------
70 208

检查这个SESSION目录在执行什么SQL

SQL> SELECT SQL_TEXT FROM V$SQL SQL, V$SESSION S
2 WHERE SQL.HASH_VALUE = S.SQL_HASH_VALUE
3 AND SQL.ADDRESS = S.SQL_ADDRESS
4 AND S.SID = 70;

SQL_TEXT
------------------------------------------------------------------------------
INSERT INTO MAN_ORDER_ITEM ( ID, REQUEST_QTY, SALER_ID, PRODUCT_ID, UNIT_PRICE, CREATE_DATE, ANSWER_DATE,
BUYER_ID ) SELECT A.RECORD_ID, A.REQUEST_QTY, A.SALER_ORGID, A.PRODUCT_ID, A.UNIT_PRICE, A.CREATE_DATE,
NULL, A.BUYER_ORGID FROM ORD_ORDER_ITEM A WHERE A.CREATE_DATE >= TO_DATE('2004-01-01 0:0:0', 'YYYY-MM-DD HH24:MI:S
S') AND A.CREATE_DATE < TRUNC(SYSDATE) AND EXISTS (SELECT 1 FROM MAN_PRODUCT WHERE ID = A.PRODUCT_ID) AND EXISTS (SEL
ECT 1 FROM MAN_DEALER WHERE ID = A.SALER_ORGID) AND EXISTS (SELECT 1 FROM MAN_BUYER WHERE ID = A.BUYER_ORGID)

SQL上看,就是刚才失败的那个SQL语句,那么看看SESSION在等待什么:

SQL> SELECT SID, EVENT, P1TEXT, P1RAW, P2TEXT, P2, SECONDS_IN_WAIT FROM V$SESSION_WAIT
2 WHERE SID = 70;

SID EVENT P1TEXT P1RAW P2TEXT P2 SECONDS_IN_WAIT
------- -------------- -------- ---------------- ------- ----- ---------------
70 latch free address 00000004125AB718 number 98 330

通过观察发现,Session的等待事件一直是LATCH FREE。第一个感觉是可能和其他的进程产生了争用。

查询一下Oracle等待的具体latch的类型。

SQL> SELECT LATCH#, NAME FROM V$LATCH WHERE LATCH# = 98;

LATCH# NAME
---------- ----------------------------------------------------------------
98 cache buffers chains

而查询V$LOCKV$LATCHHOLDER视图,发现没有其他的进程对JOB运行构成影响:

SQL> SELECT SID, TYPE, ID1, ID2, LMODE, REQUEST, CTIME, BLOCK
2 FROM V$LOCK
3 WHERE SID > 8;

SID TY ID1 ID2 LMODE REQUEST CTIME BLOCK
---------- -- ---------- ---------- ---------- ---------- ---------- ----------
70 TM 35258 0 3 0 12072 0
70 JQ 0 208 6 0 12155 0

SQL> SELECT * FROM V$LATCHHOLDER;

no rows selected

SQL> SELECT * FROM V$LATCHHOLDER;

no rows selected

SQL> SELECT * FROM V$LATCHHOLDER;

no rows selected

SQL> SELECT * FROM V$LATCHHOLDER;

PID SID LADDR NAME
---------- ---------- ---------------- ------------------------------
15 70 0000000412564D98 cache buffers chains

SQL> SELECT * FROM V$LATCHHOLDER;

no rows selected

可以看到,并没有其他对象影响JOB进程。

由于等待事件是LATCH FREE,怀疑和系统本身的问题有关。

通过下面的脚本可以看到,目前正在等待的这个子LATCH的信息:

SQL> SELECT ADDR, LATCH#, CHILD#, NAME FROM V$LATCH_CHILDREN
2 WHERE ADDR IN (SELECT P1RAW FROM V$SESSION_WAIT WHERE SID = 70);

ADDR LATCH# CHILD# NAME
---------------- ---------- ---------- ---------------------------------------
0000000412550518 98 327 cache buffers chains

观察LATCH_MISSES的信息:

SQL> COL PARENT_NAME FORMAT A20
SQL> COL WHERE FORMAT A35
SQL> SELECT *
2 FROM
3 (
4 SELECT PARENT_NAME, "WHERE", SLEEP_COUNT, WTR_SLP_COUNT, LONGHOLD_COUNT
5 FROM V$LATCH_MISSES
6 WHERE PARENT_NAME = 'cache buffers chains'
7 ORDER BY SLEEP_COUNT + WTR_SLP_COUNT + LONGHOLD_COUNT DESC
8 )
9 WHERE ROWNUM < 20;

PARENT_NAME WHERE SLEEP_COUNT WTR_SLP_COUNT LONGHOLD_COUNT
-------------------- ------------------------------- ----------- ------------- --------------
cache buffers chains kcbgtcr: kslbegin excl 1202658 884364 906374
cache buffers chains kcbrls: kslbegin 480030 852471 335799
cache buffers chains kcbzwb 95994 90482 84373
cache buffers chains kcbgtcr: kslbegin shared 89385 84911 62640
cache buffers chains kcbgtcr: fast path 69352 113120 51014
cache buffers chains kcbchg: kslbegin: bufs not pinned 86476 51934 58687
cache buffers chains kcbzsc 76224 55 75045
cache buffers chains kcbbxsv 37425 8306 35681
cache buffers chains kcbchg: kslbegin: call CR func 1337 20943 745
cache buffers chains kcbzib: finish free bufs 685 18544 432
cache buffers chains kcbcge 168 8767 107
cache buffers chains kcbgcur: kslbegin 344 4974 203
cache buffers chains kcbget: pin buffer 542 4508 383
cache buffers chains kcbgtcr 2400 395 1769
cache buffers chains kcbbic1 14 4015 11
cache buffers chains kcbzgb: scan from tail. nowait 2048 0 1896
cache buffers chains kcbbic2 38 2920 32
cache buffers chains kcbzib: multi-block read: nowait 1502 0 970
cache buffers chains kcbnew 497 331 289

19 rows selected.

感觉问题和热点块有关,那么看看到底是哪些块出现的问题:

SQL> SELECT OBJ, OBJECT_NAME, TCH, TIM
2 FROM X$BH A, DBA_OBJECTS B
3 WHERE HLADDR IN (SELECT P1RAW FROM V$SESSION_WAIT WHERE SID = 70)
4 AND A.OBJ = B.DATA_OBJECT_ID;

OBJ OBJECT_NAME TCH TIM
---------- ------------------------------ ---------- ----------
109 I_OBJAUTH2 0 0
45761 STATS$SQLTEXT_PK 1 1174381376
.
.
.
62275 ORD_ORDER_ITEM_ZJ 1 1174380112
62275 ORD_ORDER_ITEM_ZJ 1 1174380085
62275 ORD_ORDER_ITEM_ZJ 1 1174380088
62275 ORD_ORDER_ITEM_ZJ 1 1174380097
62275 ORD_ORDER_ITEM_ZJ 1 1174380100
62275 ORD_ORDER_ITEM_ZJ 1 1174380103.
.
.
.
200403 ORD_ORDER_ITEM 6 1174381870
200403 ORD_ORDER_ITEM 6 1174381871
200403 ORD_ORDER_ITEM 6 1174381870
200403 ORD_ORDER_ITEM 6 1174381870
200403 ORD_ORDER_ITEM 6 1174381871
200403 ORD_ORDER_ITEM 6 1174381870
200403 ORD_ORDER_ITEM 6 1174381870
200403 ORD_ORDER_ITEM 6 1174381871
200403 ORD_ORDER_ITEM 6 1174381871
200403 ORD_ORDER_ITEM 6 1174381871
200403 ORD_ORDER_ITEM 6 1174381870
200403 ORD_ORDER_ITEM 6 1174381870
.
.
.
62275 ORD_ORDER_ITEM_ZJ 678 1174381878
62275 ORD_ORDER_ITEM_ZJ 1 1174380060
62275 ORD_ORDER_ITEM_ZJ 1 1174380091
62126 ORD_ORDER_ITEM_CEN 0 0
62126 ORD_ORDER_ITEM_CEN 0 0
62126 ORD_ORDER_ITEM_CEN 0 0
62126 ORD_ORDER_ITEM_CEN 0 0
45772 STATS$UNDOSTAT 0 0
45772 STATS$UNDOSTAT 0 0

125 rows selected.

从这些热点块的所属对象来看,大部分都是那个长时间运行的SQL访问的。难道转了一个大圈,结果是执行计划的问题?

检查该SQL的执行计划:

SQL> EXPLAIN PLAN FOR
2 INSERT INTO MAN_ORDER_ITEM (
3 ID,
4 REQUEST_QTY,
5 SALER_ID,
6 PRODUCT_ID,
7 UNIT_PRICE,
8 CREATE_DATE,
9 ANSWER_DATE,
10 BUYER_ID
11 )
12 SELECT
13 A.RECORD_ID,
14 A.REQUEST_QTY,
15 A.SALER_ORGID,
16 A.PRODUCT_ID,
17 A.UNIT_PRICE,
18 A.CREATE_DATE,
19 NULL,
20 A.BUYER_ORGID
21 FROM ORD_ORDER_ITEM A
22 WHERE A.CREATE_DATE >= TO_DATE('2004-01-01 0:0:0', 'YYYY-MM-DD HH24:MI:SS')
23 AND A.CREATE_DATE < TRUNC(SYSDATE)
24 AND EXISTS (SELECT 1 FROM MAN_PRODUCT WHERE ID = A.PRODUCT_ID)
25 AND EXISTS (SELECT 1 FROM MAN_DEALER WHERE ID = A.SALER_ORGID)
26 AND EXISTS (SELECT 1 FROM MAN_BUYER WHERE ID = A.BUYER_ORGID)
27 ;

Explained.

SQL> SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY());

PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost |
-------------------------------------------------------------------------------------------------
| 0 | INSERT STATEMENT | | 1 | 212 | 37 |
|* 1 | FILTER | | | | |
| 2 | NESTED LOOPS | | 1 | 212 | 37 |
| 3 | MERGE JOIN CARTESIAN | | 1 | 76 | 12 |
| 4 | MERGE JOIN CARTESIAN | | 1 | 51 | 9 |
| 5 | SORT UNIQUE | | | | |
| 6 | INDEX FAST FULL SCAN | PK_MAN_PRODUCT | 1 | 26 | 3 |
| 7 | BUFFER SORT | | 8138 | 198K| 6 |
| 8 | SORT UNIQUE | | | | |
| 9 | INDEX FAST FULL SCAN | PK_MAN_BUYER | 8138 | 198K| 3 |
| 10 | BUFFER SORT | | 14238 | 347K| 9 |
| 11 | SORT UNIQUE | | | | |
| 12 | INDEX FAST FULL SCAN | PK_MAN_DEALER | 14238 | 347K| 3 |
|* 13 | VIEW | ORD_ORDER_ITEM | 1 | 136 | 37 |
| 14 | UNION-ALL PARTITION | | | | |
|* 15 | FILTER | | | | |
|* 16 | TABLE ACCESS BY INDEX ROWID| ORD_ORDER_ITEM_CEN | 1 | 116 | 14 |
| 17 | AND-EQUAL | | | | |
|* 18 | INDEX RANGE SCAN | TU_ORD_ORD_ITEM_PRODUCT_ID | | | |
|* 19 | INDEX RANGE SCAN | TU_ORD_ORDER_ITEM_SALER | | | |
|* 20 | FILTER | | | | |
|* 21 | TABLE ACCESS BY INDEX ROWID| ORD_ORDER_ITEM_ZJ | 1 | 116 | 2 |
|* 22 | INDEX RANGE SCAN | TU_ORD_ORD_ITEM_PRODUCT_ID1 | 179 | | 1 |
-------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

1 - filter(TRUNC(SYSDATE@!)>TO_DATE('2004-01-01 00:00:00', 'yyyy-mm-dd hh24:mi:ss') AND
TO_DATE('2004-01-01 00:00:00', 'yyyy-mm-dd hh24:mi:ss')13 - filter("MAN_PRODUCT"."ID"="A"."PRODUCT_ID" AND "MAN_DEALER"."ID"="A"."SALER_ORGID" AND
"MAN_BUYER"."ID"="A"."BUYER_ORGID")
15 - filter(TRUNC(SYSDATE@!)>TO_DATE('2004-01-01 00:00:00', 'yyyy-mm-dd hh24:mi:ss') AND
TO_DATE('2004-01-01 00:00:00', 'yyyy-mm-dd hh24:mi:ss')16 - filter("ORD_ORDER_ITEM_CEN"."BUYER_ORGID"="MAN_BUYER"."ID" AND
"ORD_ORDER_ITEM_CEN"."SALER_ORGID"="MAN_DEALER"."ID" AND
"ORD_ORDER_ITEM_CEN"."PRODUCT_ID"="MAN_PRODUCT"."ID" AND
"ORD_ORDER_ITEM_CEN"."CREATE_DATE">=TO_DATE('2004-01-01 00:00:00', 'yyyy-mm-dd hh24:mi:ss') AND
"ORD_ORDER_ITEM_CEN"."CREATE_DATE"18 - access("ORD_ORDER_ITEM_CEN"."PRODUCT_ID"="MAN_PRODUCT"."ID")
19 - access("ORD_ORDER_ITEM_CEN"."SALER_ORGID"="MAN_DEALER"."ID")
20 - filter(TRUNC(SYSDATE@!)>TO_DATE('2004-01-01 00:00:00', 'yyyy-mm-dd hh24:mi:ss') AND
TO_DATE('2004-01-01 00:00:00', 'yyyy-mm-dd hh24:mi:ss')21 - filter("ORD_ORDER_ITEM_ZJ"."BUYER_ORGID"="MAN_BUYER"."ID" AND
"ORD_ORDER_ITEM_ZJ"."SALER_ORGID"="MAN_DEALER"."ID" AND
"ORD_ORDER_ITEM_ZJ"."CREATE_DATE">=TO_DATE('2004-01-01 00:00:00', 'yyyy-mm-dd hh24:mi:ss') AND
"ORD_ORDER_ITEM_ZJ"."CREATE_DATE"22 - access("ORD_ORDER_ITEM_ZJ"."PRODUCT_ID"="MAN_PRODUCT"."ID")

Note: cpu costing is off

54 rows selected.

晕倒,怎么会产生一个这样的执行计划。不看别的,只看上面两个MERGE JOIN CARTESIAN就已经足够了。

这下总算明白系统中为什么拼命在等待LATCH FREE了。Oracle按照这个执行计划执行,不知道需要多大的内存来容纳中间结果集,所以只能不停的将中间结果装入内存,然后清空内存,一直不断的持续这个过程。

明白了系统目前的运行状况,那么Oracle为什么会产生这种恐怖的执行计划呢?从执行计划信息上看,使用的CBOOracle不会无缘无故采用这种执行计划的。

首先怀疑的是统计信息出现了偏差。

SQL> SELECT TABLE_NAME, NUM_ROWS FROM USER_TABLES
2 WHERE TABLE_NAME IN ('MAN_PRODUCT', 'MAN_BUYER', 'MAN_DEALER');

TABLE_NAME NUM_ROWS
------------------------------ ----------
MAN_BUYER 8138
MAN_DEALER 14238
MAN_PRODUCT 0

SQL> SELECT COUNT(*) FROM MAN_PRODUCT;

COUNT(*)
----------
91750

已经很明显了,MAN_PRODUCT的统计信息有误。正是由于Oracle认为MAN_PRODUCT的记录为0,所以选择了MERGE JOIN的方式,这样可以最迅速的得到最终的结果——0条记录。但是MAN_PRODUCT的记录数实际上并不为0,而是有将近10万的记录。这就是导致问题产生的真正原因。

其实问题到这里并没有完。Oracle产生错误的统计信息也是有原因的。首先这个脚本会首先清空相关表的数据,然后重新生成。而在前一次执行这个过程的时候,中途失败了。导致MAN_PRODUCT表中没有数据。而每周运行一次的收集统计信息的JOBMAN_PRODUCT表的0记录的统计信息记录了下来。

再次运行脚本的时候,虽然将记录写入到MAN_PRODUCT表中,但是统计信息并没有更新,因此导致了这个问题。

了解了问题的产生原因,解决起来就十分容易了,收集MAN_PRODUCT表的统计信息并检查执行计划,杀掉运行中JOB,重新启动JOB就可以了。

SQL> EXEC DBMS_STATS.GATHER_TABLE_STATS(USER, 'MAN_PRODUCT', CASCADE => TRUE)

PL/SQL procedure successfully completed.

SQL> EXPLAIN PLAN FOR
2 INSERT INTO MAN_ORDER_ITEM (
3 ID,
4 REQUEST_QTY,
5 SALER_ID,
6 PRODUCT_ID,
7 UNIT_PRICE,
8 CREATE_DATE,
9 ANSWER_DATE,
10 BUYER_ID
11 )
12 SELECT
13 A.RECORD_ID,
14 A.REQUEST_QTY,
15 A.SALER_ORGID,
16 A.PRODUCT_ID,
17 A.UNIT_PRICE,
18 A.CREATE_DATE,
19 NULL,
20 A.BUYER_ORGID
21 FROM ORD_ORDER_ITEM A
22 WHERE A.CREATE_DATE >= TO_DATE('2004-01-01 0:0:0', 'YYYY-MM-DD HH24:MI:SS')
23 AND A.CREATE_DATE < TRUNC(SYSDATE)
24 AND EXISTS (SELECT 1 FROM MAN_PRODUCT WHERE ID = A.PRODUCT_ID)
25 AND EXISTS (SELECT 1 FROM MAN_DEALER WHERE ID = A.SALER_ORGID)
26 AND EXISTS (SELECT 1 FROM MAN_BUYER WHERE ID = A.BUYER_ORGID)
27 ;

Explained.

SQL> SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY());

PLAN_TABLE_OUTPUT
----------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost |
--------------------------------------------------------------------------------------------------------------
| 0 | INSERT STATEMENT | | 32977 | 6795K| | 72667 |
|* 1 | FILTER | | | | | |
| 2 | NESTED LOOPS SEMI | | 32977 | 6795K| | 72667 |
| 3 | NESTED LOOPS SEMI | | 32977 | 5989K| | 72667 |
|* 4 | HASH JOIN SEMI | | 32977 | 5184K| 4768K| 72667 |
| 5 | VIEW | ORD_ORDER_ITEM | 32977 | 4379K| | 72502 |
| 6 | UNION-ALL | | | | | |
|* 7 | FILTER | | | | | |
| 8 | TABLE ACCESS BY INDEX ROWID| ORD_ORDER_ITEM_CEN | 5797K| 641M| | 479 |
|* 9 | INDEX RANGE SCAN | TU_ORD_ORDER_ITEM_CREATE_DATE | 5797K| | | 16 |
|* 10 | FILTER | | | | | |
| 11 | TABLE ACCESS BY INDEX ROWID| ORD_ORDER_ITEM_ZJ | 2464K| 272M| | 643 |
|* 12 | INDEX RANGE SCAN | TU_ORD_ORDER_ITEM_CREATE_DATE1 | 2464K| | | 21 |
| 13 | INDEX FAST FULL SCAN | PK_MAN_PRODUCT | 91750 | 2239K| | 39 |
|* 14 | INDEX UNIQUE SCAN | PK_MAN_BUYER | 8138 | 198K| | |
|* 15 | INDEX UNIQUE SCAN | PK_MAN_DEALER | 14238 | 347K| | |
---------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

1 - filter(TRUNC(SYSDATE@!)>TO_DATE('2004-01-01 00:00:00', 'yyyy-mm-dd hh24:mi:ss') AND
TO_DATE('2004-01-01 00:00:00', 'yyyy-mm-dd hh24:mi:ss')4 - access("MAN_PRODUCT"."ID"="A"."PRODUCT_ID")
7 - filter(TRUNC(SYSDATE@!)>TO_DATE('2004-01-01 00:00:00', 'yyyy-mm-dd hh24:mi:ss') AND
TO_DATE('2004-01-01 00:00:00', 'yyyy-mm-dd hh24:mi:ss')9 - access("ORD_ORDER_ITEM_CEN"."CREATE_DATE">=TO_DATE('2004-01-01 00:00:00', 'yyyy-mm-dd hh24:mi:ss') AND
"ORD_ORDER_ITEM_CEN"."CREATE_DATE"10 - filter(TRUNC(SYSDATE@!)>TO_DATE('2004-01-01 00:00:00', 'yyyy-mm-dd hh24:mi:ss') AND
TO_DATE('2004-01-01 00:00:00', 'yyyy-mm-dd hh24:mi:ss')12 - access("ORD_ORDER_ITEM_ZJ"."CREATE_DATE">=TO_DATE('2004-01-01 00:00:00', 'yyyy-mm-dd hh24:mi:ss') AND
"ORD_ORDER_ITEM_ZJ"."CREATE_DATE"14 - access("MAN_BUYER"."ID"="A"."BUYER_ORGID")
15 - access("MAN_DEALER"."ID"="A"."SALER_ORGID")

Note: cpu costing is off

40 rows selected.

SQL> SELECT SPID FROM V$PROCESS WHERE ADDR IN (SELECT PADDR FROM V$SESSION WHERE SID = 70);

SPID
------------
27488

找到JOB运行的操作系统进程,然后通过操作系统命令kill -9来杀掉进程。JOB运行的会话很难通过ALTER SYSTEM KILL SESSION语句来杀掉,因此选择使用操作系统命令的方式。

SQL> host
$ ps -ef|grep 27488
oracle 28672 28671 0 18:07:20 pts/2 0:00 grep 27488
oracle 27488 1 13 12:46:56 ? 317:21 ora_j000_repdb01
$ kill -9 27488
$ exit

检查SESSIONJOB状态,确认JOB重新启动。

至此,问题解决。其实当时发现问题的时候,有两条路可以选择,一方面从系统运行的情况入手,也就是这篇文章中选择的方式。另一方面是直接从SQL语句入手,先检查执行计划。

由于这个SQL以前过多次,都是正常的,所以没有想到执行计划会发生那么大的变化。因此,一直从第一方面着手来诊断问题。所幸殊途同归,虽然绕了一个大圈子,最终还是找到问题的所在。

来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/11269677/viewspace-613444/,如需转载,请注明出处,否则将追究法律责任。

上一篇: 自治事务
下一篇: ROLLUP和CUBE
请登录后发表评论 登录
全部评论

注册时间:2008-09-20

  • 博文量
    17
  • 访问量
    23135