系统内存不足导致oracle进程被误杀terminating the instance due to error 822
<div>
今天收到一个报警邮件,oracle进程已经不存在了
</div>
<div style="font-family:宋体, Tahoma;font-size:14px;white-space:normal;widows:auto;background-color:#FFFFFF;">
Alarm Time:2015-09-21 17:45:38
</div>
<div style="font-family:宋体, Tahoma;font-size:14px;white-space:normal;widows:auto;background-color:#FFFFFF;">
Trigger: Alive xyxdb_oa
</div>
<div style="font-family:宋体, Tahoma;font-size:14px;white-space:normal;widows:auto;background-color:#FFFFFF;">
Trigger status: PROBLEM
</div>
<div style="font-family:宋体, Tahoma;font-size:14px;white-space:normal;widows:auto;background-color:#FFFFFF;">
Trigger severity: High
</div>
<div style="font-family:宋体, Tahoma;font-size:14px;white-space:normal;widows:auto;background-color:#FFFFFF;">
Trigger URL:
</div>
<div style="font-family:宋体, Tahoma;font-size:14px;white-space:normal;widows:auto;background-color:#FFFFFF;">
Item values:
</div>
<div style="font-family:宋体, Tahoma;font-size:14px;white-space:normal;widows:auto;background-color:#FFFFFF;">
1. Alive (x.x.x.x:alive): 0
</div>
<div style="white-space:normal;background-color:#FFFFFF;">
<span style="font-size:14px;"><span style="font-family:宋体, Tahoma;">2. *UNKNOWN* (</span></span>x.x.x.x <span style="font-size:14px;"><span style="font-family:宋体, Tahoma;">:*UNKNOWN*): *UNKNOWN*</span></span>
</div>
<div style="font-family:宋体, Tahoma;font-size:14px;white-space:normal;widows:auto;background-color:#FFFFFF;">
Original event ID: 760121
</div>
<div>
<br />
</div>
<div>
<br />
</div>
<div>
查看到alert日志
</div>
<div>
System state dump requested by (instance=1, osid=2044 (PMON)), summary=[abnormal instance termination].<br />
System State dumped to trace file /u01/app/oracle/diag/rdbms/xyxdbp/xyxdb/trace/xyxdb_diag_2062_20150921174417.trc<br />
Mon Sep 21 17:44:18 2015<br />
PMON (ospid: 2044): terminating the instance due to error 822<br />
Dumping diagnostic data in directory=[cdmp_20150921174417], requested by (instance=1, osid=2044 (PMON)), summary=[abnormal instance termination].<br />
Instance terminated by PMON, pid = 2044<br />
Mon Sep 21 17:46:39 2015<br />
Starting ORACLE instance (normal)<br />
************************ Large Pages Information *******************<br />
Per process system memlock (soft) limit = 64 KB<br />
<br />
Total Shared Global Region in Large Pages = 0 KB (0%)<br />
<br />
Large Pages used by this instance: 0 (0 KB)<br />
Large Pages unused system wide = 0 (0 KB)<br />
Large Pages configured system wide = 0 (0 KB)<br />
Large Page size = 2048 KB<br />
<br />
RECOMMENDATION:<br />
Total System Global Area size is 3282 MB. For optimal performance,<br />
prior to the next instance restart:<br />
1. Increase the number of unused large pages by
</div>
<div>
at least 1641 (page size 2048 KB, total size 3282 MB) system wide to
</div>
<div>
<br />
</div>
<div>
<br />
</div>
<div>
RECOMMENDATION:<br />
Total System Global Area size is 3282 MB. For optimal performance,<br />
prior to the next instance restart:<br />
1. Increase the number of unused large pages by<br />
at least 1641 (page size 2048 KB, total size 3282 MB) system wide to<br />
get 100% of the System Global Area allocated with large pages<br />
2. Large pages are automatically locked into physical memory.<br />
Increase the per process memlock (soft) limit to at least 3290 MB to lock<br />
100% System Global Area's large pages into physical memory<br />
********************************************************************<br />
LICENSE_MAX_SESSION = 0<br />
LICENSE_SESSIONS_WARNING = 0<br />
Initial number of CPU is 6<br />
Number of processor cores in the system is 6<br />
Number of processor sockets in the system is 1<br />
CELL communication is configured to use 0 interface(s):<br />
CELL IP affinity details:<br />
NUMA status: non-NUMA system<br />
cellaffinity.ora status: N/A<br />
CELL communication will use 1 IP group(s):
</div>
<div>
Grp 0:
</div>
<div>
<br />
</div>
<div>
<br />
</div>
<div>
<br />
</div>
<div>
[root@OA01-1-24 scripts]# cat /proc/50966/oom_
</div>
<div>
oom_adj oom_score oom_score_adj <br />
[root@OA01-1-24 scripts]# cat /proc/50966/oom_adj<br />
0<br />
[root@OA01-1-24 scripts]# vim oomscore.sh<br />
[root@OA01-1-24 scripts]# chmod u+x oomscore.sh<br />
[root@OA01-1-24 scripts]# ./oomscore.sh<br />
63 37608 /usr/bin/java -Djava.util.logging.config.file=/usr<br />
31 51010 ora_mman_xyxdb<br />
20 37579 /usr/bin/java -Djava.util.logging.config.file=/usr<br />
16 51938 /usr/java/jdk1.7.0_79/jre/bin/java -Djava.util.log<br />
14 51496 oraclexyxdb (LOCAL=NO)<br />
13 51026 ora_smon_xyxdb<br />
8 51167 oraclexyxdb (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROT<br />
7 51034 ora_mmon_xyxdb<br />
7 51014 ora_dbw0_xyxdb
</div>
<div>
6 51480 oraclexyxdb (LOCAL=NO)
</div>
<div>
<br />
</div>
<div>
查询系统日志
</div>
<div>
Sep 21 17:44:15 OA01-1-24 kernel: [39519] 500 39519 900699 5848 2 0 0 oracle<br />
Sep 21 17:44:15 OA01-1-24 kernel: [39521] 500 39521 900699 5877 5 0 0 oracle<br />
Sep 21 17:44:15 OA01-1-24 kernel: [42514] 500 42514 900846 10963 1 0 0 oracle<br />
Sep 21 17:44:15 OA01-1-24 kernel: [42578] 500 42578 900706 9012 1 0 0 oracle<br />
Sep 21 17:44:15 OA01-1-24 kernel: [43519] 0 43519 24998 1489 5 0 0 sshd<br />
Sep 21 17:44:15 OA01-1-24 kernel: [43533] 0 43533 14309 550 5 0 0 sftp-server<br />
Sep 21 17:44:15 OA01-1-24 kernel: [43557] 0 43557 14432 671 5 0 0 sftp-server<br />
Sep 21 17:44:15 OA01-1-24 kernel: [44331] 89 44331 20234 861 2 0 0 pickup<br />
Sep 21 17:44:15 OA01-1-24 kernel: [44491] 0 44491 1107908 148835 4 0 0 java<br />
Sep 21 17:44:15 OA01-1-24 kernel: [44684] 500 44684 900015 4658 0 0 0 oracle<br />
Sep 21 17:44:15 OA01-1-24 kernel: [45199] 500 45199 900699 5525 3 0 0 oracle<br />
Sep 21 17:44:15 OA01-1-24 kernel: [45201] 500 45201 900699 5548 4 0 0 oracle<br />
Sep 21 17:44:15 OA01-1-24 kernel: [45203] 500 45203 900704 8184 5 0 0 oracle<br />
Sep 21 17:44:15 OA01-1-24 kernel: [45211] 500 45211 900699 5506 0 0 0 oracle<br />
Sep 21 17:44:15 OA01-1-24 kernel: [45213] 500 45213 900699 5504 4 0 0 oracle<br />
Sep 21 17:44:15 OA01-1-24 kernel: [45901] 0 45901 1051478 117538 2 0 0 java<br />
Sep 21 17:44:15 OA01-1-24 kernel: [45943] 500 45943 900956 7194 0 0 0 oracle<br />
Sep 21 17:44:15 OA01-1-24 kernel: [45945] 500 45945 900315 5444 1 0 0 oracle<br />
Sep 21 17:44:15 OA01-1-24 kernel: [45947] 500 45947 900315 5423 5 0 0 oracle<br />
Sep 21 17:44:15 OA01-1-24 kernel: [46232] 0 46232 25226 152 4 0 0 sleep<br />
Sep 21 17:44:15 OA01-1-24 kernel: Out of memory: Kill process 2074 (oracle) score 125 or sacrifice child
</div>
<div>
Sep 21 17:44:15 OA01-1-24 kernel: Killed process 2074, UID 500, (oracle) total-vm:3600064kB, anon-rss:3444kB, file-rss:1510892kB
</div>
<div>
<br />
</div>
<div>
<span style="color:#333333;font-family:'Hiragino Sans GB', 'Microsoft YaHei', 微软雅黑, Georgia, tahoma, arial, simsun, 宋体;font-size:13px;white-space:normal;background-color:#FFFFFF;">通常是因为某时刻应用程序大量请求内存导致系统内存不足造成的，这通常会触发 Linux 内核里的 Out of Memory (OOM) killer，OOM killer 会杀掉某个进程以腾出内存留给系统用，不致于让系统立刻崩溃。</span>
</div>
<div>
后来查看到开发人员在这台db服务器启用了两个tomcat应用，由于程序故障导致大量内存使用
</div>
<div>
至于oom killer 原理可以参阅<a href="http://www.vpsee.com/2013/10/how-to-configure-the-linux-oom-killer/">http://www.vpsee.com/2013/10/how-to-configure-the-linux-oom-killer/</a>这篇文章很详细
</div>
<div>
oracle有一部分相关文档
</div>
<div>
<a href="http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html"></a><a href="http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html">http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html</a>
</div>
<div>
<br />
</div>
<div>
我们可以配置内核参数来防止进程被杀
</div>
<div>
通过脚本找出最容易被杀的进程
</div>
<div>
# vi oomscore.sh<br />
#!/bin/bash<br />
for proc in $(find /proc -maxdepth 1 -regex '/proc/[0-9]+'); do<br />
printf "%2d %5d %s\n" \<br />
"$(cat $proc/oom_score)" \<br />
"$(basename $proc)" \<br />
"$(cat $proc/cmdline | tr '\0' ' ' | head -c 50)"
</div>
<div>
done 2>/dev/null | sort -nr | head -n 10
</div>
<div>
<br />
</div>
<div>
<br />
</div>
<div>
[root@OA01-1-24 scripts]# ./oomscore.sh<br />
63 37608 /usr/bin/java -Djava.util.logging.config.file=/usr<br />
31 51010 ora_mman_xyxdb<br />
20 37579 /usr/bin/java -Djava.util.logging.config.file=/usr<br />
16 51938 /usr/java/jdk1.7.0_79/jre/bin/java -Djava.util.log<br />
14 51496 oraclexyxdb (LOCAL=NO)<br />
13 51026 ora_smon_xyxdb<br />
8 51167 oraclexyxdb (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROT<br />
7 51034 ora_mmon_xyxdb<br />
7 51014 ora_dbw0_xyxdb
</div>
<div>
6 51480 oraclexyxdb (LOCAL=NO)
</div>
<div>
[root@OA01-1-24 scripts]# cat /proc/51034/oom_score<br />
7<br />
[root@OA01-1-24 scripts]# cat /proc/51034/oom_score_adj<br />
0<br />
[root@OA01-1-24 scripts]# echo -15 >/proc/51034/oom_adj<br />
[root@OA01-1-24 scripts]# cat /proc/51034/oom_score<br />
1<br />
[root@OA01-1-24 scripts]# cat /proc/51026/oom_adj<br />
0<br />
[root@OA01-1-24 scripts]# cat /proc/51026/oom_score<br />
13<br />
[root@OA01-1-24 scripts]# echo -15 >/proc/51026/oom_adj<br />
[root@OA01-1-24 scripts]# cat /proc/51026/oom_adj<br />
-15<br />
[root@OA01-1-24 scripts]# cat /proc/51026/oom_score<br />
1
</div>
<div>
[root@OA01-1-24 scripts]# echo -15 >/proc/51010/oom_adj
</div>
<div>
[root@OA01-1-24 scripts]# cat /proc/51010/oom_score<br />
1<br />
[root@OA01-1-24 scripts]# ./<br />
alertbyday.sh oracle_cron.sh sendrman.py updatedb/ <br />
installora/ rmanbackup.sh sync_date.sh uploadbackup.sh <br />
oomscore.sh senderrorlog.py tablespace_monitor.py <br />
[root@OA01-1-24 scripts]# ./oomscore.sh<br />
63 37608 /usr/bin/java -Djava.util.logging.config.file=/usr<br />
20 37579 /usr/bin/java -Djava.util.logging.config.file=/usr<br />
16 51938 /usr/java/jdk1.7.0_79/jre/bin/java -Djava.util.log<br />
14 51496 oraclexyxdb (LOCAL=NO)<br />
8 51167 oraclexyxdb (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROT<br />
7 51014 ora_dbw0_xyxdb<br />
6 51480 oraclexyxdb (LOCAL=NO)<br />
5 52007 oraclexyxdb (LOCAL=NO)<br />
5 51474 oraclexyxdb (LOCAL=NO)
</div>
<div>
5 51466 oraclexyxdb (LOCAL=NO)
</div>
<div>
<br />
</div>
<div>
后来还检查到一个问题，关于swap使用配置
</div>
<div>
[root@OA01-1-24 scripts]# cat /proc/sys/vm/swappiness
</div>
<div>
0
</div>
<div>
这里0代表不使用swap
</div>
<div>
系统工程师更改的时候没有注意，oracle最好不要关掉swap
</div>
<div>
重新修改
</div>
<div>
[root@OA01-1-24 scripts]# cat /proc/sys/vm/swappiness
</div>
<div>
60
</div>
<div>
总结:DB服务器尽量专用，不然会出现很多意想不到事儿
</div>
来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/30527248/viewspace-2097123/ ，如需转载，请注明出处，否则将追究法律责任。
上一篇：
没有了~
下一篇：
没有了~
请登录后发表评论
登录
全部评论
