OS临时端口耗尽导致无法建立数据库连接
1. 报错信息
|
zhjs3$[/home/oracle]lsnrctl status
LSNRCTL for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production on 16-JAN-2013 16:12:43
Copyright (c) 1991, 2011, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=133.37.253.82)(PORT=1521))) TNS-12547: TNS:lost contact TNS-12560: TNS:protocol adapter error TNS-00517: Lost contact IBM/AIX RISC System/6000 Error: 32: Broken pipe Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC))) TNS-12547: TNS:lost contact TNS-12560: TNS:protocol adapter error TNS-00517: Lost contact IBM/AIX RISC System/6000 Error: 32: Broken pipe
|
|
zhjs3$[/home/oracle]tnsping `hostname`
TNS Ping Utility for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production on 16-JAN-2013 16:12:55
Copyright (c) 1997, 2011, Oracle. All rights reserved.
Used parameter files:
Used HOSTNAME adapter to resolve the alias Attempting to contact (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=133.37.253.82)(PORT=1521))) TNS-12547: TNS:lost contact
|
2. 连接只是在本机存在异常，如果在第三方服务器上请求连接，一些OK
3. 对lsnrctl命令做TRUSS，没有发现太多异常，只是最后返回一个异常
|
1200408: 1413557: thread_setmystate(0x0FFFFFFFFFFF7630, 0x0000000000000000) = 0 1200408: 1413557: kwrite(5, "\0 肻0\00104\0\001 :01 ,".., 195) = -1 1200408: Received signal #13, SIGPIPE [ignored] 1200408: 1413557: close(5) = 0 1200408: 1413557: lseek(4, 40448, 0) = 40448 |
4. 从报错来看，应该是资源不足造成的，但是检查也未发现什么异常。
|
zhjs3$[/home/oracle]ulimit -a time(seconds) unlimited file(blocks) unlimited data(kbytes) unlimited stack(kbytes) 32768 memory(kbytes) 32768 coredump(blocks) 2097151 nofiles(descriptors) 2000 |
5. topas检查发现几个sftp CPU消耗极高。了解到这个服务器上同时部署了数据库与应用。检查一下网络使用情况，发现存在异常的本机联机到133.37.116.121的FTP连接，高达3W+个
|
zhjs3$[/home/oracle]netstat -an tcp4 0 0 133.37.253.82.50436 133.37.116.121.21 CLOSE_WAIT tcp4 0 0 133.37.253.82.50437 133.37.116.121.21 CLOSE_WAIT tcp4 0 0 133.37.253.82.50438 133.37.116.121.21 CLOSE_WAIT tcp4 0 0 133.37.253.82.50439 133.37.116.121.21 CLOSE_WAIT tcp4 0 0 133.37.253.82.50440 133.37.116.121.21 CLOSE_WAIT tcp4 0 0 133.37.253.82.50441 133.37.116.121.21 CLOSE_WAIT tcp4 0 0 133.37.253.82.50442 133.37.116.121.21 CLOSE_WAIT tcp4 0 0 133.37.253.82.50443 133.37.116.121.21 CLOSE_WAIT tcp4 0 0 133.37.253.82.50444 133.37.116.121.21 CLOSE_WAIT tcp4 0 0 133.37.253.82.50445 133.37.116.121.21 CLOSE_WAIT tcp4 0 0 133.37.253.82.50446 133.37.116.121.21 CLOSE_WAIT tcp4 0 0 133.37.253.82.50447 133.37.116.121.21 CLOSE_WAIT tcp4 0 0 133.37.253.82.50448 133.37.116.121.21 CLOSE_WAIT tcp4 0 0 133.37.253.82.50449 133.37.116.121.21 CLOSE_WAIT tcp4 0 0 133.37.253.82.50450 133.37.116.121.21 CLOSE_WAIT tcp4 0 0 133.37.253.82.50451 133.37.116.121.21 CLOSE_WAIT tcp4 0 0 133.37.253.82.50452 133.37.116.121.21 CLOSE_WAIT tcp4 0 0 133.37.253.82.50453 133.37.116.121.21 CLOSE_WAIT tcp4 0 0 133.37.253.82.50454 133.37.116.121.21 CLOSE_WAIT tcp4 0 0 133.37.253.82.50455 133.37.116.121.21 CLOSE_WAIT tcp4 0 0 133.37.253.82.50456 133.37.116.121.21 CLOSE_WAIT
zhjs3$[/home/oracle]netstat -an|grep 133.37.116.121.21|wc -l 32341 |
6. 当数据库客户端与监听建立连接时，需要在本机分配一个临时的端口来建立socket连接。如果没有临时的端口可用，则无法建立连接。32341个指向133.37.116.121.21的连接，这个数目和常见的限制值32768很接近。怀疑由于FTP客户端程序耗尽了端口，导致本机连接数据库请求不到端口报错。
7. 开发商停止应用程序后，端口释放出来，数据库连接恢复正常
8. 查询资料得知，AIX平台下，临时端口通过tcp_ephemeral_low,tcp_ephemeral_high 参数指定，在这台服务器上，其定义的临时端口范围确实为32768个。
Normal 0 7.8 磅 0 2 false false false EN-US ZH-CN X-NONE来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/8242091/viewspace-752757/ ，如需转载，请注明出处，否则将追究法律责任。
