[20190313]备份问题.txt

--//生产系统遇到备份问题,同事请求检查看看,提示应该是ulimit参数限制.

--//首先我不熟悉这套备份系统,我心里很反感这套备份模式,我开始以为修改/etc/security/limits.conf就ok了.

# grep -v "^#" /etc/security/limits.conf

oracle soft nproc 2047

oracle hard nproc 16384

oracle soft nofile 65536

oracle hard nofile 65536

oracle soft memlock 2457600

oracle hard memlock 2457600

--//发现还是不行,问了一些细节,发现通过root进程下的一个服务完成备份:

# ps -ef | grep avata[r]

root 23601 1 0 2018 ? 00:00:06 /usr/local/avatar/client_server/sbin/avatardevice -v -c /usr/local/avatar/client_server/etc/avatardevice.conf

root 23613 1 0 2018 ? 05:16:39 /usr/local/avatar/client_server/sbin/avatarclient -v -c /usr/local/avatar/client_server/etc/avatarclient.conf

--//而这两个进程是root用户的,这样应该修改root用户对于参数加入如下:

* soft nofile 65536

* hard nofile 65536

--//我在猜想为什么该进程要打开如此之多的文件句柄.

# ls -l /proc/23613/fd |wc

591 4902 47629

--//昏,已经使用591个文件句柄.

# ls -l /proc/23613/fd | cut -f2 -d ">" | sort | uniq -c

3 /dev/null

1 socket:[76605529]

1 total 0

292 /usr/local/avatar/client_server/data

292 /usr/local/avatar/client_server/data/ora_scripts

1 /usr/local/avatar/client_server/log

1 /usr/local/avatar/client_server/log/avatarclient20181022.log

--//很明显这家公司的产品写的程序有问题,大量的重复打开相同的文件句柄,!!估计每次备份打开1次,292天就292次,并且完成后竟然不

--//写关闭句柄的语句,国内的水平就是这样!!打着所谓的高科技骗人,

--//这样再大的参数设置最终也会撑爆的,也许最佳的方式定时重启这个服务.

# service avatarclient restart

Stopping the Avatar Client daemon

Stopping the Avatar Device daemon

Starting the Avatar Device daemon

Starting the Avatar Client daemon

# ps -ef | grep avata[r]

root 5414 1 0 15:53 ? 00:00:00 /usr/local/avatar/client_server/sbin/avatardevice -v -c /usr/local/avatar/client_server/etc/avatardevice.conf

root 5425 1 0 15:53 ? 00:00:00 /usr/local/avatar/client_server/sbin/avatarclient -v -c /usr/local/avatar/client_server/etc/avatarclient.conf

# ls -l /proc/5425/fd

total 0

lr-x------ 1 root root 64 2019-03-13 15:53:52 0 -> /dev/null

lr-x------ 1 root root 64 2019-03-13 15:53:52 1 -> /dev/null

lr-x------ 1 root root 64 2019-03-13 15:53:52 2 -> /dev/null

lrwx------ 1 root root 64 2019-03-13 15:53:52 3 -> socket:[86753325]

lr-x------ 1 root root 64 2019-03-13 15:53:52 4 -> /usr/local/avatar/client_server/log

lrwx------ 1 root root 64 2019-03-13 15:53:52 5 -> /usr/local/avatar/client_server/log/avatarclient20190313.log

--//现在没有前面看到的2个文件句柄.

# cat /proc/5425/limits

Limit Soft Limit Hard Limit Units

Max cpu time unlimited unlimited seconds

Max file size unlimited unlimited bytes

Max data size unlimited unlimited bytes

Max stack size 10485760 unlimited bytes

Max core file size 0 unlimited bytes

Max resident set unlimited unlimited bytes

Max processes 1024 30148 processes

Max open files 1024 1024 files

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Max locked memory 65536 65536 bytes

Max address space unlimited unlimited bytes

Max file locks unlimited unlimited locks

Max pending signals 30148 30148 signals

Max msgqueue size 819200 819200 bytes

Max nice priority 0 0

Max realtime priority 0 0

Max realtime timeout unlimited unlimited us

--//注:可以发现我这样操作不行,必须注销root用户然后再重新启动服务才生效.以下是退出重启后的情况:

ps -ef | grep avata[r]

root 5905 1 0 16:00 ? 00:00:00 /usr/local/avatar/client_server/sbin/avatardevice -v -c /usr/local/avatar/client_server/etc/avatardevice.conf

root 5916 1 0 16:00 ? 00:00:00 /usr/local/avatar/client_server/sbin/avatarclient -v -c /usr/local/avatar/client_server/etc/avatarclient.conf

# cat /proc/5916/limits