ITPub博客
首页 博客 专栏 学院 社区 IT新媒体评选
发博

首页 > > > Linux 下 如何检测内存泄漏？这篇文章给你答案～

Linux 下 如何检测内存泄漏？这篇文章给你答案～

作者：u_453e1a12781dea 时间：2017-10-10 09:43:04 0 删除 编辑
点击蓝色字关注 "CU 技术社区" 一起玩耍哦~

在实际的项目中，最难缠的问题就是内存泄漏，当然还有 panic 之类的，内存泄漏分为两部分用户空间的和内核空间的. 我们就分别从这两个层面分析一下.


用户空间查看内存泄漏和解决都相对简单。

定位问题的方法和工具也很多相对容易. 

我们来看看.


1. 查看内存信息


 cat /proc/meminfo、free、cat /proc/slabinfo 等 


2.  查看进程的状态信息


 top、ps、cat /proc/pid/maps/status/fd 等 


通常我们定位问题先在 shell 下 ps 查看当前运行进程的状态，嵌入式上可能显示的信息会少一些.


    

  1. root@hos-machine:~# ps -uaxw
    2. 

  2. USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
    3. 

  3. root 1 0.0 0.1 119872 3328 ? Ss 8 月 10 0:24 /sbin/init splash
    4. 

  4. root 2 0.0 0.0 0 0 ? S 8 月 10 0:00 [kthreadd]
    5. 

  5. root 3 0.0 0.0 0 0 ? S 8 月 10 0:44 [ksoftirqd/0]
    6. 

  6. root 5 0.0 0.0 0 0 ? S< 8 月 10 0:00 [kworker/0:0H]
    7. 

  7. root 7 0.0 0.0 0 0 ? S 8 月 10 3:50 [rcu_sched]
    8. 

  8. root 8 0.0 0.0 0 0 ? S 8 月 10 0:00 [rcu_bh]
    9. 

  9. root 9 0.0 0.0 0 0 ? S 8 月 10 0:12 [migration/0]
    10. 

  10. root 10 0.0 0.0 0 0 ? S 8 月 10 0:01 [watchdog/0]
    11. 

  11. root 11 0.0 0.0 0 0 ? S 8 月 10 0:01 [watchdog/1]
    12. 

  12. root 12 0.0 0.0 0 0 ? S 8 月 10 0:12 [migration/1]
    13. 

  13. root 13 0.0 0.0 0 0 ? S 8 月 10 1:18 [ksoftirqd/1]
    14. 

  14. root 15 0.0 0.0 0 0 ? S< 8 月 10 0:00 [kworker/1:0H]
    15. 

  15. root 16 0.0 0.0 0 0 ? S 8 月 10 0:01 [watchdog/2]
    16. 

  16. root 17 0.0 0.0 0 0 ? S 8 月 10 0:12 [migration/2]
    17. 

  17. root 18 0.0 0.0 0 0 ? S 8 月 10 1:19 [ksoftirqd/2]
    18. 

  18. root 20 0.0 0.0 0 0 ? S< 8 月 10 0:00 [kworker/2:0H]
    19. 

  19. root 21 0.0 0.0 0 0 ? S 8 月 10 0:01 [watchdog/3]
    20. 

  20. root 22 0.0 0.0 0 0 ? S 8 月 10 0:13 [migration/3]
    21. 

  21. root 23 0.0 0.0 0 0 ? S 8 月 10 0:41 [ksoftirqd/3]
    22. 

  22. root 25 0.0 0.0 0 0 ? S< 8 月 10 0:00 [kworker/3:0H]
    23. 

  23. root 26 0.0 0.0 0 0 ? S 8 月 10 0:00 [kdevtmpfs]
    24. 

  24. root 27 0.0 0.0 0 0 ? S< 8 月 10 0:00 [netns]
    25. 

  25. root 329 0.0 0.0 0 0 ? S< 8 月 10 0:00 [ext4-rsv-conver]
    26. 

  26. root 339 0.0 0.0 0 0 ? S< 8 月 10 0:05 [kworker/1:1H]
    27. 

  27. root 343 0.0 0.0 0 0 ? S< 8 月 10 0:11 [kworker/3:1H]
    28. 

  28. root 368 0.0 0.0 39076 1172 ? Ss 8 月 10 0:10 /lib/systemd/systemd-journald
    29. 

  29. root 373 0.0 0.0 0 0 ? S 8 月 10 0:00 [kauditd]
    30. 

  30. root 403 0.0 0.0 45772 48 ? Ss 8 月 10 0:01 /lib/systemd/systemd-udevd
    31. 

  31. root 444 0.0 0.0 0 0 ? S< 8 月 10 0:09 [kworker/2:1H]
    32. 

  32. systemd+ 778 0.0 0.0 102384 516 ? Ssl 8 月 10 0:04 /lib/systemd/systemd-timesyncd
    33. 

  33. root 963 0.0 0.0 191264 8 ? Ssl 8 月 10 0:00 /usr/bin/vmhgfs-fuse -o subtype=vmhgfs-fuse,allow_other /mnt/hgfs
    34. 

  34. root 987 9.6 0.0 917024 0 ? Ssl 8 月 10 416:08 /usr/sbin/vmware-vmblock-fuse -o subtype=vmware-vmblock,default_permi
    35. 

  35. root 1007 0.2 0.1 162728 3084 ? Sl 8 月 10 10:14 /usr/sbin/vmtoolsd
    36. 

  36. root 1036 0.0 0.0 56880 844 ? S 8 月 10 0:00 /usr/lib/vmware-vgauth/VGAuthService -s
    37. 

  37. root 1094 0.0 0.0 203216 388 ? Sl 8 月 10 1:48 ./ManagementAgentHost
    38. 

  38. root 1100 0.0 0.0 28660 136 ? Ss 8 月 10 0:02 /lib/systemd/systemd-logind
    39. 

  39. message+ 1101 0.0 0.1 44388 2608 ? Ss 8 月 10 0:21 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile
    40. 

  40. root 1110 0.0 0.0 173476 232 ? Ssl 8 月 10 0:54 /usr/sbin/thermald --no-daemon --dbus-enable
    41. 

  41. root 1115 0.0 0.0 4400 28 ? Ss 8 月 10 0:14 /usr/sbin/acpid
    42. 

  42. root 1117 0.0 0.0 36076 568 ? Ss 8 月 10 0:01 /usr/sbin/cron -f
    43. 

  43. root 1133 0.0 0.0 337316 976 ? Ssl 8 月 10 0:00 /usr/sbin/ModemManager
    44. 

  44. root 1135 0.0 0.2 634036 5340 ? Ssl 8 月 10 0:19 /usr/lib/snapd/snapd
    45. 

  45. root 1137 0.0 0.0 282944 392 ? Ssl 8 月 10 0:06 /usr/lib/accountsservice/accounts-daemon
    46. 

  46. syslog 1139 0.0 0.0 256396 352 ? Ssl 8 月 10 0:04 /usr/sbin/rsyslogd -n
    47. 

  47. avahi 1145 0.0 0.0 44900 1092 ? Ss 8 月 10 0:11 avahi-daemon: running [hos-machine.local]
    48.

上面


这个是 ubuntu 系统里的信息比较详细，我们可以很清晰看到 VMZ 和 RSS 的对比信息. VMZ 就是这个进程申请的虚拟地址空间，而 RSS 是这个进程占用的实际物理内存空间.

通常一个进程如果有内存泄露 VMZ 会不断增大，相对的物理内存也会增加，如果是这样一般需要检查 malloc/free 是否匹配。根据进程 ID 我们可以查看详细的 VMZ 相关的信息。例：


    

  1. root@hos-machine:~# cat /proc/1298/status 
    2. 

  2. Name:    sshd
    3. 

  3. State:    S (sleeping)
    4. 

  4. Tgid:    1298
    5. 

  5. Ngid:    0
    6. 

  6. Pid:    1298
    7. 

  7. PPid:    1
    8. 

  8. TracerPid:    0
    9. 

  9. Uid:    0    0    0    0
    10. 

  10. Gid:    0    0    0    0
    11. 

  11. FDSize:    128
    12. 

  12. Groups:    
    13. 

  13. NStgid:    1298
    14. 

  14. NSpid:    1298
    15. 

  15. NSpgid:    1298
    16. 

  16. NSsid:    1298
    17. 

  17. VmPeak:     65620 kB
    18. 

  18. VmSize:     65520 kB
    19. 

  19. VmLck:     0 kB
    20. 

  20. VmPin:     0 kB
    21. 

  21. VmHWM:     5480 kB
    22. 

  22. VmRSS:     5452 kB
    23. 

  23. VmData:     580 kB
    24. 

  24. VmStk:     136 kB
    25. 

  25. VmExe:     764 kB
    26. 

  26. VmLib:     8316 kB
    27. 

  27. VmPTE:     148 kB
    28. 

  28. VmPMD:     12 kB
    29. 

  29. VmSwap:     0 kB
    30. 

  30. HugetlbPages:     0 kB
    31. 

  31. Threads:    1
    32. 

  32. SigQ:    0/7814
    33. 

  33. SigPnd:    0000000000000000
    34. 

  34. ShdPnd:    0000000000000000
    35. 

  35. SigBlk:    0000000000000000
    36. 

  36. SigIgn:    0000000000001000
    37. 

  37. SigCgt:    0000000180014005
    38. 

  38. CapInh:    0000000000000000
    39. 

  39. CapPrm:    0000003fffffffff
    40. 

  40. CapEff:    0000003fffffffff
    41. 

  41. CapBnd:    0000003fffffffff
    42. 

  42. CapAmb:    0000000000000000
    43. 

  43. Seccomp:    0
    44. 

  44. Cpus_allowed:    ffffffff,ffffffff
    45. 

  45. Cpus_allowed_list:    0-63
    46. 

  46. Mems_allowed:    00000000,00000001
    47. 

  47. Mems_allowed_list:    0
    48. 

  48. voluntary_ctxt_switches:    1307
    49. 

  49. nonvoluntary_ctxt_switches:    203
    50.


如果我们想查看这个进程打开了多少文件可以


  ls -l /proc/1298/fd/* | wc 


查看进程详细的内存映射信息


 cat /proc/7393/maps 


我们看一下 meminfo 各个注释：参考 documentation/filesystem/proc.txt 


    

  1. MemTotal: Total usable ram (i.e. physical ram minus a few reserved bits and the kernel binary code) 
    2. 

  2. MemFree: The sum of LowFree+HighFree
    3. 

  3. Buffers: Relatively temporary storage for raw disk blocks shouldn't get tremendously large (20MB or so)
    4. 

  4.  Cached: in-memory cache for files read from the disk (the pagecache). Doesn't include 
    5. 

  5. SwapCached SwapCached: Memory that once was swapped out, is swapped back in but still also is in the swapfile (if memory is needed it
    6. 

  6.               doesn't need to be swapped out AGAIN because it is already in the swapfile. This saves I/O)
    7. 

  7.  Active: Memory that has been used more recently and usually not reclaimed unless absolutely necessary. 
    8. 

  8. Inactive: Memory which has been less recently used. It is more eligible to be reclaimed for other purposes 
    9. 

  9. HighTotal: 
    10. 

  10. HighFree: Highmem is all memory above ~860MB of physical memory Highmem areas are for use by userspace programs, or
    11. 

  11.               for the pagecache. The kernel must use tricks to access this memory, making it slower to access than lowmem.
    12. 

  12.  LowTotal:
    13. 

  13.  LowFree: Lowmem is memory which can be used for everything that highmem can be used for, but it is also available for the
    14. 

  14.               kernel's use for its own data structures. Among many other things, it is where everything from the Slab is
    15. 

  15.               allocated. Bad things happen when you're out of lowmem. 
    16. 

  16. SwapTotal: total amount of swap space available
    17. 

  17.  SwapFree: Memory which has been evicted from RAM, and is temporarily on the disk 
    18. 

  18. Dirty: Memory which is waiting to get written back to the disk 
    19. 

  19. Writeback: Memory which is actively being written back to the disk
    20. 

  20.  AnonPages: Non-file backed pages mapped into userspace page tables
    21. 

  21.  AnonHugePages: Non-file backed huge pages mapped into userspace page tables 
    22. 

  22. Mapped: files which have been mmaped, such as libraries 
    23. 

  23. Slab: in-kernel data structures cache
    24. 

  24.  SReclaimable: Part of Slab, that might be reclaimed, such as caches
    25. 

  25.  SUnreclaim: Part of Slab, that cannot be reclaimed on memory pressure 
    26. 

  26. PageTables: amount of memory dedicated to the lowest level of page tables. 
    27. 

  27. NFS_Unstable: NFS pages sent to the server, but not yet committed to stable storage 
    28. 

  28. Bounce: Memory used for block device "bounce buffers" 
    29. 

  29. WritebackTmp: Memory used by FUSE for temporary writeback buffers 
    30. 

  30. CommitLimit: Based on the overcommit ratio ('vm.overcommit_ratio'), this is the total amount of memory currently available to
    31. 

  31.               be allocated on the system. This limit is only adhered to if strict overcommit accounting is enabled (mode 2 in
    32. 

  32.               'vm.overcommit_memory').
    33. 

  33.               The CommitLimit is calculated with the following formula: CommitLimit = ('vm.overcommit_ratio' * Physical RAM) + Swap
    34. 

  34.               For example, on a system with 1G of physical RAM and 7G
    35. 

  35.               of swap with a `vm.overcommit_ratio` of 30 it would
    36. 

  36.               yield a CommitLimit of 7.3G.
    37. 

  37.               For more details, see the memory overcommit documentation in vm/overcommit-accounting.
    38. 

  38.  Committed_AS: The amount of memory presently allocated on the system. The committed memory is a sum of all of the memory which
    39. 

  39.               has been allocated by processes, even if it has not been
    40. 

  40.               "used" by them as of yet. A process which malloc()'s 1G
    41. 

  41.               of memory, but only touches 300M of it will only show up as using 300M of memory even if it has the address space
    42. 

  42.               allocated for the entire 1G. This 1G is memory which has been "committed" to by the VM and can be used at any time
    43. 

  43.               by the allocating application. With strict overcommit enabled on the system (mode 2 in 'vm.overcommit_memory'),
    44. 

  44.               allocations which would exceed the CommitLimit (detailed above) will not be permitted. This is useful if one needs
    45. 

  45.               to guarantee that processes will not fail due to lack of memory once that memory has been successfully allocated. 
    46. 

  46. VmallocTotal: total size of vmalloc memory area
    47. 

  47.  VmallocUsed: amount of vmalloc area which is used 
    48. 

  48. VmallocChunk: largest contiguous block of vmalloc area which is free
    49.


我们只需要关注几项就 ok.  


buffers/cache/slab/active/anonpages

Active= Active(anon) + Active(file)    (同样 Inactive)

AnonPages: Non-file backed pages mapped into userspace page tables\ 


buffers 和 cache 的区别注释说的很清楚了.


有时候不是内存泄露，同样也会让系统崩溃，比如 cache、buffers 等占用的太多，打开太多文件，而等待系统自动回收是一个非常漫长的过程.


从 proc 目录下的 meminfo 文件了解到当前系统内存的使用情况汇总，其中可用的物理内存 = memfree+buffers+cached，当 memfree 不够时，内核会通过


回写机制 (pdflush 线程) 把 cached 和 buffered 内存回写到后备存储器，从而释放相关内存供进程使用，或者通过手动方式显式释放 cache 内存


    

  1. drop_caches
    2. 

  2. Writing to this will cause the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free.
    3. 

  3. To free pagecache:
    4. 

  4. echo 1 > /proc/sys/vm/drop_caches 
    5. 

  5. To free dentries and inodes: 
    6. 

  6. echo 2 > /proc/sys/vm/drop_caches
    7. 

  7.  To free pagecache, dentries and inodes: 
    8. 

  8. echo 3 > /proc/sys/vm/drop_caches
    9. 

  9. As this is a non-destructive operation and dirty objects are not freeable, the user should run `sync`first
    10.


用户空间内存检测也可以通过 mtrace 来检测用法也非常简单，之前文章我们有提到过. 包括比较有名的工具 valgrind、以及 dmalloc、memwatch 等. 各有特点.


内核内存泄露的定位比较复杂，先判断是否是内核泄露了，然后在具体定位什么操作，然后再排查一些可疑的模块，内核内存操作基本都是 kmalloc


即通过 slab/slub/slob 机制，所以如果 meminfo 里 slab 一直增长那么很有可能是内核的问题. 我们可以更加详细的查看 slab 信息


 cat /proc/slabinfo  


如果支持 slabtop 更好，基本可以判断内核是否有内存泄漏，并且是在操作什么对象的时候发生的。


    

  1. cat /proc/slabinfo 
    2. 

  2. slabinfo - version: 2.1
    3. 

  3. # name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
    4. 

  4. fuse_request 0 0 288 28 2 : tunables 0 0 0 : slabdata 0 0 0
    5. 

  5. fuse_inode 0 0 448 18 2 : tunables 0 0 0 : slabdata 0 0 0
    6. 

  6. fat_inode_cache 0 0 424 19 2 : tunables 0 0 0 : slabdata 0 0 0
    7. 

  7. fat_cache 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0
    8.


在内核的配置中里面已经支持了一部分 memleak 自动检查的选项，可以打开来进行跟踪调试.



金秋十月与你饮酒论道

SACC2017

作为国内最受欢迎的架构师盛会，2017 第九届中国系统架构师大会 (SACC) 将于将于 2017 年 10 月 19-21 日在北京新云南皇冠假日酒店震撼来袭。


大会以 “云智未来” 为主题，云集国内外顶级专家，围绕云计算、人工智能、大数据、移动互联网、产业应用等热点领域展开技术探讨与交流。本届大会共设置 2 大主会场，18 个技术专场；邀请来自互联网、金融、制造业、电商等多个领域，100 余位技术专家及行业领袖来分享他们的经验；并将吸引 4000 + 人次的系统运维、架构师及 IT 决策人士参会，为他们提供最具价值的交流平台。


即日起，凡填写 SACC2017 会前调查问卷【点击阅读原文即可参与~】，即有机会赢取价值 2600 元的大会门票 1 张！仅限 10 张！


点击“阅读原文”，填写问卷赢门票哦~

原文作者：；如有侵权，请联系管理员删除。

0
0
上一篇： 当你「ping 一下」的时候，你知道它背后的逻辑吗？
下一篇： 三年 Git 使用心得 & 常见问题整理
请登录后发表评论 登录
全部评论
u_453e1a12781dea

注册时间：2019-07-17

  • 博文量
    1982
  • 访问量
    17924

最新文章

支持我们 作者招募 用户协议 FAQ Contact Us

北京盛拓优讯信息技术有限公司. 版权所有  京ICP备16024965号-8 北京市公安局海淀分局网监中心备案编号：11010802021510 niuxiaotong@pcpop.com 17352615567 未成年人举报专区

广播电视节目制作经营许可证(京) 字第1234号 中国互联网协会会员

违法和不良信息举报电话:18101376593