ITPub博客
首页 博客 专栏 学院 社区 IT新媒体评选
发博

首页 > 大数据 > 数据分析 > hadoop hdfs 命令使用

hadoop hdfs 命令使用

原创 数据分析 作者：lcz393537415 时间：2016-05-31 16:21:54 0 删除 编辑

参考链接：http://lxw1234.com/archives/2015/08/452.htm

在HDFS中，提供了fsck命令，用于检查HDFS上文件和目录的健康状态、获取文件的block信息和位置信息等。

fsck命令必须由HDFS超级用户来执行，普通用户无权限。


  1. [hadoop@dev~]$ hdfs fsck
  2. Usage:DFSck[-list-corruptfileblocks|[-move|-delete|-openforwrite][-files[-blocks[-locations|-racks]]]]
  3. start checkingfromthispath
  4. -move move corrupted files to/lost+found
  5. -deletedeletecorrupted files
  6. -filesprintoutfiles beingchecked
  7. -openforwriteprintoutfiles openedforwrite
  8. -includeSnapshots include snapshot dataifthe given path indicates a snapshottable directoryorthere are snapshottable directories under it
  9. -list-corruptfileblocksprintoutlist of missing blocksandfiles they belong to
  10. -blocksprintoutblock report
  11. -locationsprintoutlocationsforevery block
  12. -racksprintoutnetwork topologyfordata-node locations


下面介绍每一个选项的含义及用法。

查看文件中损坏的块（-list-corruptfileblocks）


  1. [hadoop@dev~]$ hdfs fsck/hivedata/warehouse/liuxiaowen.db/lxw_product_names/-list-corruptfileblocks
  2. Thefilesystem under path'/hivedata/warehouse/liuxiaowen.db/lxw_product_names/'has0CORRUPT files


将损坏的文件移动至/lost+found目录（-move）


  1. [hadoop@dev~]$ hdfs fsck/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168-move
  2. FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168atThuAug1309:36:35CST2015
  3. .Status:HEALTHY
  4. Totalsize:13497058B
  5. Totaldirs:0
  6. Totalfiles:1
  7. Totalsymlinks:0
  8. Totalblocks(validated):1(avg.block size13497058B)
  9. Minimallyreplicated blocks:1(100.0%)
  10. Over-replicated blocks:0(0.0%)
  11. Under-replicated blocks:0(0.0%)
  12. Mis-replicated blocks:0(0.0%)
  13. Defaultreplication factor:2
  14. Averageblock replication:2.0
  15. Corruptblocks:0
  16. Missingreplicas:0(0.0%)
  17. Numberof data-nodes:15
  18. Numberof racks:1
  19. FSCK ended atThuAug1309:36:35CST2015in1milliseconds
  20.  
  21.  
  22. Thefilesystem under path'/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168'isHEALTHY


删除损坏的文件（-delete）


  1. [hadoop@dev~]$ hdfs fsck/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168-delete
  2. FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168atThuAug1309:37:58CST2015
  3. .Status:HEALTHY
  4. Totalsize:13497058B
  5. Totaldirs:0
  6. Totalfiles:1
  7. Totalsymlinks:0
  8. Totalblocks(validated):1(avg.block size13497058B)
  9. Minimallyreplicated blocks:1(100.0%)
  10. Over-replicated blocks:0(0.0%)
  11. Under-replicated blocks:0(0.0%)
  12. Mis-replicated blocks:0(0.0%)
  13. Defaultreplication factor:2
  14. Averageblock replication:2.0
  15. Corruptblocks:0
  16. Missingreplicas:0(0.0%)
  17. Numberof data-nodes:15
  18. Numberof racks:1
  19. FSCK ended atThuAug1309:37:58CST2015in1milliseconds
  20.  
  21.  
  22. Thefilesystem under path'/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168'isHEALTHY


检查并列出所有文件状态（-files）


  1. [hadoop@dev~]$ hdfs fsck/hivedata/warehouse/liuxiaowen.db/lxw_product_names/-files
  2. FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/hivedata/warehouse/liuxiaowen.db/lxw_product_names/atThuAug1309:39:38CST2015
  3. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/dir
  4. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/_SUCCESS0bytes,0block(s):OK
  5. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000013583807bytes,1block(s):OK
  6. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000113577427bytes,1block(s):OK
  7. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000213588601bytes,1block(s):OK
  8. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000313479213bytes,1block(s):OK
  9. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000413497012bytes,1block(s):OK
  10. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000513557451bytes,1block(s):OK
  11. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000613580267bytes,1block(s):OK
  12. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000713486035bytes,1block(s):OK
  13. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-0000813481498bytes,1block(s):OK
  14. ...


检查并打印正在被打开执行写操作的文件（-openforwrite）


  1. [hadoop@dev~]$ hdfs fsck/hivedata/warehouse/liuxiaowen.db/lxw_product_names/-openforwrite
  2. FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/hivedata/warehouse/liuxiaowen.db/lxw_product_names/atThuAug1309:41:28CST2015
  3. ....................................................................................................
  4. ....................................................................................................
  5. .Status:HEALTHY
  6. Totalsize:2704782548B
  7. Totaldirs:1
  8. Totalfiles:201
  9. Totalsymlinks:0
  10. Totalblocks(validated):200(avg.block size13523912B)
  11. Minimallyreplicated blocks:200(100.0%)
  12. Over-replicated blocks:0(0.0%)
  13. Under-replicated blocks:0(0.0%)
  14. Mis-replicated blocks:0(0.0%)
  15. Defaultreplication factor:2
  16. Averageblock replication:2.0
  17. Corruptblocks:0
  18. Missingreplicas:0(0.0%)
  19. Numberof data-nodes:15
  20. Numberof racks:1
  21. FSCK ended atThuAug1309:41:28CST2015in10milliseconds
  22.  
  23. Thefilesystem under path'/hivedata/warehouse/liuxiaowen.db/lxw_product_names/'isHEALTHY


打印文件的Block报告（-blocks）

需要和-files一起使用。


  1. [hadoop@dev~]$ hdfs fsck/logs/site/2015-08-08/lxw1234.log-files-blocks
  2. FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/logs/site/2015-08-08/lxw1234.log atThuAug1309:45:59CST2015
  3. /logs/site/2015-08-08/lxw1234.log7408754725bytes,56block(s):OK
  4. 0.BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728repl=2
  5. 1.BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728repl=2
  6. 2.BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728repl=2
  7. 3.BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728repl=2
  8. 4.BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728repl=2
  9. 5.BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728repl=2
  10. 6.BP-1034052771-172.16.212.130-1405595752491:blk_1075892999_2152398 len=134217728repl=2
  11. 7.BP-1034052771-172.16.212.130-1405595752491:blk_1075893000_2152399 len=134217728repl=2
  12. 8.BP-1034052771-172.16.212.130-1405595752491:blk_1075893001_2152400 len=134217728repl=2
  13. 9.BP-1034052771-172.16.212.130-1405595752491:blk_1075893002_2152401 len=134217728repl=2
  14. 10.BP-1034052771-172.16.212.130-1405595752491:blk_1075893003_2152402 len=134217728repl=2
  15. 11.BP-1034052771-172.16.212.130-1405595752491:blk_1075893004_2152403 len=134217728repl=2
  16. 12.BP-1034052771-172.16.212.130-1405595752491:blk_1075893005_2152404 len=134217728repl=2
  17. 13.BP-1034052771-172.16.212.130-1405595752491:blk_1075893006_2152405 len=134217728repl=2
  18. 14.BP-1034052771-172.16.212.130-1405595752491:blk_1075893007_2152406 len=134217728repl=2
  19. ...


其中，/logs/site/2015-08-08/lxw1234.log 7408754725 bytes, 56 block(s): 表示文件的总大小和block数；

0. BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728 repl=2
1. BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728 repl=2
2. BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728 repl=2

前面的0. 1. 2.代表该文件的block索引，56的文件块，就从0-55;

BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381表示block id；

len=134217728 表示该文件块大小；

repl=2 表示该文件块副本数；

打印文件块的位置信息（-locations）

需要和-files -blocks一起使用。


  1. [hadoop@dev~]$ hdfs fsck/logs/site/2015-08-08/lxw1234.log-files-blocks-locations
  2. FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/logs/site/2015-08-08/lxw1234.log atThuAug1309:45:59CST2015
  3. /logs/site/2015-08-08/lxw1234.log7408754725bytes,56block(s):OK
  4. 0.BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728repl=2[172.16.212.139:50010,172.16.212.135:50010]
  5. 1.BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728repl=2[172.16.212.140:50010,172.16.212.133:50010]
  6. 2.BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728repl=2[172.16.212.136:50010,172.16.212.141:50010]
  7. 3.BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728repl=2[172.16.212.133:50010,172.16.212.135:50010]
  8. 4.BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728repl=2[172.16.212.142:50010,172.16.212.139:50010]
  9. 5.BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728repl=2[172.16.212.133:50010,172.16.212.139:50010]
  10. 6.BP-1034052771-172.16.212.130-1405595752491:blk_1075892999_2152398 len=134217728repl=2[172.16.212.141:50010,172.16.212.135:50010]
  11. 7.BP-1034052771-172.16.212.130-1405595752491:blk_1075893000_2152399 len=134217728repl=2[172.16.212.144:50010,172.16.212.142:50010]
  12. 8.BP-1034052771-172.16.212.130-1405595752491:blk_1075893001_2152400 len=134217728repl=2[172.16.212.133:50010,172.16.212.138:50010]
  13. 9.BP-1034052771-172.16.212.130-1405595752491:blk_1075893002_2152401 len=134217728repl=2[172.16.212.140:50010,172.16.212.134:50010]
  14. ...


和打印出的文件块信息相比，多了一个文件块的位置信息：[172.16.212.139:50010, 172.16.212.135:50010]

打印文件块位置所在的机架信息（-racks）


  1. [hadoop@dev~]$ hdfs fsck/logs/site/2015-08-08/lxw1234.log-files-blocks-locations-racks
  2. FSCK startedbyhadoop(auth:SIMPLE)from/172.16.212.17forpath/logs/site/2015-08-08/lxw1234.log atThuAug1309:45:59CST2015
  3. /logs/site/2015-08-08/lxw1234.log7408754725bytes,56block(s):OK
  4. 0.BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728repl=2[/default-rack/172.16.212.139:50010,/default-rack/172.16.212.135:50010]
  5. 1.BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728repl=2[/default-rack/172.16.212.140:50010,/default-rack/172.16.212.133:50010]
  6. 2.BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728repl=2[/default-rack/172.16.212.136:50010,/default-rack/172.16.212.141:50010]
  7. 3.BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728repl=2[/default-rack/172.16.212.133:50010,/default-rack/172.16.212.135:50010]
  8. 4.BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728repl=2[/default-rack/172.16.212.142:50010,/default-rack/172.16.212.139:50010]
  9. 5.BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728repl=2[/default-rack/172.16.212.133:50010,/default-rack/172.16.212.139:50010]
  10. ...


和前面打印出的信息相比，多了机架信息：[/default-rack/172.16.212.139:50010, /default-rack/172.16.212.135:50010]

来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/31347383/viewspace-2109850/ ，如需转载，请注明出处，否则将追究法律责任。

0
0
上一篇： storm 中遇到问题调整参数详解
下一篇： HA手动切换方式及方法
请登录后发表评论 登录
全部评论
lcz393537415

注册时间：2016-05-23

  • 博文量
    65
  • 访问量
    84700

最新文章

支持我们 作者招募 用户协议 FAQ Contact Us

北京盛拓优讯信息技术有限公司. 版权所有  京ICP备16024965号-8 北京市公安局海淀分局网监中心备案编号：11010802021510 niuxiaotong@pcpop.com 17352615567 未成年人举报专区

广播电视节目制作经营许可证(京) 字第1234号 中国互联网协会会员

违法和不良信息举报电话:18101376593