ITPub博客
首页 博客 专栏 学院 社区 IT新媒体评选
发博

首页 > 大数据 > Hadoop > 一个MapReduce 程序示例 细节决定成败(二) ：观察日志及 Counter

一个MapReduce 程序示例 细节决定成败(二) ：观察日志及 Counter

原创 Hadoop 作者：self_control 时间：2016-05-27 12:47:18 0 删除 编辑
编写一个mapreduce 程序：http://blog.itpub.net/30066956/viewspace-2107549/ 

下面是一个计算输入文件中a~z每个单字符的数量的一个map reduce 程序。

点击(此处)折叠或打开

  1. package wordcount;

  3. import java.io.IOException;

  5. import org.apache.commons.lang.StringUtils;
  6. import org.apache.hadoop.conf.Configuration;
  7. import org.apache.hadoop.conf.Configured;
  8. import org.apache.hadoop.fs.Path;
  9. import org.apache.hadoop.io.IntWritable;
  10. import org.apache.hadoop.io.LongWritable;
  11. import org.apache.hadoop.io.Text;
  12. import org.apache.hadoop.mapreduce.Job;
  13. import org.apache.hadoop.mapreduce.Mapper;
  14. import org.apache.hadoop.mapreduce.Reducer;
  15. import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
  16. import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
  17. import org.apache.hadoop.util.Tool;
  18. import org.apache.hadoop.util.ToolRunner;
  19. import org.apache.log4j.Logger;

  21. public class MyWordCountJob extends Configured implements Tool {
  22.         Logger log = Logger.getLogger(MyWordCountJob.class);

  24.         public static class MyWordCountMapper extends
  25.                         Mapper<LongWritable, Text, Text, IntWritable> {
  26.                 Logger log = Logger.getLogger(MyWordCountJob.class);

  28.                 Text mapKey = new Text();
  29.                 IntWritable mapValue = new IntWritable(1);
  30.                 @Override
  31.                 protected void map(LongWritable key, Text value, Context context)
  32.                                 throws IOException, InterruptedException {
  33.                         for(char c :value.toString().toLowerCase().toCharArray()){
  34.                                 if(c>='a' && c <='z'){
  35.                                         mapKey.set(String.valueOf(c));
  36.                                         context.write(mapKey, mapValue);
  37.                                 }
  38.                         }
  39.                 }

  41.         }


  44.         public static class MyWordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
  45.                 Text rkey = new Text();
  46.                 IntWritable rvalue = new IntWritable(1);
  47.                 @Override
  48.                 protected void reduce(Text key, Iterable<IntWritable> values,Context context)
  49.                                 throws IOException, InterruptedException {
  50.                         int n=0;
  51.                         for(IntWritable value :values){
  52.                                 n+= value.get();
  53.                         }
  54.                         rvalue.set(n);
  55.                         context.write(key, rvalue);
  56.                 }
  57.         }

  59.         @Override
  60.         public int run(String[] args) throws Exception {
  61.                 //valid the parameters
  62.                 if(args.length !=2){
  63.                         return -1;
  64.                 }

  66.                 Job job = Job.getInstance(getConf(), "MyWordCountJob");
  67.                 job.setJarByClass(MyWordCountJob.class);

  69.                 Path inPath = new Path(args[0]);
  70.                 Path outPath = new Path(args[1]);

  72.                 outPath.getFileSystem(getConf()).delete(outPath,true);
  73.                 TextInputFormat.setInputPaths(job, inPath);
  74.                 TextOutputFormat.setOutputPath(job, outPath);


  77.             job.setMapperClass(MyWordCountJob.MyWordCountMapper.class);
  78.                 job.setReducerClass(MyWordCountJob.MyWordCountReducer.class);
  79.             job.setInputFormatClass(TextInputFormat.class);
  80.             job.setOutputFormatClass(TextOutputFormat.class);

  82.                 job.setMapOutputKeyClass(Text.class);
  83.                 job.setMapOutputValueClass(IntWritable.class);
  84.                 job.setOutputKeyClass(Text.class);
  85.                 job.setOutputValueClass(IntWritable.class);

  87.                 return job.waitForCompletion(true)?0:1;
  88.         }
  89.         public static void main(String [] args){
  90.                 int result = 0;
  91.                 try {
  92.                         result = ToolRunner.run(new Configuration(), new MyWordCountJob(), args);
  93.                 } catch (Exception e) {
  94.                         e.printStackTrace();
  95.                 }
  96.                 System.exit(result);
  97.         }

  99. }

输入文件： 


点击(此处)折叠或打开

  1. [train@sandbox MyWordCount]$ hdfs dfs -ls mrdemo
  2. Found 3 items
  3. -rw-r--r-- 3 train hdfs 34 2016-05-11 01:41 mrdemo/demoinput1.txt
  4. -rw-r--r-- 3 train hdfs 42 2016-05-11 01:41 mrdemo/demoinput2.txt
  5. -rw-r--r-- 3 train hdfs 81 2016-05-11 01:41 mrdemo/demoinput3.txt

点击(此处)折叠或打开

  1. [train@sandbox MyWordCount]$ hdfs dfs -cat mrdemo/*input*.txt
  2. hello world
  3. how are you
  4. i am hero
  5. what is your name
  6. where are you come from
  7. abcdefghijklmnopqrsturwxyz
  8. abcdefghijklmnopqrsturwxyz
  9. abcdefghijklmnopqrsturwxyz

运行mr 任务

先看一下结果文件，可以看到按我们预期计算出了对应字符的个数（ps：这不是重点）


点击(此处)折叠或打开

  1. a 8
  2. b 3
  3. c 4
  4. d 4
  5. e 11
  6. f 4
  7. g 3
  8. h 8
  9. i 5
  10. j 3
  11. k 3
  12. l 6
  13. m 7
  14. n 4
  15. o 12
  16. p 3
  17. q 3
  18. r 13
  19. s 4
  20. t 4
  21. u 6
  22. w 7
  23. x 3
  24. y 6
  25. z 3


下面看一个运行日志关注标红的部分（这才是重点）

点击(此处)折叠或打开

  1. [train@sandbox MyWordCount]$ hadoop jar mywordcount.jar mrdemo/ mrdemo/output
  2. 16/05/11 04:00:45 INFO client.RMProxy: Connecting to ResourceManager at sandbox.hortonworks.com/192.168.252.131:8050
  3. 16/05/11 04:00:46 INFO input.FileInputFormat: Total input paths to process : 3
  4. 16/05/11 04:00:46 INFO mapreduce.JobSubmitter: number of splits:3
  5. 16/05/11 04:00:46 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
  6. 16/05/11 04:00:46 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
  7. 16/05/11 04:00:46 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
  8. 16/05/11 04:00:46 INFO Configuration.deprecation: mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class
  9. 16/05/11 04:00:46 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
  10. 16/05/11 04:00:46 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
  11. 16/05/11 04:00:46 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
  12. 16/05/11 04:00:46 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
  13. 16/05/11 04:00:46 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
  14. 16/05/11 04:00:46 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
  15. 16/05/11 04:00:46 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
  16. 16/05/11 04:00:46 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
  17. 16/05/11 04:00:46 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
  18. 16/05/11 04:00:46 INFO Configuration.deprecation: mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class
  19. 16/05/11 04:00:46 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
  20. 16/05/11 04:00:46 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1462517728035_0048
  21. 16/05/11 04:00:47 INFO impl.YarnClientImpl: Submitted application application_1462517728035_0048 to ResourceManager at sandbox.hortonworks.com/192.168.252.131:8050
  22. 16/05/11 04:00:47 INFO mapreduce.Job: The url to track the job: http://sandbox.hortonworks.com:8088/proxy/application_1462517728035_0048/
  23. 16/05/11 04:00:47 INFO mapreduce.Job: Running job: job_1462517728035_0048
  24. 16/05/11 04:00:55 INFO mapreduce.Job: Job job_1462517728035_0048 running in uber mode : false
  25. 16/05/11 04:00:55 INFO mapreduce.Job: map 0% reduce 0%
  26. 16/05/11 04:01:10 INFO mapreduce.Job: map 33% reduce 0%
  27. 16/05/11 04:01:11 INFO mapreduce.Job: map 100% reduce 0%
  28. 16/05/11 04:01:19 INFO mapreduce.Job: map 100% reduce 100%
  29. 16/05/11 04:01:19 INFO mapreduce.Job: Job job_1462517728035_0048 completed successfully
  30. 16/05/11 04:01:19 INFO mapreduce.Job: Counters: 43
  31.         File System Counters
  32.                 FILE: Number of bytes read=1102
  33.                 FILE: Number of bytes written=339257
  34.                 FILE: Number of read operations=0
  35.                 FILE: Number of large read operations=0
  36.                 FILE: Number of write operations=0
  37.                 HDFS: Number of bytes read=556
  38.                 HDFS: Number of bytes written=103
  39.                 HDFS: Number of read operations=12
  40.                 HDFS: Number of large read operations=0
  41.                 HDFS: Number of write operations=2
  42.         Job Counters
  43.                 Launched map tasks=3
  44.                 Launched reduce tasks=1
  45.                 Data-local map tasks=3
  46.                 Total time spent by all maps in occupied slots (ms)=314904
  47.                 Total time spent by all reduces in occupied slots (ms)=34648
  48.         Map-Reduce Framework
  49.                 Map input records=8
  50.                 Map output records=137
  51.                 Map output bytes=822
  52.                 Map output materialized bytes=1114
  53.                 Input split bytes=399
  54.                 Combine input records=0
  55.                 Combine output records=0
  56.                 Reduce input groups=25
  57.                 Reduce shuffle bytes=1114
  58.                 Reduce input records=137
  59.                 Reduce output records=25
  60.                 Spilled Records=274
  61.                 Shuffled Maps =3
  62.                 Failed Shuffles=0
  63.                 Merged Map outputs=3
  64.                 GC time elapsed (ms)=241
  65.                 CPU time spent (ms)=3340
  66.                 Physical memory (bytes) snapshot=1106452480
  67.                 Virtual memory (bytes) snapshot=3980922880
  68.                 Total committed heap usage (bytes)=884604928
  69.         Shuffle Errors
  70.                 BAD_ID=0
  71.                 CONNECTION=0
  72.                 IO_ERROR=0
  73.                 WRONG_LENGTH=0
  74.                 WRONG_MAP=0
  75.                 WRONG_REDUCE=0
  76.         File Input Format Counters
  77.                 Bytes Read=157
  78.         File Output Format Counters
  79.                 Bytes Written=103
从日志中我们可以得到：我们这个job的id是job_1462517728035_0048。 有3个split读文件，有3个mapper 任务，1个reducer任务。
map的输出记录数是137，reduce的输入记录数也是137。也就是说这137条记录是通过网络进行传输，送到reducer任务中的。
在下一篇中，我们使用一个combiner，来优化这个mapreduce 任务。

 一个MapReduce 程序示例 细节决定成败(三) ：Combiner

来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/30066956/viewspace-2107875/ ，如需转载，请注明出处，否则将追究法律责任。

0
0
上一篇： 一个MapReduce 程序示例 细节决定成败（一）
下一篇： 一个MapReduce 程序示例 细节决定成败(三) ：Combiner
请登录后发表评论 登录
全部评论
self_control

注册时间：2014-12-16

  • 博文量
    49
  • 访问量
    406897

最新文章

支持我们 作者招募 用户协议 FAQ Contact Us

北京盛拓优讯信息技术有限公司. 版权所有  京ICP备16024965号-8 北京市公安局海淀分局网监中心备案编号：11010802021510 niuxiaotong@pcpop.com 17352615567 未成年人举报专区

广播电视节目制作经营许可证(京) 字第1234号 中国互联网协会会员

违法和不良信息举报电话:18101376593