hadoop2.3.0安装及eclipse开发环境配置
目录
Unbuntu下载地址：http://www.ubuntu.org.cn/download/desktop 32bit
Hadoop下载地址：http://mirror.esocc.com/apache/hadoop/common/hadoop-2.3.0/
1）操作系统安装完毕后，需要安装ssh，关闭防火墙等。
设置主机名：
hadoop@node1:~/桌面$ sudo vim /etc/hostname
配置hosts和IP:
查看IP地址是否生效：
hadoop@node1:~/.ssh$ ifconfig
eth0 Link encap:以太网 硬件地址 00:0c:29:ad:92:26
inet 地址:192.168.204.241 广播:192.168.204.255 掩码:255.255.255.0
inet6 地址: fe80::20c:29ff:fead:9226/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 跃点数:1
接收数据包:2201 错误:0 丢弃:0 过载:0 帧数:0
发送数据包:1486 错误:0 丢弃:0 过载:0 载波:0
碰撞:0 发送队列长度:1000
接收字节:2743122 (2.7 MB) 发送字节:101896 (101.8 KB)
中断:19 基本地址:0x2000
lo Link encap:本地环回
inet 地址:127.0.0.1 掩码:255.0.0.0
inet6 地址: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 跃点数:1
接收数据包:340 错误:0 丢弃:0 过载:0 帧数:0
发送数据包:340 错误:0 丢弃:0 过载:0 载波:0
碰撞:0 发送队列长度:0
接收字节:44957 (44.9 KB) 发送字节:44957 (44.9 KB)
修改hosts：
hadoop@node1:~/.ssh$ sudo vim /etc/hosts
[sudo] password for hadoop:
可以使用ping node1进行测试。
关闭防火墙：
hadoop@node1:/etc$ sudo ufw disable
防火墙在系统启动时自动禁用
安装sshd：
hadoop@node1:~$ sudo apt-get install openssh-server
[sudo] password for hadoop:
hadoop@node1:~$ ps -ef |grep ssh
hadoop 1840 1801 0 11:03 ? 00:00:00 /usr/bin/ssh-agent /usr/bin/dbus-launch --exit-with-session gnome-session --session=ubuntu
root 4112 1 0 11:52 ? 00:00:00 /usr/sbin/sshd -D
hadoop 4171 3378 0 11:53 pts/4 00:00:00 grep --color=auto ssh
配置互相(由于采用虚拟机直接克隆，真实配置，需要每个节点均执行，则authorized_keys需要包含所有节点的id_rsq.pub文件内容)：
hadoop@node1:~$ pwd
/home/hadoop
hadoop@node1:~$ mkdir .ssh
hadoop@node1:~$ cd .ssh
hadoop@node1:~/.ssh$ ssh-keygen -t rsa -P ""
hadoop@node1:~/.ssh$ ls -tlr
总用量 8
-rw-r--r-- 1 hadoop hadoop 394 4月 2 11:54 id_rsa.pub
-rw------- 1 hadoop hadoop 1679 4月 2 11:54 id_rsa
hadoop@node1:~/.ssh$ cat id_rsa.pub >>authorized_keys
hadoop@node1:~/.ssh$ ssh node1
The authenticity of host 'node1 (192.168.204.241)' can't be established.
ECDSA key fingerprint is 2b:fc:88:24:fe:9a:25:04:c4:75:8f:bd:42:05:10:95.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node1,192.168.204.241' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 12.04.4 LTS (GNU/Linux 3.11.0-15-generic i686)
* Documentation: https://help.ubuntu.com/
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
hadoop@node1:~$ exit
登出
Connection to node1 closed.
hadoop@node1:~/.ssh$ ssh node1 date
2014年 04月 02日 星期三 12:01:56 CST
hadoop@node1:~/.ssh$ ssh node1 date
2014年 04月 02日 星期三 12:01:58 CST
2） 安装JAVA:
hadoop@node1:~/桌面$ ls -tlr
总用量 416604
-rwxrw-rw- 1 hadoop hadoop 293457920 3月 1 07:59 jdk-7u51-linux-i586.tar
-rwxrw-rw- 1 hadoop hadoop 133136805 3月 7 12:50 hadoop-2.3.0.tar.gz
hadoop@node1:/usr$ ls -tlr
总用量 116
drwxr-xr-x 10 root root 4096 2月 4 19:58 local
drwxr-xr-x 4 root root 4096 2月 4 20:00 src
drwxr-xr-x 2 root root 4096 2月 4 20:00 games
drwxr-xr-x 35 root root 4096 2月 4 20:00 include
drwxr-xr-x 272 root root 12288 4月 2 10:38 share
drwxr-xr-x 2 hadoop hadoop 4096 4月 2 11:16 hadoop
drwxr-xr-x 3 hadoop hadoop 4096 4月 2 11:18 java
drwxr-xr-x 2 root root 12288 4月 2 11:52 sbin
drwxr-xr-x 168 root root 32768 4月 2 11:52 lib
drwxr-xr-x 2 root root 36864 4月 2 11:52 bin
hadoop@node1:~/桌面$cp jdk-7u51-linux-i586.tar /usr/java
hadoop@node1:/usr/java$ tar -xvf jdk-7u51-linux-i586.tar
设置环境变量：
hadoop@node1:/etc$ sudo vim /etc/profile
[sudo] password for hadoop:
添加：
export JAVA_HOME=/usr/java/jdk1.7.0_51
export JRE_HOME=/usr/java/jdk1.7.0_51/jre
export CLASSPATH=.:$JAVA_HOME/lib
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
测试：
hadoop@node1:/etc$ source /etc/profile
hadoop@node1:/etc$ java -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) Client VM (build 24.51-b03, mixed mode)
3） 安装hadoop：
hadoop@node1:~/桌面$ cp hadoop-2.3.0.tar.gz /usr/hadoop/
hadoop@node1:~/桌面$ cd /usr/hadoop/
hadoop@node1:/usr/hadoop$ ls -tlr
总用量 130020
-rwxrw-r-- 1 hadoop hadoop 133136805 4月 2 14:26 hadoop-2.3.0.tar.gz
hadoop@node1:/usr/hadoop$ gunzip hadoop-2.3.0.tar.gz
hadoop@node1:/usr/hadoop$ tar -xvf hadoop-2.3.0.tar
hadoop@node1:/usr/hadoop$ pwd
/usr/hadoop
hadoop@node1:/usr/hadoop$ ls
bin etc include lib libexec LICENSE.txt NOTICE.txt README.txt sbin share
修改配置文件：
hadoop@node1:/usr/hadoop/etc/hadoop$ ls |grep sh
hadoop-env.sh
mapred-env.sh
yarn-env.sh
修改这三个文件中关于JAVA_HOME的环境变量。
克隆虚拟机1to3.修改hostname以及IP，测试各节点间ssh。
修改slaves:
hadoop@node1:/usr/hadoop/etc/hadoop$ vim slaves
hadoop@node1:/usr/hadoop/etc/hadoop$ cat slaves
node2
node3
修改core-site.xml文件：
hadoop@node1:/usr/hadoop/etc/hadoop$ cat core-site.xml
hadoop@node1:/usr/hadoop/etc/hadoop$
hadoop@node1:/usr/hadoop/etc/hadoop$ cat mapred-site.xml
hadoop@node1:/usr/hadoop/etc/hadoop$ cat hdfs-site.xml
hadoop@node1:/usr/hadoop/etc/hadoop$ cat yarn-site.xml
<!-- Site specific YARN configuration properties -->
拷贝至各节点：
Scp /usr/hadoop/etc/hadoop/* node2:/usr/hadoop/etc/hadoop
Scp /usr/hadoop/etc/hadoop/* node3:/usr/hadoop/etc/hadoop
启动hadoop：
初始化hdfs文件系统：
hadoop@node1:/usr/hadoop/etc/hadoop$ hdfs namenode -format
14/04/02 16:35:33 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = node1/192.168.204.241
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.3.0
STARTUP_MSG: classpath = /usr/hadoop/etc/hadoop:/usr/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/hadoop/share………………………………………………………………………usr/hadoop/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common -r 1567123; compiled by 'jenkins' on 2014-02-11T13:40Z
STARTUP_MSG: java = 1.7.0_51
************************************************************/
14/04/02 16:35:33 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
Formatting using clusterid: CID-400e6574-6b15-433c-83ca-ca518765f146
14/04/02 16:35:34 INFO namenode.FSNamesystem: fsLock is fair:true
14/04/02 16:35:34 INFO namenode.HostFileManager: read includes:
HostSet(
)
14/04/02 16:35:34 INFO namenode.HostFileManager: read excludes:
HostSet(
)
14/04/02 16:35:34 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
14/04/02 16:35:34 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
14/04/02 16:35:34 INFO util.GSet: Computing capacity for map BlocksMap
14/04/02 16:35:34 INFO util.GSet: VM type = 32-bit
14/04/02 16:35:34 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
14/04/02 16:35:34 INFO util.GSet: capacity = 2^22 = 4194304 entries
14/04/02 16:35:34 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
14/04/02 16:35:34 INFO blockmanagement.BlockManager: defaultReplication = 2
14/04/02 16:35:34 INFO blockmanagement.BlockManager: maxReplication = 512
14/04/02 16:35:34 INFO blockmanagement.BlockManager: minReplication = 1
14/04/02 16:35:34 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
14/04/02 16:35:34 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
14/04/02 16:35:34 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
14/04/02 16:35:34 INFO blockmanagement.BlockManager: encryptDataTransfer = false
14/04/02 16:35:34 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
14/04/02 16:35:34 INFO namenode.FSNamesystem: fsOwner = hadoop (auth:SIMPLE)
14/04/02 16:35:34 INFO namenode.FSNamesystem: supergroup = supergroup
14/04/02 16:35:34 INFO namenode.FSNamesystem: isPermissionEnabled = true
14/04/02 16:35:34 INFO namenode.FSNamesystem: HA Enabled: false
14/04/02 16:35:34 INFO namenode.FSNamesystem: Append Enabled: true
14/04/02 16:35:35 INFO util.GSet: Computing capacity for map INodeMap
14/04/02 16:35:35 INFO util.GSet: VM type = 32-bit
14/04/02 16:35:35 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
14/04/02 16:35:35 INFO util.GSet: capacity = 2^21 = 2097152 entries
14/04/02 16:35:35 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/04/02 16:35:35 INFO util.GSet: Computing capacity for map cachedBlocks
14/04/02 16:35:35 INFO util.GSet: VM type = 32-bit
14/04/02 16:35:35 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
14/04/02 16:35:35 INFO util.GSet: capacity = 2^19 = 524288 entries
14/04/02 16:35:35 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
14/04/02 16:35:35 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
14/04/02 16:35:35 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
14/04/02 16:35:35 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
14/04/02 16:35:35 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
14/04/02 16:35:35 INFO util.GSet: Computing capacity for map Namenode Retry Cache
14/04/02 16:35:35 INFO util.GSet: VM type = 32-bit
14/04/02 16:35:35 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
14/04/02 16:35:35 INFO util.GSet: capacity = 2^16 = 65536 entries
14/04/02 16:35:35 INFO common.Storage: Storage directory /usr/hadoop/dfs/name has been successfully formatted.
14/04/02 16:35:35 INFO namenode.FSImage: Saving image file /usr/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
14/04/02 16:35:35 INFO namenode.FSImage: Image file /usr/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 218 bytes saved in 0 seconds.
14/04/02 16:35:35 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
14/04/02 16:35:35 INFO util.ExitUtil: Exiting with status 0
14/04/02 16:35:35 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at node1/192.168.204.241
************************************************************/
hadoop@node1:/usr/hadoop/etc/hadoop$
启动dfs：
hadoop@node1:/usr/hadoop/sbin$ ./start-dfs.sh
Starting namenodes on [node1]
node1: starting namenode, logging to /usr/hadoop/logs/hadoop-hadoop-namenode-node1.out
node3: starting datanode, logging to /usr/hadoop/logs/hadoop-hadoop-datanode-node3.out
node2: starting datanode, logging to /usr/hadoop/logs/hadoop-hadoop-datanode-node2.out
Starting secondary namenodes [node1]
node1: starting secondarynamenode, logging to /usr/hadoop/logs/hadoop-hadoop-secondarynamenode-node1.out
hadoop@node1:/usr/hadoop/sbin$ jps
5728 SecondaryNameNode
5472 NameNode
5855 Jps
启动yarn：
hadoop@node1:/usr/hadoop/sbin$ ./start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/hadoop/logs/yarn-hadoop-resourcemanager-node1.out
node3: starting nodemanager, logging to /usr/hadoop/logs/yarn-hadoop-nodemanager-node3.out
node2: starting nodemanager, logging to /usr/hadoop/logs/yarn-hadoop-nodemanager-node2.out
节点1运行状态：
hadoop@node1:/usr/hadoop/sbin$ jps
6150 Jps
5728 SecondaryNameNode
5472 NameNode
5907 ResourceManager
节点2运行状态：
hadoop@node2:/usr/hadoop/etc/hadoop$ jps
4878 DataNode
5106 NodeManager
5201 Jps
节点3运行状态：
hadoop@node3:/usr/hadoop/etc/hadoop$ jps
4775 Jps
4668 NodeManager
4458 DataNode
查看集群状态：
hadoop@node1:/usr/hadoop/sbin$ hdfs dfsadmin -report
Configured Capacity: 29324574720 (27.31 GB)
Present Capacity: 18415513600 (17.15 GB)
DFS Remaining: 18415464448 (17.15 GB)
DFS Used: 49152 (48 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 2 (2 total, 0 dead)
Live datanodes:
Name: 192.168.204.243:50010 (node3)
Hostname: node3
Decommission Status : Normal
Configured Capacity: 14662287360 (13.66 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 5454675968 (5.08 GB)
DFS Remaining: 9207586816 (8.58 GB)
DFS Used%: 0.00%
DFS Remaining%: 62.80%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Last contact: Wed Apr 02 16:44:48 CST 2014
Name: 192.168.204.242:50010 (node2)
Hostname: node2
Decommission Status : Normal
Configured Capacity: 14662287360 (13.66 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 5454385152 (5.08 GB)
DFS Remaining: 9207877632 (8.58 GB)
DFS Used%: 0.00%
DFS Remaining%: 62.80%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Last contact: Wed Apr 02 16:44:48 CST 2014
登录网页进行查看HDFS：
Datanode信息：
Hadoop网页管理界面：
hadoop@node1:/usr/hadoop/etc/hadoop$ hadoop fs -put *.sh /input
hadoop@node1:/usr/hadoop/etc/hadoop$ hadoop fs -ls /input
Found 4 items
-rw-r--r-- 2 hadoop supergroup 3399 2014-04-02 17:02 /input/hadoop-env.sh
-rw-r--r-- 2 hadoop supergroup 1449 2014-04-02 17:02 /input/httpfs-env.sh
-rw-r--r-- 2 hadoop supergroup 1377 2014-04-02 17:02 /input/mapred-env.sh
-rw-r--r-- 2 hadoop supergroup 4078 2014-04-02 17:02 /input/yarn-env.sh
hadoop@node1:/usr/hadoop/share/hadoop/mapreduce$ pwd
/usr/hadoop/share/hadoop/mapreduce
hadoop@node1:/usr/hadoop/share/hadoop/mapreduce$ hadoop jar hadoop-mapreduce-examples-2.3.0.jar wordcount /input /output
14/04/03 10:29:57 INFO client.RMProxy: Connecting to ResourceManager at node1/192.168.204.241:8032
14/04/03 10:29:58 INFO input.FileInputFormat: Total input paths to process : 4
14/04/03 10:29:59 INFO mapreduce.JobSubmitter: number of splits:4
14/04/03 10:29:59 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1396492140420_0001
14/04/03 10:29:59 INFO impl.YarnClientImpl: Submitted application application_1396492140420_0001
14/04/03 10:30:00 INFO mapreduce.Job: The url to track the job: http://node1:8088/proxy/application_1396492140420_0001/
14/04/03 10:30:00 INFO mapreduce.Job: Running job: job_1396492140420_0001
14/04/03 10:30:11 INFO mapreduce.Job: Job job_1396492140420_0001 running in uber mode : false
14/04/03 10:30:11 INFO mapreduce.Job: map 0% reduce 0%
14/04/03 10:30:32 INFO mapreduce.Job: map 50% reduce 0%
14/04/03 10:30:33 INFO mapreduce.Job: map 100% reduce 0%
14/04/03 10:30:42 INFO mapreduce.Job: map 100% reduce 100%
14/04/03 10:30:42 INFO mapreduce.Job: Job job_1396492140420_0001 completed successfully
14/04/03 10:30:43 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=11240
FILE: Number of bytes written=448399
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=10709
HDFS: Number of bytes written=5841
HDFS: Number of read operations=15
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=4
Launched reduce tasks=1
Data-local map tasks=4
Total time spent by all maps in occupied slots (ms)=80305
Total time spent by all reduces in occupied slots (ms)=6285
Total time spent by all map tasks (ms)=80305
Total time spent by all reduce tasks (ms)=6285
Total vcore-seconds taken by all map tasks=80305
Total vcore-seconds taken by all reduce tasks=6285
Total megabyte-seconds taken by all map tasks=82232320
Total megabyte-seconds taken by all reduce tasks=6435840
Map-Reduce Framework
Map input records=269
Map output records=1341
Map output bytes=15531
Map output materialized bytes=11258
Input split bytes=406
Combine input records=1341
Combine output records=698
Reduce input groups=409
Reduce shuffle bytes=11258
Reduce input records=698
Reduce output records=409
Spilled Records=1396
Shuffled Maps =4
Failed Shuffles=0
Merged Map outputs=4
GC time elapsed (ms)=2089
CPU time spent (ms)=5190
Physical memory (bytes) snapshot=693395456
Virtual memory (bytes) snapshot=1826213888
Total committed heap usage (bytes)=500973568
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=10303
File Output Format Counters
Bytes Written=5841
hadoop@node1:/usr/hadoop/share/hadoop/mapreduce$
查看运行情况：
hadoop@node1:~/下载$ tail -n 100 F3Wf8eyB.bin
may 11
might 1
more 3
multiple 1
nodes. 1
4） 安装eclipse：
Eclipse下载地址：http://www.eclipse.org/downloads/?osType=linux&release=undefined
安装在/usr/eclipse目录中：
hadoop@node1:/usr/eclipse$ pwd
/usr/eclipse
hadoop@node1:/usr/eclipse$ ls -tlr
总用量 224364
drwxrwxr-x 9 hadoop hadoop 4096 2月 24 18:31 eclipse
-rwxrw-rw- 1 hadoop hadoop 229744640 4月 3 10:46 eclipse-standard-kepler-SR2-linux-gtk.tar
下载eclipse的hadoop插件：
http://pan.baidu.com/s/1hqGC6by
下载后放置于eclipse的plugins中。
创建到桌面的快捷方式，双击启动：
配置Hadoop开发环境：
选中Map/Reduce视图：
在右下窗口中找到Map/Reduce Locations，此处用来配置路径：
配置信息如下：
Lcation name：为配额的路径指定一个名称：
其余的参照mapred-site.xml和core-site.xml文件中的配置信息填写。
完成后可在Project Explorer中找到hadoop，点开既可以看到刚才创建的文件夹和文件。
建立测试项目：
配置hadoop安装目录路径：
配置运行参数，指定任务的输入、输出目录(输出目录在任务中自动创建)
配置完成后，直接运行。
在hadoop dfs中新增wordcount目录，打开其中的part-r-00000，发现运行结果：
Wordcount代码如下：
package test;
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;
public class wordcount {
public static class TokenizerMapper
extends Mapper
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value, Context context
) throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}
public static class IntSumReducer
extends Reducer
private IntWritable result = new IntWritable();
public void reduce(Text key,
Iterable
Context context
) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
String[] otherArgs = new GenericOptionsParser(conf, args)
.getRemainingArgs();
if (otherArgs.length != 2) {
System.err.println("Usage: wordcount
System.exit(2);
}
Job job = new Job(conf, "word count");
job.setJarByClass(wordcount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/15173958/viewspace-1142325/ ，如需转载，请注明出处，否则将追究法律责任。
