本文使用了4台虚拟机成功的部署了hadoop2.7.2的集群开发环境。

1. 前言

假设你的环境已经具备了运行Hadoop的必要条件，如果没有你可以参考Centos上源码安装Hadoop2.7.2。

2. 准备工作

这里我准备了4台Centos7虚拟机，下面我们马上来进行配置。

# 编辑host

vi /etc/hosts

# 每个节点添加IP、主机名称

192.168.10.162 hmaster

192.168.10.163 hslave1

192.168.10.164 hslave2

192.168.10.166 hslave3

# 创建hadoop用户

useradd hadoop

# 设置密码

passwd hadoop

# 设置权限

chown hadoop /app/hadoop-2.7.2/ -R

chgrp hadoop /app/hadoop-2.7.2/ -R

# ssh自动登陆

su hadoop

# 这里一路回车即可

ssh-keygen -t rsa

ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@hmaster

ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@hslave1

ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@hslave2

ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@hslave3

chmod 0600 ~/.ssh/authorized_keys

# 关闭防火墙

systemctl stop firewalld

# 也可以编辑sysctl.conf

vi /etc/sysctl.conf

# 添加以下内容

net.ipv6.conf.all.disable_ipv6 = 1

net.ipv6.conf.default.disable_ipv6 = 1

3. 部署

3.1. 设置hadoop环境变量

# 切换到hadoop用户，添加hadoop环境变量

vi ~/.bashrc

# 添加以下内容

export HADOOP_PREFIX=/app/hadoop-2.7.2

export HADOOP_HOME=$HADOOP_PREFIX

export HADOOP_COMMON_HOME=$HADOOP_PREFIX

export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop

export HADOOP_HDFS_HOME=$HADOOP_PREFIX

export HADOOP_MAPRED_HOME=$HADOOP_PREFIX

export HADOOP_YARN_HOME=$HADOOP_PREFIX

export PATH=$PATH:$HADOOP_PREFIX/sbin:$HADOOP_PREFIX/bin

# 退出，使之生效

source ~/.bashrc

3.2. 设置master

# 1. 编辑core-site.xml

vi /app/hadoop-2.7.2/etc/hadoop/core-site.xml

# 添加以下内容

<configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://hmaster:9000/</value>

</property>

</configuration>

# 2. 创建NameNode目录

mkdir /home/hadoop/namenode

chown hadoop /home/hadoop/namenode/

chgrp hadoop /home/hadoop/namenode/

# 3. 编辑hdfs-site.xml

vi /app/hadoop-2.7.2/etc/hadoop/hdfs-site.xml

# 添加以下内容

<configuration>

<property>

<name>dfs.replication</name>

<value>3</value>

</property>

<property>

<name>dfs.permissions</name>

<value>false</value>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>/home/hadoop/datanode</value>

</property>

<property>

<name>dfs.namenode.data.dir</name>

<value>/home/hadoop/namenode</value>

</property>

</configuration>

# 4. 编辑mapred-site.xml

vi /app/hadoop-2.7.2/etc/hadoop/mapred-site.xml

# 添加以下内容

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

</configuration>

# 5. 编辑yarn-site.xml设置ResourceManager和NodeManagers

<configuration>

<property>

<name>yarn.resourcemanager.hostname</name>

<value>hmaster</value>

</property>

<property>

<name>yarn.nodemanager.hostname</name>

<value>hmaster</value>

</property>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

</configuration>

# 6. 编辑slaves添加DataNode

vi /app/hadoop-2.7.2/etc/hadoop/slaves

# 添加以下内容

hmaster

hslave1

hslave2

hslave3

3.3. 设置DataNode

# 1. 编辑core-site.xml

vi /app/hadoop-2.7.2/etc/hadoop/core-site.xml

# 添加以下内容

<configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://hmaster:9000/</value>

</property>

</configuration>

# 2. 创建DataNode目录

mkdir /home/hadoop/datanode

chown hadoop /home/hadoop/datanode/

chgrp hadoop /home/hadoop/datanode/

# 3. 编辑hdfs-site.xml

vi /app/hadoop-2.7.2/etc/hadoop/hdfs-site.xml

# 添加以下内容

<configuration>

<property>

<name>dfs.replication</name>

<value>3</value>

</property>

<property>

<name>dfs.permissions</name>

<value>false</value>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>/home/hadoop/datanode</value>

</property>

</configuration>

3.4. 启动

# 在master上切换到hadoop用户

su hadoop

hdfs namenode -format

# 启动命令后, 使用jps查看任务是否启动, Slave节点:DataNode, Master节点: NameNode, DataNode, SecondaryNameNode

cd /app/hadoop-2.7.2/etc/hadoop

sbin/start-dfs.sh

# 启动YARN, 查看Master节点上的ResourceManager和NodeManager是否启动

sbin/start-yarn.sh

# 查看集群状态

hdfs dfsadmin -report

Configured Capacity: 74985766912 (69.84 GB)

Present Capacity: 21625159680 (20.14 GB)

DFS Remaining: 21625114624 (20.14 GB)

DFS Used: 45056 (44 KB)

DFS Used%: 0.00%

Under replicated blocks: 0

Blocks with corrupt replicas: 0

Missing blocks: 0

Missing blocks (with replication factor 1): 0

-------------------------------------------------

Live datanodes (4):

Name: 192.168.10.163:50010 (hslave1)

Hostname: hslave1

Decommission Status : Normal

Configured Capacity: 18746441728 (17.46 GB)

DFS Used: 12288 (12 KB)

Non DFS Used: 13337149440 (12.42 GB)

DFS Remaining: 5409280000 (5.04 GB)

DFS Used%: 0.00%

DFS Remaining%: 28.85%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Sat Apr 09 02:18:49 CST 2016

Name: 192.168.10.164:50010 (hslave2)

Hostname: hslave2

Decommission Status : Normal

Configured Capacity: 18746441728 (17.46 GB)

DFS Used: 12288 (12 KB)

Non DFS Used: 13339152384 (12.42 GB)

DFS Remaining: 5407277056 (5.04 GB)

DFS Used%: 0.00%

DFS Remaining%: 28.84%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Sat Apr 09 02:18:50 CST 2016

Name: 192.168.10.162:50010 (hmaster)

Hostname: hmaster

Decommission Status : Normal

Configured Capacity: 18746441728 (17.46 GB)

DFS Used: 12288 (12 KB)

Non DFS Used: 13345161216 (12.43 GB)

DFS Remaining: 5401268224 (5.03 GB)

DFS Used%: 0.00%

DFS Remaining%: 28.81%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Sat Apr 09 02:18:50 CST 2016

Name: 192.168.10.166:50010 (hslave3)

Hostname: hslave3

Decommission Status : Normal

Configured Capacity: 18746441728 (17.46 GB)

DFS Used: 8192 (8 KB)

Non DFS Used: 13339144192 (12.42 GB)

DFS Remaining: 5407289344 (5.04 GB)

DFS Used%: 0.00%

DFS Remaining%: 28.84%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Sat Apr 09 02:18:51 CST 2016

你也可以通过查看 http://hmaster:50070进行查看