hadoop2.3 配置（二）

1 环境准备

Hadoop1(Master): NameNode/ResouceManager

Hadoop2(Slave): DataNode/NodeManager

Hadoop3(Slave): DataNode/NodeManager

假定三台虚拟机的IP地址如下，后面会用到。

Hadoop1:10.3.0.3

Hadoop2:10.3.0.4

Hadoop3:10.3.0.5

分别检查并修改三台虚拟机的/etc/hosts中的内容，确保包含如下配置：

10.3.0.3 hadoop1

10.3.0.4 hadoop2

10.3.0.5 hadoop3

检查三台虚拟机的/etc/hostname中的内容，分别为：

hadoop1

hadoop2

hadoop3

2 配置文件

查找hadoop-env.sh，位置在/home/username/hadoop/etc/hadoop/hadoop-env.sh，在里面找到JAVA_HOME，并将它的值设置为/usr/lib/jvm/jdk1.6.0_45。依次修改core-site.xml、yarn-site.xml、mapredsite.xml与hdfs-site.xml。

core-site.xml:

fs.default.name

hdfs://10.3.0.3:8010

true

hadoop.tmp.dir

/home/username/hadoop_space/tmp/hadoop-${user.name}

注：tmp目录需要手动创建

yarn-site.xml:

    yarn.nodemanager.aux-services

    mapreduce_shuffle

    yarn.nodemanager.aux-services.mapreduce.shuffle.class

    org.apache.hadoop.mapred.ShuffleHandler

    yarn.resourcemanager.address

    10.3.0.3:8032

    yarn.resourcemanager.scheduler.address

     10.3.0.3:8030

    yarn.resourcemanager.resource-tracker.address

     10.3.0.3:8031

    yarn.resourcemanager.admin.address

     10.3.0.3:8033

    yarn.resourcemanager.webapp.address

     10.3.0.3:8088

mapred-site.xml:

mapreduce.framework.name

yarn

mapred.system.dir

file:/home/username/hadoop_space/mapred/system

true

mapred.local.dir

file:/home/username/hadoop_space/mapred/local

true

mapred.job.tracker

10.3.0.3:54311

The host and port that theMapReduce job tracker runs

        at. If "local", thenjobs are runin-process as a single map

        and reducetask.

hdfs-site.xml:

dfs.namenode.name.dir

file:/home/username/hadoop_space/dfs/name

true

dfs.datanode.data.dir

file:/home/username/hadoop_space/dfs/data

Determines where on the local filesystem an DFS data node

        should store its blocks. If this is a comma-delimited list of

        directories, then data will be stored in all named directories,

        typically on different devices. Directories that do not exist are ignored.

true

    dfs.replication//每个块的备份副本数，默认为3

    2

    dfs.permissions

    false

slave:

将里面填入

hadoop2

hadoop3

配置文件填好后，将这些文件按位置复制到其他node节点机(slave)上。

3 无密码登录(SSH)

将以下命令分别在三台机子上做一遍：

1 终端下输入：sudo apt-get install openssh-server。

2 ssh-keygen -t rsa -P ""

3 cat.ssh/id_rsa.pub >>.ssh/authorized_keys

分别将各台机子上的.ssh/id_rsa.pub的内容追加到其他两台的.ssh/authorized_keys中，这样三台机子相互访问就不需要输入密码了。可通过ssh hadoop1来测试。

4 初始化

在master机上格式化HDFS，命令如下：

hdfs namenode –format

然后运行:start-dfs.sh、start-yarn.sh，启动完之后可分别用jps命令查看各个机子的进程是否正常，还可以通过hdfs dfsadmin -report查看dfs是否正常。

浏览器可查询：http://10.3.0.3:8088与http://10.3.0.3:50070