一、规划
使用VMWare Workstation创建3台虚拟机。定义如下表：
|
|
Hostname |
IP |
OS |
|
Master |
n1 |
192.168.10.101 |
Cent OS 5 |
|
Slave1 |
d1 |
192.168.10.102 |
Cent OS 5 |
|
Slave2 |
d2 |
192.168.10.103 |
Cent OS 5 |
二、前期准备
1、在所有主机上，创建相同的用户（组）
[root@n1 ~]# groupadd -g 500 hadoop
[root@n1 ~]# useradd -u 500 -g hadoop -G hadoop -d /home/hadoop hadoop
[root@n1 ~]# passwd hadoop
Changing password for user hadoop.
New UNIX password:
BAD PASSWORD: it is based on a dictionary word
Retype new UNIX password:
passwd: all authentication tokens updated successfully.
编辑/etc/hosts文件，形如：
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
192.168.10.101 n1 n1
192.168.10.102 d1 d1
192.168.10.103 d2 d2
2、安装JDK
使用WinSCP将JDK1.6安装包传送到服务器主机上。
[root@n1 ~]# chown -R hadoop:hadoop /software
[root@n1 ~]# su - hadoop
[hadoop@n1 ~]$ ll /software/
total 70144
-rw-r--r-- 1 hadoop hadoop 71748681 Jul 15 22:33 jdk-6u33-linux-i586.bin
[hadoop@n1 ~]$ chmod -R 755 /software/
[hadoop@n1 ~]$ ll /software/
total 70144
-rwxr-xr-x 1 hadoop hadoop 71748681 Jul 15 22:33 jdk-6u33-linux-i586.bin
然后在另外两个节点执行下面命令，使用scp复制jdk安装包。
[root@d2 ~]# mkdir /software
[root@d2 ~]# scp root@192.168.10.101:/software/jdk-6u33-linux-i586.bin /software/.
The authenticity of host '192.168.10.101 (192.168.10.101)' can't be established.
RSA key fingerprint is 48:ef:fc:67:3f:43:88:0e:fa:e6:a6:ec:75:b0:c4:7f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.10.101' (RSA) to the list of known hosts.
root@192.168.10.101's password:
jdk-6u33-linux-i586.bin 100% 68MB 5.3MB/s 00:13
现在开始进行jdk安装。
[root@n1 ~]# su - hadoop
[hadoop@n1 ~]$ cd /software/
[hadoop@n1 software]$ ll
total 70144
-rwxr-xr-x 1 hadoop hadoop 71748681 Oct 9 02:20 jdk-6u33-linux-i586.bin
[hadoop@n1 software]$ ./jdk-6u33-linux-i586.bin
Unpacking...
Checksumming...
。。。。。。。
Java(TM) SE Development Kit 6 successfully installed.
Product Registration is FREE and includes many benefits:
* Notification of new versions, patches, and updates
* Special offers on Oracle products, services and training
* Access to early releases and documentation
Product and system data will be collected. If your configuration
supports a browser, the JDK Product Registration form. will
be presented. If you do not register, none of this information
will be saved. You may also register your JDK later by
opening the register.html file (located in the JDK installation
directory) in a browser.
For more information on what data Registration collects and
how it is managed and used, see:
http://java.sun.com/javase/registration/JDKRegistrationPrivacy.html
Press Enter to continue.....
Done.
配置JDK的环境变量。在.bash_profile文件中加入如下代码。
JAVA_HOME=/software/jdk1.6.0_33
CLASSPATH=.:$JAVA_HOME/lib
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME CLASSPATH PATH
别忘了执行
[hadoop@n1 ~]$ . .bash_profile
3、配置SSH互信
在n1上执行
[hadoop@n1 ~]$ mkdir .ssh
[hadoop@n1 ~]$ chmod 700 .ssh
[hadoop@n1 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
57:f2:fe:1b:e9:21:9f:fa:84:55:4c:13:37:d2:10:48 hadoop@n1
[hadoop@n1 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_dsa.
Your public key has been saved in /home/hadoop/.ssh/id_dsa.pub.
The key fingerprint is:
24:6e:83:17:f0:5d:72:8f:00:9e:8b:7c:ed:28:9d:88 hadoop@n1
[hadoop@n1 ~]$ touch .ssh/authorized_keys
在d1上执行
[hadoop@d1 ~]$ mkdir .ssh
[hadoop@d1 ~]$ chmod 700 .ssh
[hadoop@d1 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
83:bc:0d:3b:fc:b8:53:e6:aa:cc:46:ed:2b:8d:8f:17 hadoop@d1
[hadoop@d1 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_dsa.
Your public key has been saved in /home/hadoop/.ssh/id_dsa.pub.
The key fingerprint is:
86:7b:5e:66:99:22:7f:2c:38:07:08:73:91:70:3a:cb hadoop@d1
在d2上执行
[hadoop@d2 ~]$ mkdir .ssh
[hadoop@d2 ~]$ chmod 700 .ssh
[hadoop@d2 ~]$
[hadoop@d2 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
50:c1:c8:fa:8e:c0:0e:3e:eb:26:dc:23:ee:1d:27:fe hadoop@d2
[hadoop@d2 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_dsa.
Your public key has been saved in /home/hadoop/.ssh/id_dsa.pub.
The key fingerprint is:
ae:f4:a2:96:2a:f2:24:33:c0:bc:63:08:2c:ef:af:51 hadoop@d2
然后在n1上继续下面的执行。
[hadoop@n1 ~]$ cd .ssh
[hadoop@n1 .ssh]$ ll
total 16
-rw-rw-r-- 1 hadoop hadoop 0 Oct 9 02:49 authorized_keys
-rw------- 1 hadoop hadoop 668 Oct 9 02:48 id_dsa
-rw-r--r-- 1 hadoop hadoop 599 Oct 9 02:48 id_dsa.pub
-rw------- 1 hadoop hadoop 1675 Oct 9 02:48 id_rsa
-rw-r--r-- 1 hadoop hadoop 391 Oct 9 02:48 id_rsa.pub
[hadoop@n1 .ssh]$ ssh n1 cat /home/hadoop/.ssh/id_rsa.pub >> authorized_keys
The authenticity of host 'n1 (192.168.10.101)' can't be established.
RSA key fingerprint is 48:ef:fc:67:3f:43:88:0e:fa:e6:a6:ec:75:b0:c4:7f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'n1,192.168.10.101' (RSA) to the list of known hosts.
hadoop@n1's password:
[hadoop@n1 .ssh]$ ssh n1 cat /home/hadoop/.ssh/id_dsa.pub >> authorized_keys
hadoop@n1's password:
[hadoop@n1 .ssh]$ ssh d1 cat /home/hadoop/.ssh/id_rsa.pub >> authorized_keys
The authenticity of host 'd1 (192.168.10.102)' can't be established.
RSA key fingerprint is 48:ef:fc:67:3f:43:88:0e:fa:e6:a6:ec:75:b0:c4:7f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'd1,192.168.10.102' (RSA) to the list of known hosts.
hadoop@d1's password:
[hadoop@n1 .ssh]$ ssh d1 cat /home/hadoop/.ssh/id_dsa.pub >> authorized_keys
hadoop@d1's password:
[hadoop@n1 .ssh]$ ssh d2 cat /home/hadoop/.ssh/id_rsa.pub >> authorized_keys
The authenticity of host 'd2 (192.168.10.103)' can't be established.
RSA key fingerprint is 48:ef:fc:67:3f:43:88:0e:fa:e6:a6:ec:75:b0:c4:7f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'd2,192.168.10.103' (RSA) to the list of known hosts.
hadoop@d2's password:
[hadoop@n1 .ssh]$ ssh d2 cat /home/hadoop/.ssh/id_dsa.pub >> authorized_keys
hadoop@d2's password:
[hadoop@n1 .ssh]$ ll
total 24
-rw-rw-r-- 1 hadoop hadoop 2970 Oct 9 02:56 authorized_keys
-rw------- 1 hadoop hadoop 668 Oct 9 02:48 id_dsa
-rw-r--r-- 1 hadoop hadoop 599 Oct 9 02:48 id_dsa.pub
-rw------- 1 hadoop hadoop 1675 Oct 9 02:48 id_rsa
-rw-r--r-- 1 hadoop hadoop 391 Oct 9 02:48 id_rsa.pub
-rw-r--r-- 1 hadoop hadoop 1197 Oct 9 02:56 known_hosts
然后，在3个节点上分别执行下面操作。
[hadoop@n1 ~]$ chmod 600 .ssh/authorized_keys
[hadoop@n1 ~]$ exec /ssh-agent $SHELL
-bash: /ssh-agent: No such file or directory
-bash: exec: /ssh-agent: cannot execute: No such file or directory
[hadoop@n1 ~]$ exec ssh-agent $SHELL
[hadoop@n1 ~]$ ssh-add
Identity added: /home/hadoop/.ssh/id_rsa (/home/hadoop/.ssh/id_rsa)
Identity added: /home/hadoop/.ssh/id_dsa (/home/hadoop/.ssh/id_dsa)
使用ssh 『机器名』date命令，验证SSH互信的建立。
三、安装hadoop
JDK和SSH已经安装配置完成，可以开始部署Hadoop了。
再次使用WinSCP，将hadoop-0.20.2.tar.gz传送到服务器上；然后使用scp指令传送到其它节点上。
[hadoop@n1 software]$ gunzip hadoop-0.20.2.tar.gz
[hadoop@n1 software]$ tar -xvf hadoop-0.20.2.tar
。。。。。
[hadoop@n1 software]$ mv hadoop-0.20.2 /home/hadoop/.
[hadoop@n1 ~]$ scp /software/hadoop-0.20.2.tar hadoop@192.168.10.102:/home/hadoop/.
hadoop-0.20.2.tar 100% 129MB 6.2MB/s 00:21
[hadoop@n1 ~]$ scp /software/hadoop-0.20.2.tar hadoop@192.168.10.103:/home/hadoop/.
hadoop-0.20.2.tar 100% 129MB 5.6MB/s 00:23
现在3个节点上的hadoop解压完成，下面分别进行如下配置。
1）conf/Hadoop-env.sh文件的修改，设置JAVA_HOME。
# The java implementation to use. Required.
export JAVA_HOME=/software/jdk1.6.0_33
2）修改conf/core-site.xml
[hadoop@n1 conf]$ mkdir -p /software/hdata
3）修改conf/hdfs-site.xml
4）修改conf/mapred-site.xml
5）修改conf/masters
n1
6）修改conf/slaves
d1
d2
7）将conf目录及修改的文件复制到其它两个节点的相应目录中
[hadoop@n1 conf]$ scp *.* d1:/home/hadoop/hadoop-0.20.2/conf/.
[hadoop@n1 conf]$ scp *.* d2:/home/hadoop/hadoop-0.20.2/conf/.
8）格式化hdfs
[hadoop@n1 hadoop-0.20.2]$ bin/hadoop namenode -format
12/10/09 04:12:40 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = n1/192.168.10.101
。。。。。。
12/10/09 04:12:41 INFO common.Storage: Storage directory /tmp/hadoop-hadoop/dfs/name has been successfully formatted.
12/10/09 04:12:41 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at n1/192.168.10.101
************************************************************/
9）启动hadoop
[hadoop@n1 hadoop-0.20.2]$ bin/start-all.sh
starting namenode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-namenode-n1.out
d1: starting datanode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-datanode-d1.out
d2: starting datanode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-datanode-d2.out
n1: starting secondarynamenode, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-secondarynamenode-n1.out
starting jobtracker, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-jobtracker-n1.out
d2: starting tasktracker, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-tasktracker-d2.out
d1: starting tasktracker, logging to /home/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hadoop-tasktracker-d1.out
[hadoop@n1 hadoop-0.20.2]$
启动成功，没有报错。
四、验证
方法一：
[hadoop@n1 hadoop-0.20.2]$ bin/hadoop dfsadmin -report
Configured Capacity: 37176967168 (34.62 GB)
。。。。。。
DFS Remaining%: 75.57%
Last contact: Tue Oct 09 04:15:35 CST 2012
方法二：
[hadoop@n1 hadoop-0.20.2]$ /software/jdk1.6.0_33/bin/jps
30673 NameNode
。。。。。
[hadoop@n1 hadoop-0.20.2]$ ssh d1 /software/jdk1.6.0_33/bin/jps
30472 DataNode
。。。。。。
[hadoop@n1 hadoop-0.20.2]$ ssh d2 /software/jdk1.6.0_33/bin/jps
30339 DataNode
。。。。。。
方法三：
使用浏览器，输入n1:50070或n1:50030
自己使用浏览器看看吧，这里就不贴图啦！哈哈......
来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/26812308/viewspace-746172/ ，如需转载，请注明出处，否则将追究法律责任。
