1:下载Hadoop安装包
wget http://mirror.esocc.com/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0.tar.gz
2:下载Jdk
wget https://edelivery.oracle.com/otn-pub/java/jdk/7u45-b18/jdk-7u45-linux-i586.tar.gz
3:redhat6.3配置JDK
步骤一:tar
-zxvf
jdk-7u45-linux-i586.tar.gz
步骤二:mv
jdk-xxxs
jdk1.7
步骤三:配置环境变量
vim /etc/profile
添加内容如下:
# set java environment
export JAVA_HOME=/opt/software/jdk1.7
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
步骤四:让profile文件生效
source /etc/profile
步骤五:配置默认JDK
update-alternatives --install /usr/bin/java
java /opt/software/jdk1.7/bin/java
300
update-alternatives --install /usr/bin/javac
javac /opt/software/jdk1.7/bin/javac
300
update-alternatives --install /usr/bin/jar
jar /opt/software/jdk1.7/bin/jar
300
update-alternatives --install /usr/bin/javah
javah /opt/software/jdk1.7/bin/javah
300
update-alternatives --install /usr/bin/javap
javap /opt/software/jdk1.7/bin/javap
300
步骤六:验证
java -version
java version "1.7.0_45"
Java(TM) SE
Runtime Environment (build
1.7.0_45-b18)
Java HotSpot(TM) Server VM
(build
24.45-b08, mixed
mode)
4:解压hadoop-2.2.0安装包
tar -zxvf
hadoop-2.2.0.tar.gz
mv hadoopxxxxx hadoop2.0
5:修改hadoop配置文件,配置文件所在位置/opt/software/hadoop2.0/etc/hadoop
步骤一:修改hadoop-env.sh,添加java安装目录
export JAVA_HOME=/opt/software/jdk1.7
步骤二:修改mapred-site.xml,修改此文件之前记得要重命名。添加内容如下:
mapreduce.framework.name</name>
yarnvalue>
</property>
configuration>
和hadoop1.0比较,用户无须配置mapred.job.tarcker。因为在2.0中统一由资源调度框架yarn管理。所以只需要指定运行时框架yarn即可。
步骤三:修改core-site.xml
。添加内容如下:
fs.default.name</name>
hdfs://hadoop:8020</value>
property>
hadoop.tmp.dir</name>
/opt/software/hadoop2.0/tmp</value>
property>
这里简单的配置下hdfs的工作模式,并没配置hdfs
federation等特性。
步骤四:修改yarn-site.xml。添加内容如下:
yarn.nodemanager.aux-services</name>
mapreduce_shufflevalue>
</property>
为了能够运行MapReduce工作,需要让各个nodemanager在启动时加载shuffle server。
步骤五:修改hdfs-site.xml。添加内容如下:
dfs.replicationname>
1</value>
property>
dfs.namenode.name.dir</name>
/opt/software/hadoop2.0/data/dfs/name</value>
property>
dfs.datanode.data.dir</name>
/opt/software/hadoop2.0/data/dfs/data</value>
property>
配置下文件块的备份策略(默认是3),以及hdfs格式换之后的目录。
6:初始化Hdfs,命令在/opt/software/hadoop2.0/bin目录下
./hadoop
namenode -format
7:启动hadoop2.0,命令在/opt/software/hadoop2.0/sbin
./start-all.sh
8:启动后的jps进程
8084 ResourceManager
7944 SecondaryNameNode
7776 DataNode
7683 NameNode
问题
1:nodemanager无法启动。
报错信息如下:The ServiceName: mapreduce-shuffle set in yarn.nodemanager.aux-services is invalid.The valid service
name should only contain a-zA-Z0-9_
and can
not start
with numbers
解决方案:
修改yarn-site.xml文件:
yarn.nodemanager.aux-services</name>
mapreduce-shufflevalue>
</property>
改成
yarn.nodemanager.aux-servicesname>
mapreduce_shuffle</value>
property>
另外关于maprduce.shuffle,也有一个解决方案:The correct
value is now
mapreduce_shuffle instead of mapreduce.shuffle