1、配置hadoop-env.sh(不同情况路径不同,我的情况如下)
/Users/JuneMAC/hadoop/hadoop-2.2.0/etc/hadoop/hadoop-env.sh
export
JAVA_HOME=/Library/Java/Home
export
HADOOP_CONF_DIR=/Users/JuneMAC/hadoop/hadoop-2.2.0/etc/hadoop
export
HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK
-Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk”
注:最后一个export的由来:
如果不配置HADOOP_OPTS,在通过bin/start-all.sh启动Hadoop时,会爆出下面的这个错误信息:
java[12113:1603]
Unable to load realm info from SCDynamicStore
Google一下,貌似这个问题是Mac OS
X下特有的问题。也在Hadoop的官方Jira上有记录了:https://issues.apache.org/jira/browse/HADOOP-7489
网上有一种配置可以解决这个问题,就是在conf/hadoop-env.sh中加入这么一行:
export
HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK
-Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk”
2、编辑$HOME目录下的.bash_profile(有些参考说是.bashrc但我只能是.bash_profile),添加以下项:
export
HADOOP_HOME=/Users/JuneMAC/hadoop/hadoop-2.2.0/
export
HADOOP_MAPRED_HOME=$HADOOP_HOME
export
HADOOP_COMMON_HOME=$HADOOP_HOME
export
HADOOP_HDFS_HOME=$HADOOP_HOME
export
YARN_HOME=$HADOOP_HOME
export
HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export
PATH=$PATH:$HADOOP_HOME/bin
export
HADOOP_HEAPSIZE=2000
export
HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK
-Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk"
export JAVA_HOME=/Library/Java/Home
3、添加namenode和datanode目录
mkdir
-p $HADOOP_HOME/yarn/yarn_data/hdfs/namenode
mkdir
-p $HADOOP_HOME/yarn/yarn_data/hdfs/datanode
注:请将$HADOOP_HOME替换成hadoop的安装目录
4、core-site.xml文件,配置hdfs的地址和端口号,如下:
<!--
Put site-specific property overrides in this file.
-->
fs.default.name
hdfs://localhost:9000
默认是没有中间区域的,需要自己加上。
5、编辑yarn-site.xml,加入以下配置:
yarn.resourcemanager.resource-tracker.address
localhost:8025
host is the hostname of
the resource manager and
port is the port on which the NodeManagers contact the Resource
Manager.
yarn.resourcemanager.scheduler.address
localhost:18030
host is the hostname of
the resource manager and port is the port
on which the Applications in the cluster talk to the Resource
Manager.
yarn.resourcemanager.scheduler.class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
In case you do not want
to use the default
scheduler
yarn.resourcemanager.address
localhost:18040
the host is the hostname
of the ResourceManager and the port is the port on
which the clients can talk to the Resource Manager.
yarn.nodemanager.local-dirs
/Users/JuneMAC/hadoop/hadoop-2.2.0/nodemanager-local-dirs
the local directories
used by the
nodemanager
yarn.nodemanager.address
0.0.0.0:50070
the nodemanagers bind to
this port
yarn.nodemanager.resource.memory-mb
1024
the amount of memory on
the NodeManager in
GB
yarn.nodemanager.remote-app-log-dir
/Users/JuneMAC/hadoop/hadoop-2.2.0/app-logs
directory on hdfs where
the application logs are moved to
yarn.nodemanager.log-dirs
/Users/JuneMAC/hadoop/hadoop-2.2.0/nodemanager-log-dirs
the directories used by
Nodemanagers as log
directories
yarn.nodemanager.aux-services
mapreduce_shuffle
shuffle service that
needs to be set for Map Reduce to run
注:根据需要,可以更改端口号以及本地文件目录。如果使用了本地文件目录,最好事先创建好这些对应的目录。比如上面的配置文件中,我们使用了/Users/JuneMAC/hadoop/hadoop-2.2.0/nodemanager-local-dirs这个目录。
6、编辑hdfs-site.xml,添加以下配置:
dfs.replication
1
dfs.namenode.name.dir
/Users/JuneMAC/hadoop/hadoop-2.2.0/yarn/yarn_data/hdfs/namenode
dfs.datanode.data.dir
/Users/JuneMAC/hadoop/hadoop-2.2.0/yarn/yarn_data/hdfs/datanode
7、编辑mapred-site.xml,添加以下配置:
mapreduce.cluster.temp.dir
No
description
true
mapreduce.cluster.local.dir
No
description
true
注:如果该文件不存在,可通过复制mapred-site.xml.template文件而得。
8、初始化noamenode
hadoop
namenode -format
9、启动hadoop守护进程
sh
hadoop start-all.sh
此时报错:
WARN
util.NativeCodeLoader: Unable to load native-hadoop library for
your platform... using builtin-java classes where
applicable
原因是os是64位机,hadoop安装包是32位的,需要重新编译
可以file查看一下libhadoop.so.1.0.0的信息file
/Users/JuneMAC/hadoop/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0
返回:
/Users/JuneMAC/hadoop/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0:
ELF
32-bit LSB
shared object, Intel 80386, version 1 (SYSV), dynamically linked,
not stripped
解决方法:重新编译hadoop
这个话就长了,再开一篇来讲。
<!-- 正文结束 -->
来自 “ ITPUB博客 ” ,链接:https://blog.itpub.net/23600510/viewspace-1114382/
,如需转载,请注明出处,否则将追究法律责任。