ITPub博客

首页 > 大数据 > Hadoop > [hadoop实战]安装并运行hadoop

[hadoop实战]安装并运行hadoop

Hadoop 作者:kiloperottix 时间:2013-12-10 22:42:56 0 删除 编辑
1、配置hadoop-env.sh(不同情况路径不同,我的情况如下) 
/Users/JuneMAC/hadoop/hadoop-2.2.0/etc/hadoop/hadoop-env.sh 
export JAVA_HOME=/Library/Java/Home
export HADOOP_CONF_DIR=/Users/JuneMAC/hadoop/hadoop-2.2.0/etc/hadoop
export HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk” 
注:最后一个export的由来:
如果不配置HADOOP_OPTS,在通过bin/start-all.sh启动Hadoop时,会爆出下面的这个错误信息:
java[12113:1603] Unable to load realm info from SCDynamicStore
Google一下,貌似这个问题是Mac OS X下特有的问题。也在Hadoop的官方Jira上有记录了:https://issues.apache.org/jira/browse/HADOOP-7489
网上有一种配置可以解决这个问题,就是在conf/hadoop-env.sh中加入这么一行:
export HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk” 

2、编辑$HOME目录下的.bash_profile(有些参考说是.bashrc但我只能是.bash_profile),添加以下项:
export HADOOP_HOME=/Users/JuneMAC/hadoop/hadoop-2.2.0/
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export HADOOP_HEAPSIZE=2000
export HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk"
export JAVA_HOME=/Library/Java/Home

3、添加namenode和datanode目录
mkdir -p $HADOOP_HOME/yarn/yarn_data/hdfs/namenode
mkdir -p $HADOOP_HOME/yarn/yarn_data/hdfs/datanode
注:请将$HADOOP_HOME替换成hadoop的安装目录

4、core-site.xml文件,配置hdfs的地址和端口号,如下:
   
   
<!-- Put site-specific property overrides in this file. -->   
   
       
        fs.default.name   
        hdfs://localhost:9000   
       
 
默认是没有中间区域的,需要自己加上。

5、编辑yarn-site.xml,加入以下配置:
        
                yarn.resourcemanager.resource-tracker.address
                localhost:8025
                host is the hostname of the resource manager and
                        port is the port on which the NodeManagers contact the Resource Manager.
               
       

       
                yarn.resourcemanager.scheduler.address
                localhost:18030
                host is the hostname of the resource manager and port is the port
                        on which the Applications in the cluster talk to the Resource Manager.
               
       

       
                yarn.resourcemanager.scheduler.class
                org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
                In case you do not want to use the default scheduler
       

       
                yarn.resourcemanager.address
                localhost:18040
                the host is the hostname of the ResourceManager and the port is the port on
                        which the clients can talk to the Resource Manager.
       

       
                yarn.nodemanager.local-dirs
                /Users/JuneMAC/hadoop/hadoop-2.2.0/nodemanager-local-dirs
                the local directories used by the nodemanager
       
       
                yarn.nodemanager.address
                0.0.0.0:50070
                the nodemanagers bind to this port
       

       
                yarn.nodemanager.resource.memory-mb
                1024
                the amount of memory on the NodeManager in GB
       

       
                yarn.nodemanager.remote-app-log-dir
                /Users/JuneMAC/hadoop/hadoop-2.2.0/app-logs
                directory on hdfs where the application logs are moved to
       

       
                yarn.nodemanager.log-dirs
                /Users/JuneMAC/hadoop/hadoop-2.2.0/nodemanager-log-dirs
                the directories used by Nodemanagers as log directories
       

       
                yarn.nodemanager.aux-services
                mapreduce_shuffle
                shuffle service that needs to be set for Map Reduce to run
 
        
注:根据需要,可以更改端口号以及本地文件目录。如果使用了本地文件目录,最好事先创建好这些对应的目录。比如上面的配置文件中,我们使用了/Users/JuneMAC/hadoop/hadoop-2.2.0/nodemanager-local-dirs这个目录。

6、编辑hdfs-site.xml,添加以下配置:
        
                dfs.replication
                1
       

       
                dfs.namenode.name.dir
                /Users/JuneMAC/hadoop/hadoop-2.2.0/yarn/yarn_data/hdfs/namenode
       

       
                dfs.datanode.data.dir
                /Users/JuneMAC/hadoop/hadoop-2.2.0/yarn/yarn_data/hdfs/datanode
       
7、编辑mapred-site.xml,添加以下配置:
       
                mapreduce.cluster.temp.dir
               
                No description
                true
       

       
                mapreduce.cluster.local.dir
               
                No description
                true
       
注:如果该文件不存在,可通过复制mapred-site.xml.template文件而得。

8、初始化noamenode
hadoop namenode -format

9、启动hadoop守护进程

sh hadoop start-all.sh
此时报错:
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

原因是os是64位机,hadoop安装包是32位的,需要重新编译
可以file查看一下libhadoop.so.1.0.0的信息file /Users/JuneMAC/hadoop/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0
返回:
/Users/JuneMAC/hadoop/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped 

解决方法:重新编译hadoop 

这个话就长了,再开一篇来讲。


<!-- 正文结束 -->

来自 “ ITPUB博客 ” ,链接:https://blog.itpub.net/23600510/viewspace-1114382/ ,如需转载,请注明出处,否则将追究法律责任。

上一篇: 没有了~
下一篇: 没有了~
请登录后发表评论 登录
全部评论

注册时间:2010-03-27