ITPub博客

首页 > 大数据 > Hadoop > Hadoop-2.3.0-cdh5.0.1完全分布式环境搭建(NameNode,ResourceManager HA)

Hadoop-2.3.0-cdh5.0.1完全分布式环境搭建(NameNode,ResourceManager HA)

Hadoop 作者:hackeruncle 时间:2016-02-10 17:19:29 0 删除 编辑
1.YARN的基础架构是什么?
2.哪一进程防止Yarn早就web攻击?
3.Zookeeper主进程是哪个?
4.如何下载cdh包?





本文搭建Hadoop CDH5.0.1 分布式系统,包括NameNode ,ResourceManger HA,忽略了Web Application Proxy 和Job HistoryServer。 
一概述(一)HDFS

1)基础架构

(1)NameNode(Master)
  • 命名空间管理:命名空间支持对HDFS中的目录、文件和块做类似文件系统的创建、修改、删除、列表文件和目录等基本操作。
  • 块存储管理

(2)DataNode(Slaver)
namenode和client的指令进行存储或者检索block,并且周期性的向namenode节点报告它存了哪些文件的block

2)HA架构

 


使用Active NameNode,Standby NameNode 两个结点解决单点问题,两个结点通过JounalNode共享状态,通过ZKFC 选举Active ,监控状态,自动备援。

(1)Active NameNode:
接受client的RPC请求并处理,同时写自己的Editlog和共享存储上的Editlog,接收DataNode的Block report, block location updates和heartbeat;

(2)Standby NameNode:
      同样会接到来自DataNode的Block report, block location updates和heartbeat,同时会从共享存储的Editlog上读取并执行这些log操作,使得自己的NameNode中的元数据(Namespcae information + Block locations map)都是和Active NameNode中的元数据是同步的。所以说Standby模式的NameNode是一个热备(Hot Standby NameNode),一旦切换成Active模式,马上就可以提供NameNode服务

(3)JounalNode:
用于Active NameNode , Standby NameNode 同步数据,本身由一组JounnalNode结点组成,该组结点基数个,支持Paxos协议,保证高可用,是CDH5唯一支持的共享方式(相对于CDH4 促在NFS共享方式)

(4)ZKFC:
监控NameNode进程,自动备援。

(二)YARN

1)基础架构

(1)ResourceManager(RM)
接收客户端任务请求,接收和监控NodeManager(NM)的资源情况汇报,负责资源的分配与调度,启动和监控ApplicationMaster(AM)。

(2)NodeManager
节点上的资源管理,启动Container运行task计算,上报资源、container情况给RM和任务处理情况给AM。

(3)ApplicationMaster
单个Application(Job)的task管理和调度,向RM进行资源的申请,向NM发出launch Container指令,接收NM的task处理状态信息。NodeManager

(4)Web Application Proxy
用于防止Yarn遭受Web攻击,本身是ResourceManager的一部分,可通过配置独立进程。ResourceManager Web的访问基于守信用户,当Application Master运行于一个非受信用户,其提供给ResourceManager的可能是非受信连接,Web Application Proxy可以阻止这种连接提供给RM。

(5)Job History Server
NodeManager在启动的时候会初始化LogAggregationService服务, 该服务会在把本机执行的container log (在container结束的时候)收集并存放到hdfs指定的目录下. ApplicationMaster会把jobhistory信息写到hdfs的jobhistory临时目录下, 并在结束的时候把jobhisoty移动到最终目录, 这样就同时支持了job的recovery.History会启动web和RPC服务, 用户可以通过网页或RPC方式获取作业的信息

2)HA架构

 


      ResourceManager HA 由一对Active,Standby结点构成,通过RMStateStore存储内部数据和主要应用的数据及标记。目前支持的可替代的RMStateStore实现有:基于内存的MemoryRMStateStore,基于文件系统的FileSystemRMStateStore,及基于zookeeper的ZKRMStateStore。
ResourceManager HA的架构模式同NameNode HA的架构模式基本一致,数据共享由RMStateStore,而ZKFC成为 ResourceManager进程的一个服务,非独立存在。

二 规划

(一)版本
组件名
版本
说明
JRE
java version "1.7.0_60"
Java(TM) SE Runtime Environment (build 1.7.0_60-b19)
Java HotSpot(TM) Client VM (build 24.60-b09, mixed mode)

Hadoop
hadoop-2.3.0-cdh5.0.1.tar.gz
主程序包

Zookeeper
zookeeper-3.4.5-cdh5.0.1.tar.gz
热切,Yarn 存储数据使用的协调服务



(二)主机规划
IP
Host
部署模块
进程
8.8.8.11
Hadoop-NN-01
NameNode
ResourceManager
NameNode
DFSZKFailoverController
ResourceManager
8.8.8.12
Hadoop-NN-02
NameNode
ResourceManager
NameNode
DFSZKFailoverController
ResourceManager
8.8.8.13
Hadoop-DN-01
Zookeeper-01
DataNode
NodeManager
Zookeeper
DataNode
NodeManager
JournalNode
QuorumPeerMain
8.8.8.14
Hadoop-DN-02
Zookeeper-02
DataNode
NodeManager
Zookeeper
DataNode
NodeManager
JournalNode
QuorumPeerMain
8.8.8.15
Hadoop-DN-03
Zookeeper-03
DataNode
NodeManager
Zookeeper
DataNode
NodeManager
JournalNode
QuorumPeerMain

各个进程解释:
  • NameNode
  • ResourceManager
  • DFSZKFC:DFS Zookeeper Failover Controller 激活Standby NameNode
  • DataNode
  • NodeManager
  • JournalNode:NameNode共享editlog结点服务(如果使用NFS共享,则该进程和所有启动相关配置接可省略)。
  • QuorumPeerMain:Zookeeper主进程


(三)目录规划
名称
路径
$HADOOP_HOME
/home/zero/hadoop/hadoop-2.3.0-cdh5.0.1
Data
$ HADOOP_HOME/data
Log
$ HADOOP_HOME/logs


三 环境准备

1)关闭防火墙


root 用户:

  1. [root@CentOS-Cluster-01 hadoop-2.3.0-cdh5.0.1]# service iptables stop
  2. iptables: Flushing firewall rules: [  OK  ]
  3. iptables: Setting chains to policy ACCEPT: filter [  OK  ]
  4. iptables: Unloading modules: [  OK  ]
复制代码




验证:

  1. [root@CentOS-Cluster-01 hadoop-2.3.0-cdh5.0.1]# service  iptables status
  2. iptables: Firewall is not running.
复制代码





2)安装JRE:略
3)安装Zookeeper :参见Zookeeper-3.4.5-cdh5.0.1 单机模式、副本模式安装指导

4)配置SSH互信:

(1)Hadoop-NN-01创建密钥:
  1. [zero@CentOS-Cluster-01 ~]$ ssh-keygen
  2. Generating public/private rsa key pair.
  3. Enter file in which to save the key (/home/zero/.ssh/id_rsa):
  4. Created directory '/home/zero/.ssh'.
  5. Enter passphrase (empty for no passphrase):
  6. Enter same passphrase again:
  7. Your identification has been saved in /home/zero/.ssh/id_rsa.
  8. Your public key has been saved in /home/zero/.ssh/id_rsa.pub.
  9. The key fingerprint is:
  10. 28:0a:29:1d:98:56:55:db:ec:83:93:56:8a:0f:6c:ea zero@CentOS-Cluster-01
  11. The key's randomart image is:
  12. +--[ RSA 2048]----+
  13. |   .....         |
  14. | o.     +        |
  15. |o..    . +       |
  16. |.o .. ..*        |
  17. |+ . .=.*So       |
  18. |.. .o.+ . .      |
  19. |  ..   .         |
  20. |  .              |
  21. |   E             |
  22. +-----------------+
复制代码



(2)分发密钥:
  1. [zero@CentOS-Cluster-01 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub zero@Hadoop-NN-01
  2. The authenticity of host 'hadoop-nn-01 (8.8.8.11)' can't be established.
  3. RSA key fingerprint is a6:11:09:49:8c:fe:b2:fb:49:d5:01:fa:13:1b:32:24.
  4. Are you sure you want to continue connecting (yes/no)? yes
  5. Warning: Permanently added 'hadoop-nn-01,8.8.8.11' (RSA) to the list of known hosts.
  6. puppet@hadoop-nn-01's password:
  7. Permission denied, please try again.
  8. puppet@hadoop-nn-01's password:
  9. [zero@CentOS-Cluster-01 ~][        DISCUZ_CODE_105        ]nbsp; ssh-copy-id -i ~/.ssh/id_rsa.pub zero@Hadoop-NN-01
  10. zero@hadoop-nn-01's password:
  11. Now try logging into the machine, with "ssh 'zero@Hadoop-NN-01'", and check in:

  12.   .ssh/authorized_keys

  13. to make sure we haven't added extra keys that you weren't expecting.
  14. [zero@CentOS-Cluster-01 ~][        DISCUZ_CODE_105        ]nbsp; ssh-copy-id -i ~/.ssh/id_rsa.pub zero@Hadoop-NN-02
  15. (…略…)
  16. [zero@CentOS-Cluster-01 ~][        DISCUZ_CODE_105        ]nbsp; ssh-copy-id -i ~/.ssh/id_rsa.pub zero@Hadoop-DN-01
  17. (…略…)
  18. [zero@CentOS-Cluster-01 ~][        DISCUZ_CODE_105        ]nbsp; ssh-copy-id -i ~/.ssh/id_rsa.pub zero@Hadoop-DN-02
  19. (…略…)
  20. [zero@CentOS-Cluster-01 ~][        DISCUZ_CODE_105        ]nbsp; ssh-copy-id -i ~/.ssh/id_rsa.pub zero@Hadoop-DN-03
  21. (…略…)
复制代码
分发密钥这里如出现问题,可以参考:
linux(ubuntu)ssh无密码互通、相互登录高可靠文档

CentOS6.4之图解SSH无验证双向登陆配置


(3)验证:
  1. [zero@CentOS-Cluster-01 ~]$ ssh Hadoop-NN-01
  2. Last login: Sun Jun 22 19:56:23 2014 from 8.8.8.1
  3. [zero@CentOS-Cluster-01 ~]$ exit
  4. logout
  5. Connection to Hadoop-NN-01 closed.
  6. [zero@CentOS-Cluster-01 ~]$ ssh Hadoop-NN-02
  7. Last login: Sun Jun 22 20:03:31 2014 from 8.8.8.1
  8. [zero@CentOS-Cluster-03 ~]$ exit
  9. logout
  10. Connection to Hadoop-NN-02 closed.
  11. [zero@CentOS-Cluster-01 ~]$ ssh Hadoop-DN-01
  12. Last login: Mon Jun 23 02:00:07 2014 from centos_cluster_01
  13. [zero@CentOS-Cluster-03 ~]$ exit
  14. logout
  15. Connection to Hadoop-DN-01 closed.
  16. [zero@CentOS-Cluster-01 ~]$ ssh Hadoop-DN-02
  17. Last login: Sun Jun 22 20:07:03 2014 from 8.8.8.1
  18. [zero@CentOS-Cluster-04 ~]$ exit
  19. logout
  20. Connection to Hadoop-DN-02 closed.
  21. [zero@CentOS-Cluster-01 ~]$ ssh Hadoop-DN-03
  22. Last login: Sun Jun 22 20:07:05 2014 from 8.8.8.1
  23. [zero@CentOS-Cluster-05 ~]$ exit
  24. logout
  25. Connection to Hadoop-DN-03 closed.
复制代码





5)配置/etc/hosts并分发:
  1. [root@CentOS-Cluster-01 zero]# vi /etc/hosts
  2. 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
  3. ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
  4. 8.8.8.10 CentOS-StandAlone
  5. 8.8.8.11 CentOS-Cluster-01 Hadoop-NN-01
  6. 8.8.8.12 CentOS-Cluster-02 Hadoop-NN-02
  7. 8.8.8.13 CentOS-Cluster-03 Hadoop-DN-01 Zookeeper-01
  8. 8.8.8.14 CentOS-Cluster-04 Hadoop-DN-02 Zookeeper-02
  9. 8.8.8.15 CentOS-Cluster-05 Hadoop-DN-03 Zookeeper-03
复制代码




6)配置环境变量:vi ~/.bashrc 然后 source ~/.bashrc

  1. [zero@CentOS-Cluster-01 ~]$ vi ~/.bashrc
  2. ……

  3. # hadoop cdh5
  4. export HADOOP_HOME=/home/zero/hadoop/hadoop-2.3.0-cdh5.0.1
  5. export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
  6. [zero@CentOS-Cluster-01 ~]$ source ~/.bashrc
复制代码




四 安装

1)解压
  1. [puppet@BigData-01 cdh4.4]$ tar -xvf hadoop-2.0.0-cdh4.4.0.tar.gz
复制代码

下载地址:
CDH4包汇总
http://archive.cloudera.com/cdh4/
CDH5包汇总
http://archive.cloudera.com/cdh5/


2)修改配置文件
说明:

配置名称
类型
说明
hadoop-env.sh
Bash脚本
Hadoop运行环境变量设置
core-site.xml
xml
配置Hadoop core,如IO
hdfs-site.xml
xml
配置HDFS守护进程:NN、JN、DN
yarn-env.sh
Bash脚本
Yarn运行环境变量设置
yarn-site.xml
xml
Yarn框架配置环境
mapred-site.xml
xml
MR属性设置
capacity-scheduler.xml
xml
Yarn调度属性设置
container-executor.cfg
Cfg
Yarn Container配置
mapred-queues.xml
xml
MR队列设置
hadoop-metrics.properties
Java属性
Hadoop Metrics配置
hadoop-metrics2.properties
Java属性
Hadoop Metrics配置
slaves
Plain Text
DN节点配置
exclude
Plain Text
移除DN节点配置文件
log4j.properties

系统日志设置
configuration.xsl




(1)修改$HADOOP_HOME/etc/hadoop-env.sh:

  1. #--------------------Java Env------------------------------
  2. export JAVA_HOME="/usr/runtime/java/jdk1.7.0_60"

  3. #--------------------Hadoop Env------------------------------
  4. #export HADOOP_PID_DIR=
  5. export HADOOP_PREFIX="/home/zero/hadoop/hadoop-2.3.0-cdh5.0.1"

  6. #--------------------Hadoop Daemon Options-----------------
  7. #export HADOOP_NAMENODE_OPTS="-XX:+UseParallelGC ${HADOOP_NAMENODE_OPTS}"
  8. #export HADOOP_DATANODE_OPTS=

  9. #--------------------Hadoop Logs---------------------------
  10. #export HADOOP_LOG_DIR=
复制代码



(2)修改$HADOOP_HOME/etc/hadoop-site.xml




  1.         
  2.         
  3.                 fs.defaultFS
  4.                 hdfs://mycluster
  5.         
  6.         
  7.         
  8.                 dfs.permissions.superusergroup
  9.                 zero
  10.         
  11.         
  12.         
  13.                
  14.                 fs.trash.checkpoint.interval
  15.                 0
  16.         
  17.         
  18.                
  19.                 fs.trash.interval
  20.                 1440
  21.         

复制代码

(3)修改$HADOOP_HOME/etc/hdfs-site.xml




  1.         
  2.         
  3.                 dfs.webhdfs.enabled
  4.                 true
  5.         
  6.         
  7.                 dfs.namenode.name.dir
  8.                 /home/zero/hadoop/hadoop-2.3.0-cdh5.0.1/data/dfs/name
  9.                 namenode 存放name table(fsimage)本地目录(需要修改)
  10.         
  11.         
  12.                 dfs.namenode.edits.dir
  13.                 ${dfs.namenode.name.dir}
  14.                 namenode粗放 transaction file(edits)本地目录(需要修改)
  15.         
  16.         
  17.                 dfs.datanode.data.dir
  18.                 /home/zero/hadoop/hadoop-2.3.0-cdh5.0.1/data/dfs/data
  19.                 datanode存放block本地目录(需要修改)
  20.         
  21.         
  22.                 dfs.replication
  23.                 1
  24.         
  25.         
  26.         
  27.                 dfs.blocksize
  28.                 268435456
  29.         
  30.         
  31.         
  32.         
  33.         
  34.                 dfs.nameservices
  35.                 mycluster
  36.         
  37.         
  38.                
  39.                 dfs.ha.namenodes.mycluster
  40.                 nn1,nn2
  41.         
  42.         
  43.         
  44.                 dfs.namenode.rpc-address.mycluster.nn1
  45.                 Hadoop-NN-01:8020
  46.         
  47.         
  48.                 dfs.namenode.rpc-address.mycluster.nn2
  49.                 Hadoop-NN-02:8020
  50.         
  51.         
  52.         
  53.                 dfs.namenode.http-address.mycluster.nn1
  54.                 Hadoop-NN-01:50070
  55.         
  56.         
  57.                 dfs.namenode.http-address.mycluster.nn2
  58.                 Hadoop-NN-02:50070
  59.         

  60.         
  61.         
  62.         
  63.                 dfs.journalnode.http-address
  64.                 0.0.0.0:8480
  65.         
  66.         
  67.                 dfs.journalnode.rpc-address
  68.                 0.0.0.0:8485
  69.         
  70.         
  71.                
  72.                 <!--格式:qjournal://;;/
  73.                 dfs.namenode.shared.edits.dir
  74.                 qjournal://Hadoop-DN-01:8485;Hadoop-DN-02:8485;Hadoop-DN-03:8485/mycluster
  75.         
  76.         
  77.                
  78.                 dfs.journalnode.edits.dir
  79.                 /home/zero/hadoop/hadoop-2.3.0-cdh5.0.1/data/dfs/jn
  80.         
  81.         
  82.         
  83.                
  84.                 dfs.client.failover.proxy.provider.mycluster
  85.                 org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
  86.         
  87.         
  88.         
  89.         
  90.                 dfs.ha.fencing.methods
  91.                 sshfence
  92.         
  93.         
  94.                 dfs.ha.fencing.ssh.private-key-files
  95.                 /home/zero/.ssh/id_rsa
  96.         
  97.         
  98.                
  99.                 dfs.ha.fencing.ssh.connect-timeout
  100.                 30000
  101.         

  102.         
  103.         
  104.         
  105.                 dfs.ha.automatic-failover.enabled
  106.                 true
  107.         
  108.         
  109.                 ha.zookeeper.quorum
  110.                 Zookeeper-01:2181,Zookeeper-02:2181,Zookeeper-03:2181
  111.         
  112.         
  113.                
  114.                 ha.zookeeper.session-timeout.ms
  115.                 2000
  116.         

复制代码

(4)修改$HADOOP_HOME/etc/yarn-env.sh
  1. #Yarn Daemon Options
  2. #export YARN_RESOURCEMANAGER_OPTS
  3. #export YARN_NODEMANAGER_OPTS
  4. #export YARN_PROXYSERVER_OPTS
  5. #export HADOOP_JOB_HISTORYSERVER_OPTS

  6. #Yarn Logs
  7. export YARN_LOG_DIR=” /home/zero/hadoop/hadoop-2.3.0-cdh5.0.1/logs”
复制代码

(5)$HADOOP_HOEM/etc/mapred-site.xml

  1.         
  2.         
  3.                 mapreduce.framework.name
  4.                 yarn
  5.         
  6.         
  7.         
  8.         
  9.                 mapreduce.jobhistory.address
  10.                 0.0.0.0:10020
  11.         
  12.         
  13.         
  14.                 mapreduce.jobhistory.webapp.address
  15.                 0.0.0.0:19888
  16.         

复制代码

(6)修改$HADOOP_HOME/etc/yarn-site.xml


  1.         
  2.         
  3.                 yarn.nodemanager.aux-services
  4.                 mapreduce_shuffle
  5.         
  6.         
  7.                 yarn.nodemanager.aux-services.mapreduce.shuffle.class
  8.                 org.apache.hadoop.mapred.ShuffleHandler
  9.         
  10.         
  11.                 Address where the localizer IPC is.
  12.                 yarn.nodemanager.localizer.address
  13.                 0.0.0.0:23344
  14.         
  15.         
  16.                 NM Webapp address.
  17.                 yarn.nodemanager.webapp.address
  18.                 0.0.0.0:23999
  19.         

  20.         
  21.         
  22.         
  23.                 yarn.resourcemanager.connect.retry-interval.ms
  24.                 2000
  25.         
  26.         
  27.                 yarn.resourcemanager.ha.enabled
  28.                 true
  29.         
  30.         
  31.                 yarn.resourcemanager.ha.automatic-failover.enabled
  32.                 true
  33.         
  34.         
  35.         
  36.                 yarn.resourcemanager.ha.automatic-failover.embedded
  37.                 true
  38.         
  39.         
  40.         
  41.                 yarn.resourcemanager.cluster-id
  42.                 yarn-cluster
  43.         
  44.         
  45.                 yarn.resourcemanager.ha.rm-ids
  46.                 rm1,rm2
  47.         
  48.     <!--—这里RM主备结点需要单独指定,(可选)
  49.         
  50.         yarn.resourcemanager.ha.id
  51.         rm2

  52. -->
  53.         
  54.                 yarn.resourcemanager.scheduler.class
  55.                 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
  56.         
  57.         
  58.                 yarn.resourcemanager.recovery.enabled
  59.                 true
  60.         
  61.         
  62.                 yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms
  63.                 5000
  64.         
  65.         
  66.         
  67.                 yarn.resourcemanager.store.class
  68.                 org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore
  69.         
  70.         
  71.                 yarn.resourcemanager.zk-address
  72.                 Zookeeper-01:2181,Zookeeper-02:2181,Zookeeper-03:2181
  73.         
  74.         
  75.                 yarn.resourcemanager.zk.state-store.address
  76.                 Zookeeper-01:2181,Zookeeper-02:2181,Zookeeper-03:2181
  77.         
  78.         
  79.         
  80.                 yarn.resourcemanager.address.rm1
  81.                 Hadoop-NN-01:23140
  82.         
  83.         
  84.                 yarn.resourcemanager.address.rm2
  85.                 Hadoop-NN-02:23140
  86.         
  87.         
  88.         
  89.                 yarn.resourcemanager.scheduler.address.rm1
  90.                 Hadoop-NN-01:23130
  91.         
  92.         
  93.                 yarn.resourcemanager.scheduler.address.rm2
  94.                 Hadoop-NN-02:23130
  95.         
  96.         
  97.         
  98.                 yarn.resourcemanager.admin.address.rm1
  99.                 Hadoop-NN-01:23141
  100.         
  101.         
  102.                 yarn.resourcemanager.admin.address.rm2
  103.                 Hadoop-NN-02:23141
  104.         
  105.         
  106.         
  107.                 yarn.resourcemanager.resource-tracker.address.rm1
  108.                 Hadoop-NN-01:23125
  109.         
  110.         
  111.                 yarn.resourcemanager.resource-tracker.address.rm2
  112.                 Hadoop-NN-02:23125
  113.         
  114.         
  115.         
  116.                 yarn.resourcemanager.webapp.address.rm1
  117.                 Hadoop-NN-01:23188
  118.         
  119.         
  120.                 yarn.resourcemanager.webapp.address.rm2
  121.                 Hadoop-NN-02:23188
  122.         
  123.         
  124.                 yarn.resourcemanager.webapp.https.address.rm1
  125.                 Hadoop-NN-01:23189
  126.         
  127.         
  128.                 yarn.resourcemanager.webapp.https.address.rm2
  129.                 Hadoop-NN-02:23189
  130.         

复制代码

(7)修改slaves


  1. [zero@CentOS-Cluster-01  hadoop]$ vi slaves
  2. Hadoop-DN-01
  3. Hadoop-DN-02
  4. Hadoop-DN-03
复制代码

3)分发程序
  1. [zero@CentOS-Cluster-01 ~]$ scp -r /home/zero/hadoop/hadoop-2.3.0-cdh5.0.1 zero@Hadoop-NN-02: /home/zero/hadoop/
  2. …….
  3. [zero@CentOS-Cluster-01 ~]$ scp -r /home/zero/hadoop/hadoop-2.3.0-cdh5.0.1 zero@Hadoop-DN-01: /home/zero/hadoop/
  4. …….
  5. [zero@CentOS-Cluster-01 ~]$ scp -r /home/zero/hadoop/hadoop-2.3.0-cdh5.0.1 zero@Hadoop-DN-02: /home/zero/hadoop/
  6. …….
  7. [zero@CentOS-Cluster-01 ~]$ scp -r /home/zero/hadoop/hadoop-2.3.0-cdh5.0.1 zero@Hadoop-DN-03: /home/zero/hadoop/
复制代码


4)启动HDFS

(1)启动JournalNode
格式化前需要在JournalNode结点上启动JournalNode:
  1. [zero@CentOS-Cluster-03 hadoop-2.3.0-cdh5.0.1]$ hadoop-daemon.sh start journalnode
  2. starting journalnode, logging to /home/puppet/hadoop/cdh4.4/hadoop-2.0.0-cdh4.4.0/logs/hadoop-puppet-journalnode-BigData-03.out
复制代码

验证JournalNode:
  1. [zero@CentOS-Cluster-03 hadoop-2.3.0-cdh5.0.1]$ jps
  2. 25918 QuorumPeerMain
  3. 16728 JournalNode
  4. 16772 Jps
复制代码


(2)NameNode 格式化:
结点Hadoop-NN-01:hdfs namenode -format
  1. [zero@CentOS-Cluster-01 hadoop-2.3.0-cdh5.0.1]$ hdfs namenode -format
  2. 14/06/23 21:02:49 INFO namenode.NameNode: STARTUP_MSG:
  3. /************************************************************
  4. STARTUP_MSG: Starting NameNode
  5. STARTUP_MSG:   host = CentOS-Cluster-01/8.8.8.11
  6. STARTUP_MSG:   args = [-format]
  7. STARTUP_MSG:   version = 2.3.0-cdh5.0.1
  8. STARTUP_MSG:   classpath =…
  9. STARTUP_MSG:   build =…
  10. STARTUP_MSG:   java = 1.7.0_60
  11. ************************************************************/
  12. (…略…)
  13. 14/06/23 21:02:57 INFO common.Storage: Storage directory /home/zero/hadoop/hadoop-2.3.0-cdh5.0.1/data/dfs/name has been successfully formatted. (…略…)
  14. 14/06/23 21:02:59 INFO namenode.NameNode: SHUTDOWN_MSG:
  15. /************************************************************
  16. SHUTDOWN_MSG: Shutting down NameNode at CentOS-Cluster-01/8.8.8.11
  17. ************************************************************/
复制代码


(3)同步NameNode元数据:
同步Hadoop-NN-01元数据到Hadoop-NN-02
主要是:dfs.namenode.name.dir,dfs.namenode.edits.dir还应该确保共享存储目录下(dfs.namenode.shared.edits.dir ) 包含NameNode 所有的元数据。

  1. [puppet@BigData-01 hadoop-2.0.0-cdh4.4.0]$ scp -r data/ zero@Hadoop-NN-02:/home/zero/hadoop/hadoop-2.3.0-cdh5.0.1
  2. seen_txid                           100%    2     0.0KB/s   00:00   
  3. VERSION                            100%  201     0.2KB/s   00:00   
  4. seen_txid                           100%    2     0.0KB/s   00:00   
  5. fsimage_0000000000000000000.md5    100%   62     0.1KB/s   00:00   
  6. fsimage_0000000000000000000         100%  121     0.1KB/s   00:00   
  7. VERSION                            100%  201     0.2KB/s   00:00
复制代码


(4)初始化ZFCK:
创建ZNode,记录状态信息。
结点Hadoop-NN-01:hdfs zkfc -formatZK
  1. [zero@CentOS-Cluster-01 hadoop-2.3.0-cdh5.0.1]$ hdfs zkfc -formatZK
  2. 14/06/23 23:22:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  3. 14/06/23 23:22:28 INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at Hadoop-NN-01/8.8.8.11:8020
  4. 14/06/23 23:22:28 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-cdh5.0.1--1, built on 05/06/2014 18:50 GMT
  5. 14/06/23 23:22:28 INFO zookeeper.ZooKeeper: Client environment:host.name=CentOS-Cluster-01
  6. 14/06/23 23:22:28 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_60
  7. 14/06/23 23:22:28 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
  8. 14/06/23 23:22:28 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/runtime/java/jdk1.7.0_60/jre
  9. 14/06/23 23:22:28 INFO zookeeper.ZooKeeper: Client environment:java.class.path=...
  10. 14/06/23 23:22:29 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/home/zero/hadoop/hadoop-2.3.0-cdh5.0.1/lib/native
  11. 14/06/23 23:22:29 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
  12. 14/06/23 23:22:29 INFO zookeeper.ZooKeeper: Client environment:java.compiler=
  13. 14/06/23 23:22:29 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
  14. 14/06/23 23:22:29 INFO zookeeper.ZooKeeper: Client environment:os.arch=i386
  15. 14/06/23 23:22:29 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-431.el6.i686
  16. 14/06/23 23:22:29 INFO zookeeper.ZooKeeper: Client environment:user.name=zero
  17. 14/06/23 23:22:29 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/zero
  18. 14/06/23 23:22:29 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/zero/hadoop/hadoop-2.3.0-cdh5.0.1
  19. 14/06/23 23:22:29 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=Zookeeper-01:2181,Zookeeper-02:2181,Zookeeper-03:2181 sessionTimeout=2000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@150c2b0
  20. 14/06/23 23:22:29 INFO zookeeper.ClientCnxn: Opening socket connection to server CentOS-Cluster-03/8.8.8.13:2181. Will not attempt to authenticate using SASL (unknown error)
  21. 14/06/23 23:22:29 INFO zookeeper.ClientCnxn: Socket connection established to CentOS-Cluster-03/8.8.8.13:2181, initiating session
  22. 14/06/23 23:22:29 INFO zookeeper.ClientCnxn: Session establishment complete on server CentOS-Cluster-03/8.8.8.13:2181, sessionid = 0x146cc0517b30002, negotiated timeout = 4000
  23. 14/06/23 23:22:29 INFO ha.ActiveStandbyElector: Session connected.
  24. 14/06/23 23:22:30 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/mycluster in ZK.
  25. 14/06/23 23:22:30 INFO zookeeper.ZooKeeper: Session: 0x146cc0517b30002 closed
复制代码



(5)启动
集群启动法:Hadoop-NN-01:
  1. [zero@CentOS-Cluster-01 hadoop-2.3.0-cdh5.0.1]$ start-dfs.sh
  2. 14/04/23 01:54:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  3. Starting namenodes on [Hadoop-NN-01 Hadoop-NN-02]
  4. Hadoop-NN-01: starting namenode, logging to /home/puppet/hadoop/cdh4.4/hadoop-2.0.0-cdh4.4.0/logs/hadoop-puppet-namenode-BigData-01.out
  5. Hadoop-NN-02: starting namenode, logging to /home/puppet/hadoop/cdh4.4/hadoop-2.0.0-cdh4.4.0/logs/hadoop-puppet-namenode-BigData-02.out
  6. Hadoop-DN-01: starting datanode, logging to /home/puppet/hadoop/cdh4.4/hadoop-2.0.0-cdh4.4.0/logs/hadoop-puppet-datanode-BigData-03.out
  7. Hadoop-DN-02: starting datanode, logging to /home/puppet/hadoop/cdh4.4/hadoop-2.0.0-cdh4.4.0/logs/hadoop-puppet-datanode-BigData-04.out
  8. Hadoop-DN-03: starting datanode, logging to /home/puppet/hadoop/cdh4.4/hadoop-2.0.0-cdh4.4.0/logs/hadoop-puppet-datanode-BigData-05.out
  9. Starting journal nodes [Hadoop-DN-01 Hadoop-DN-02 Hadoop-DN-03]
  10. Hadoop-DN-01: starting journalnode, logging to /home/puppet/hadoop/cdh4.4/hadoop-2.0.0-cdh4.4.0/logs/hadoop-puppet-journalnode-BigData-03.out
  11. Hadoop-DN-03: starting journalnode, logging to /home/puppet/hadoop/cdh4.4/hadoop-2.0.0-cdh4.4.0/logs/hadoop-puppet-journalnode-BigData-05.out
  12. Hadoop-DN-02: starting journalnode, logging to /home/puppet/hadoop/cdh4.4/hadoop-2.0.0-cdh4.4.0/logs/hadoop-puppet-journalnode-BigData-04.out
  13. 14/04/23 01:55:29 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  14. Starting ZK Failover Controllers on NN hosts [Hadoop-NN-01 Hadoop-NN-02]
  15. Hadoop-NN-01: starting zkfc, logging to /home/puppet/hadoop/cdh4.4/hadoop-2.0.0-cdh4.4.0/logs/hadoop-puppet-zkfc-BigData-01.out
  16. Hadoop-NN-02: starting zkfc, logging to /home/puppet/hadoop/cdh4.4/hadoop-2.0.0-cdh4.4.0/logs/hadoop-puppet-zkfc-BigData-02.out
复制代码



单进程启动法:
<1>NameNode(Hadoop-NN-01,Hadoop-NN-02):

  1. [zero@CentOS-Cluster-01 hadoop-2.3.0-cdh5.0.1]$ hadoop-daemon.sh start namenode
复制代码





<2>DataNode(Hadoop-DN-01,Hadoop-DN-02,Hadoop-DN-03):

  1. [zero@CentOS-Cluster-01 hadoop-2.3.0-cdh5.0.1]$ hadoop-daemon.sh start datanode
复制代码




<3>JournalNode(Hadoop-DN-01,Hadoop-DN-02,Hadoop-DN-03):
  1. [zero@CentOS-Cluster-01 hadoop-2.3.0-cdh5.0.1]$ hadoop-daemon.sh start journalnode
复制代码





<4>ZKFC(Hadoop-NN-01,Hadoop-NN-02):
  1. [zero@CentOS-Cluster-01 hadoop-2.3.0-cdh5.0.1]$ hadoop-daemon.sh start zkfc
复制代码




(6)验证

<1>进程

NameNode:
  1. [zero@CentOS-Cluster-01 hadoop-2.3.0-cdh5.0.1]$ jps
  2. 4001 NameNode
  3. 4290 DFSZKFailoverController
  4. 4415 Jps
复制代码





DataNode:
  1. [zero@CentOS-Cluster-03 hadoop-2.3.0-cdh5.0.1]$ jps
  2. 25918 QuorumPeerMain
  3. 19217 JournalNode
  4. 19143 DataNode
  5. 19351 Jps
复制代码





<2>页面:
Active结点:http://8.8.8.10:50070

 


JSP:
 


StandBy结点:
 


JSP:

 


(7)停止:stop-dfs.sh



5)启动Yarn

(1)启动

<1>集群启动
Hadoop-NN-01启动Yarn,命令所在目录:$HADOOP_HOME/sbin
  1. [zero@CentOS-Cluster-01 hadoop]$ start-yarn.sh
  2. starting yarn daemons
  3. starting resourcemanager, logging to /home/zero/hadoop/hadoop-2.3.0-cdh5.0.1/logs/yarn-zero-resourcemanager-CentOS-Cluster-01.out
  4. Hadoop-DN-02: starting nodemanager, logging to /home/zero/hadoop/hadoop-2.3.0-cdh5.0.1/logs/yarn-zero-nodemanager-CentOS-Cluster-04.out
  5. Hadoop-DN-03: starting nodemanager, logging to /home/zero/hadoop/hadoop-2.3.0-cdh5.0.1/logs/yarn-zero-nodemanager-CentOS-Cluster-05.out
  6. Hadoop-DN-01: starting nodemanager, logging to /home/zero/hadoop/hadoop-2.3.0-cdh5.0.1/logs/yarn-zero-nodemanager-CentOS-Cluster-03.out
复制代码





Hadoop-NN-02备机启动RM:
  1. [zero@CentOS-Cluster-02 hadoop]$ yarn-daemon.sh start resourcemanager
  2. starting resourcemanager, logging to /home/zero/hadoop/hadoop-2.3.0-cdh5.0.1/logs/yarn-zero-resourcemanager-CentOS-Cluster-02.o
复制代码





<2>单进程启动
ResourceManager(Hadoop-NN-01,Hadoop-NN-02):
  1. [zero@CentOS-Cluster-01 hadoop-2.3.0-cdh5.0.1]$ yarn-daemon.sh start resourcemanager
复制代码

DataNode(Hadoop-DN-01,Hadoop-DN-02,Hadoop-DN-03):
  1. [zero@CentOS-Cluster-03 hadoop-2.3.0-cdh5.0.1]$ yarn-daemon.sh start nodemanager
复制代码

(2)验证


<1>进程:
JobTracker:Hadoop-NN-01,Hadoop-NN-02
  1. [zero@CentOS-Cluster-01 hadoop]$ jps
  2. 7454 NameNode
  3. 14684 Jps
  4. 14617 ResourceManager
  5. 7729 DFSZKFailoverController
复制代码





TaskTracker:Hadoop-DN-01,Hadoop-DN-02,Hadoop-DN-03
  1. [zero@CentOS-Cluster-03 ~]$ jps
  2. 8965 Jps
  3. 5228 JournalNode
  4. 5137 DataNode
  5. 2514 QuorumPeerMain
  6. 8935 NodeManager
复制代码





<2>页面
ResourceManger(Active):8.8.8.11:23188

 

ResourceManager(Standby):8.8.8.12:23188 

 

 

(3)停止
Hadoop-NN-01:stop-yarn.sh
Hadoop-NN-02:yarn-daeman.sh stop resourcemanager


转载:http://shihlei.iteye.com/

来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/30089851/viewspace-1987620/,如需转载,请注明出处,否则将追究法律责任。

上一篇: 没有了~
请登录后发表评论 登录
全部评论
主要从事大数据平台产品的技术工作; 负责基于Spark、Flink生产开发及性能调优; 负责对公司的集群的部署、监控、调优和运营; 主导我司的数据平台0到1及实时中间件项目开发; 此外,精通Java、Shell、Python编程; 且荣获2016年IBM的Spark大赛全球第四名,中国区域第一名。

注册时间:2015-01-03

  • 博文量
    187
  • 访问量
    319033