HDFS Federation应用场景案例

文档历史

日期版本作者描述说明

文档最后一次修改时间2018-10-18 16:40:00

1 方案背景 1

2 文档目的 2

3 技术架构 3

4 HDFS Federaion集群说明 4

5 关键配置 5

6 客户端Viewfs路由配置 9

6.1 确定要迁移至新增namespace的db、table、partition，及对应的HDFS路径。 9

6.2 配置定义core-site.xml或者mountTable.xml路由表信息，生产示例，如下： 9

6.3 同步配置 9

6.4 重启hivemetastore及hiveserver2（是否需要hiveserver2访问冷数据？） 9

7 业务操作 10

7.1 Hadoop客户端操作 10

7.1.1 创建业务DB在HDFS上的存储路径，如： 10

7.1.2 查看不同的NameSpace上的HDFS目录： 10

7.1.3 拷贝数据： 10

7.2 Hive客户端操作 13

7.2.1 在hive中创建db: 13

7.2.2 在hive db中创建table: 13

7.2.3 修改hive table的location： 14

7.2.4 查看分区： 14

7.2.5 修复分区： 17

7.2.6 不同的NameSpace Location关联聚合查询： 17

7.3 冷数据存储迁移操作步骤 18

7.3.1 部署hdfs federation客户端 18

7.3.2 制定并收集冷数据定义 18

7.3.3 配置冷数据迁移调度任务，执行数据迁移 18

7.3.4 验证迁移的数据完整性 18

7.3.5 变更迁移的Hive Table或者Table Partition的location 18

7.3.6 验证查询 18

7.3.7 清理原NameSpace已经迁移的冷数据，释放空间 18

8 异常报错QA： 19

9 备注（警告） 20

1 方案背景

ν 生产集群中沉淀了大量的冷数据集及小文件

ν 基于Hadoop 3.x的纠删码技术，相比Hadoop2.x可以节约50%的存储空间

ν 业务在需要访问冷数据的时候，不需要再做额外的工作

2 文档目的

拟定方案，梳理相关细节，指导平台及相关数据开发人员在HDFS Federation环境下能够正常开展工作。

3 技术架构

ν wh-ns为hadoop 2.x集群，hdpclusterns1为hadoop 3.x集群

ν 联盟策略：客户端配置路由打通（datanode存储不共享）

5 关键配置

1. core-site.xml关键配置：

hadoop.proxyuser.hive.hosts=*

hadoop.proxyuser.hive.groups=*

hadoop.proxyuser.spark.hosts=*

hadoop.proxyuser.spark.groups=*

hadoop.proxyuser.yarn.hosts=*

hadoop.proxyuser.yarn.groups=*

hadoop.proxyuser.mapred.groups=*

hadoop.proxyuser.mapred.hosts=*

hadoop.proxyuser.HTTP.groups=*

hadoop.proxyuser.HTTP.hosts=*

hadoop.proxyuser.hdfs.groups=*

hadoop.proxyuser.hdfs.hosts=*

hadoop.proxyuser.hadoop.groups=*

hadoop.proxyuser.hadoop.hosts=*

2. hdfs-site.xml关键配置：

dfs.nameservices=wh-ns,hdpclusterns1

dfs.namenode.kerberos.principal.pattern=*

dfs.ha.automatic-failover.enabled.wh-ns=true

dfs.ha.namenodes.wh-ns=namenode50,namenode35

dfs.namenode.rpc-address.wh-ns.namenode50=xx1:8020

dfs.namenode.servicerpc-address.wh-ns.namenode50=xx1:8022

dfs.namenode.http-address.wh-ns.namenode50=xx1:50070

dfs.namenode.https-address.wh-ns.namenode50=xx1:50470

dfs.namenode.rpc-address.wh-ns.namenode35=xx2:8020

dfs.namenode.servicerpc-address.wh-ns.namenode35=xx2:8022

dfs.namenode.http-address.wh-ns.namenode35=xx2:50070

dfs.namenode.https-address.wh-ns.namenode35=xx2:50470

dfs.client.failover.proxy.provider.wh-ns=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

dfs.ha.namenodes.hdpclusterns1=nn1,nn2

dfs.client.failover.proxy.provider.hdpclusterns1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

dfs.ha.automatic-failover.enabled.hdpclusterns1=true

dfs.namenode.rpc-address.hdpclusterns1.nn1=yy1:8020

dfs.namenode.rpc-address.hdpclusterns1.nn2=yy2:8020

dfs.namenode.servicerpc-address.hdpclusterns1.nn1=yy1:8022

dfs.namenode.servicerpc-address.hdpclusterns1.nn2=yy2:8022

dfs.namenode.http-address.hdpclusterns1.nn1=yy1:50070

dfs.namenode.http-address.hdpclusterns1.nn2=yy2:50070

dfs.namenode.https-address.hdpclusterns1.nn1=yy1:50470

dfs.namenode.https-address.hdpclusterns1.nn2=yy2:50470

3. mapred-site.xml关键配置：

mapreduce.job.hdfs-servers.token-renewal.exclude=xx1,xx2,yy1,yy2

4. spark-defaults.conf关键配置：

ν spark.hadoop.fs.defaultFS hdfs://wh-ns ---spark使用

ν spark.yarn.access.hadoopFileSystems=hdfs://wh-ns/,hdfs://hdpclusterns1/

---spark使用

fs.defaultFS

hdfs://wh-ns

</property>

6 客户端Viewfs路由配置

6.1 确定要迁移至新增namespace的db、table、partition，及对应的HDFS路径。

6.2 配置定义core-site.xml或者mountTable.xml路由表信息，生产示例，如下：

fs.viewfs.mounttable.clusterfed.homedir

/user

</property>

fs.viewfs.mounttable.clusterfed.link./apps/hive/warehouse/test.db

hdfs://hdpclusterns1/apps/hive/warehouse/test.db</value>

</property>

fs.viewfs.mounttable.clusterfed.link./apps/hive/warehouse/test2.db

hdfs://hdpclusterns1/apps/hive/warehouse/test2.db</value>

</property>

6.3 同步配置

../etc/hadoop/core-site.xml

../etc/hadoop/hdfs-site.xml

6.4 重启hivemetastore及hiveserver2（是否需要hiveserver2访问冷数据？）

7 业务操作

7.1 Hadoop客户端操作

7.1.1 创建业务DB在HDFS上的存储路径，如：

[quyuehui@test2~]$ hadoop fs -mkdir hdfs://hdpclusterns1/apps/hive/warehouse/test.db

7.1.2 查看不同的NameSpace上的HDFS目录：

hadoop命令行操作示例如下：

hadoop fs -ls hdfs://wh-ns/tmp

输出结果，等同于：

hadoop fs -ls /tmp

原因：../etc/hadoop/core-site.xml中定义了：fs.defaultFS=hdfs://wh-ns

访问冷存储集群:

hadoop fs -ls hdfs://hdpclusterns1/tmp

原因：../etc/hadoop/hdfs-site.xml中定义了：hdpclusterns1这个namespace的名字空间地址解析

hadoop fs –ls viewfs://clusterfed/tmp 默认不支持，需配置core-site.xml或者mountTable.xml路由表映射(且配置后只能访问在core-site.xml或者mountTable.xml中定义的路径)

7.1.3 拷贝数据：

[quyuehui@test1~]$ hadoop distcp -update -skipcrccheck hdfs://wh-ns/data/hive/warehouse/test.db/tbldate hdfs://yy1:8020/apps/hive/warehouse/test.db/tbldate

18/10/15 16:40:50 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=true, deleteMissing=false, ignoreFailures=false, overwrite=false, append=false, useDiff=false, useRdiff=false, fromSnapshot=null, toSnapshot=null, skipCRC=true, blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=100, sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[], preserveRawXattrs=false, atomicWorkPath=null, logPath=null, sourceFileListing=null, sourcePaths=[hdfs://wh-ns/data/hive/warehouse/test.db/tbldate], targetPath=hdfs://yy1:8020/apps/hive/warehouse/test.db/tbldate, targetPathExists=true, filtersFile='null'}

18/10/15 16:40:50 INFO hdfs.DFSClient: Created token for test: HDFS_DELEGATION_TOKEN owner=test/user@KERBEROS.COM, renewer=yarn, realUser=, issueDate=1539592850639, maxDate=1540197650639, sequenceNumber=49243643, masterKeyId=1762 on ha-hdfs:wh-ns

18/10/15 16:40:50 INFO security.TokenCache: Got dt for hdfs://wh-ns; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:wh-ns, Ident: (token for test: HDFS_DELEGATION_TOKEN owner=test/user@KERBEROS.COM, renewer=yarn, realUser=, issueDate=1539592850639, maxDate=1540197650639, sequenceNumber=49243643, masterKeyId=1762)

18/10/15 16:40:50 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1; dirCnt = 0

18/10/15 16:40:50 INFO tools.SimpleCopyListing: Build file listing completed.

18/10/15 16:40:50 INFO Configuration.deprecation: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb

18/10/15 16:40:50 INFO Configuration.deprecation: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor

18/10/15 16:40:51 INFO tools.DistCp: Number of paths in the copy list: 1

18/10/15 16:40:51 INFO hdfs.DFSClient: Created token for test: HDFS_DELEGATION_TOKEN owner=test/user@KERBEROS.COM, renewer=, realUser=, issueDate=1539592851244, maxDate=1540197651244, sequenceNumber=352, masterKeyId=38 on 10.65.5.93:8020

18/10/15 16:40:51 INFO security.TokenCache: Got dt for hdfs://yy1:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 10.65.5.93:8020, Ident: (token for test: HDFS_DELEGATION_TOKEN owner=test/user@KERBEROS.COM, renewer=, realUser=, issueDate=1539592851244, maxDate=1540197651244, sequenceNumber=352, masterKeyId=38)

18/10/15 16:40:51 INFO mapreduce.JobSubmitter: number of splits:1

18/10/15 16:40:51 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1539590835150_0505

18/10/15 16:40:51 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: 10.65.5.93:8020, Ident: (token for test: HDFS_DELEGATION_TOKEN owner=test/user@KERBEROS.COM, renewer=, realUser=, issueDate=1539592851244, maxDate=1540197651244, sequenceNumber=352, masterKeyId=38)

18/10/15 16:40:51 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:wh-ns, Ident: (token for test: HDFS_DELEGATION_TOKEN owner=test/user@KERBEROS.COM, renewer=yarn, realUser=, issueDate=1539592850639, maxDate=1540197650639, sequenceNumber=49243643, masterKeyId=1762)

18/10/15 16:40:52 INFO impl.YarnClientImpl: Submitted application application_1539590835150_0505

18/10/15 16:40:52 INFO mapreduce.Job: The url to track the job: http://xx1:8088/proxy/application_1539590835150_0505/

18/10/15 16:40:52 INFO tools.DistCp: DistCp job-id: job_1539590835150_0505

18/10/15 16:40:52 INFO mapreduce.Job: Running job: job_1539590835150_0505

18/10/15 16:41:01 INFO mapreduce.Job: Job job_1539590835150_0505 running in uber mode : false

18/10/15 16:41:01 INFO mapreduce.Job: map 0% reduce 0%

18/10/15 16:41:15 INFO mapreduce.Job: map 100% reduce 0%

18/10/15 16:41:16 INFO mapreduce.Job: Job job_1539590835150_0505 completed successfully

18/10/15 16:41:16 INFO mapreduce.Job: Counters: 32

File System Counters

FILE: Number of bytes read=0

FILE: Number of bytes written=141063

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

HDFS: Number of bytes read=362

HDFS: Number of bytes written=64

HDFS: Number of read operations=11

HDFS: Number of large read operations=0

HDFS: Number of write operations=2

Job Counters

Launched map tasks=1

Other local map tasks=1

Total time spent by all maps in occupied slots (ms)=12582

Total time spent by all reduces in occupied slots (ms)=0

Total time spent by all map tasks (ms)=12582

Total vcore-seconds taken by all map tasks=12582

Total megabyte-seconds taken by all map tasks=12883968

Map-Reduce Framework

Map input records=1

Map output records=1

Input split bytes=115

Spilled Records=0

Failed Shuffles=0

Merged Map outputs=0

GC time elapsed (ms)=118

CPU time spent (ms)=1690

Physical memory (bytes) snapshot=193511424

Virtual memory (bytes) snapshot=3568738304

Total committed heap usage (bytes)=208142336

File Input Format Counters

Bytes Read=247

File Output Format Counters

Bytes Written=64

DistCp Counters

Bytes Skipped=171512

Files Skipped=1

7.2 Hive客户端操作

7.2.1 在hive中创建db:

hive> CREATE DATABASE IF NOT EXISTS test2 localtion 'viewfs://clusterfed/apps/hive/warehouse/test2.db';

或者，修改hive元数据表DBS：

mysql> update DBS set DB_LOCATION_URI=replace(DB_LOCATION_URI,'hdfs://wh-ns/data','viewfs://clusterfed/apps') where NAME=test2;

hive> describe database test2;

OK

test2 viewfs://clusterfed/apps/hive/warehouse/test2.db hive USER

注：

1、该步骤需确定指定的viewfs location在mountTable.xml中已经定义，并能够被正常识别解析。

2、控制只有管理员可以创建hive db，否则报错：

FAILED: SemanticException ****only administrator can CREATE/DROP/ALTER database, current user is test

7.2.2 在hive db中创建table:

##创建非分区表：

hive> create table if not exists test2.tblstockns1 like test.tblstock;

hive> describe formatted test2.tblstockns1;

OK

# col_name data_type comment

ordernumber string

locationid string

dateid string

# Detailed Table Information

Database: test2

Owner: hive

CreateTime: Mon Oct 15 18:03:41 CST 2018

LastAccessTime: UNKNOWN

Protect Mode: None

Retention: 0

Location: viewfs://clusterfed/apps/hive/warehouse/test2.db/tblstockns1

Table Type: MANAGED_TABLE

Table Parameters:

transient_lastDdlTime 1539597821

# Storage Information

SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

InputFormat: org.apache.hadoop.mapred.TextInputFormat

OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat

Compressed: No

Num Buckets: -1

Bucket Columns: []

Sort Columns: []

Storage Desc Params:

field.delim ,

line.delim \n

serialization.format ,

Time taken: 0.075 seconds, Fetched: 30 row(s)

7.2.3 修改hive table的location：

非分区表：

hive> ALTER TABLE tbldate set location "viewfs://clusterfed/apps/hive/warehouse/test.db/tbldate";

分区表：

hive> ALTER TABLE fsimage_analysis PARTITION (ds=20180731) SET LOCATION "viewfs://clusterfed/apps/hive/warehouse/test.db/fsimage_analysis/ds=20180731";

7.2.4 查看分区：

hive> describe formatted test.fsimage_analysis;

OK

# col_name data_type comment

path string

replication int

modificationtime string

accesstime string

preferredblocksize int

blockscount int

filesize bigint

nsquota string

dsquota string

type int

emptydir int

permission string

username string

groupname string

# Partition Information

# col_name data_type comment

ds string

# Detailed Table Information

Database: test

Owner: vip

CreateTime: Thu Aug 24 17:06:23 CST 2017

LastAccessTime: UNKNOWN

Protect Mode: None

Retention: 0

Location: hdfs://wh-ns/data/hive/warehouse/test.db/fsimage_analysis

Table Type: MANAGED_TABLE

Table Parameters:

last_modified_by backup

last_modified_time 1530684459

transient_lastDdlTime 1530684459

# Storage Information

SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

InputFormat: org.apache.hadoop.mapred.TextInputFormat

OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat

Compressed: No

Num Buckets: -1

Bucket Columns: []

Sort Columns: []

Storage Desc Params:

field.delim \t

serialization.format \t

Time taken: 0.076 seconds, Fetched: 47 row(s)

hive> describe formatted test.fsimage_analysis partition(ds=20180731);

OK

# col_name data_type comment

path string

replication int

modificationtime string

accesstime string

preferredblocksize int

blockscount int

filesize bigint

nsquota string

dsquota string

type int

emptydir int

permission string

username string

groupname string

# Partition Information

# col_name data_type comment

ds string

# Detailed Partition Information

Partition Value: [20180731]

Database: test

Table: fsimage_analysis

CreateTime: Tue Jul 31 05:00:34 CST 2018

LastAccessTime: UNKNOWN

Protect Mode: None

Location: viewfs://clusterfed/apps/hive/warehouse/test.db/fsimage_analysis/ds=20180731

Partition Parameters:

COLUMN_STATS_ACCURATE true

last_modified_by test

last_modified_time 1539062569

numFiles 1

numRows 0

rawDataSize 0

totalSize 44451693550

transient_lastDdlTime 1539062569

# Storage Information

SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

InputFormat: org.apache.hadoop.mapred.TextInputFormat

OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat

Compressed: No

Num Buckets: -1

Bucket Columns: []

Sort Columns: []

Storage Desc Params:

field.delim \t

serialization.format \t

Time taken: 0.167 seconds, Fetched: 51 row(s)

7.2.5 修复分区：

hive> MSCK REPAIR TABLE test.tbldate;

7.2.6 不同的NameSpace Location关联聚合查询：

hive> select count(*) from test.tbldate t1 join test.tblstock t2 where t1.dateid=t2.dateid;

hive> select count(*) from test.fsimage_analysis where ds=20180731 or ds=20180801;

注：

1、查询是否跨NameSpace，排查方法，运行describe命令查看Table或者Partition的location。

2、跨NameSpace巨量数据关联查询，受制于网络带宽瓶颈，速度会比较慢（wh-ns在万兆网络区，hdpclusterns1在千兆网络区）

7.3 冷数据存储迁移操作步骤

7.3.1 部署hdfs federation客户端

7.3.2 制定并收集冷数据定义

7.3.3 配置冷数据迁移调度任务，执行数据迁移

7.3.4 验证迁移的数据完整性

7.3.5 变更迁移的Hive Table或者Table Partition的location

7.3.6 验证查询

7.3.7 清理原NameSpace已经迁移的冷数据，释放空间

8 异常报错QA：

[quyuehui@test2 ~]$ hadoop distcp -update -skipcrccheck hdfs://wh-ns/data/hive/warehouse/test.db/tbldate hdfs://hdpclusterns1/apps/hive/warehouse/test.db/tbldate

报错：

Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1539590835150_3108 to YARN : Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:hdpclusterns1

问题原因：

YARN服务端未配置hdpclusterns1的地址解析。

临时解决方案：

[quyuehui@test2 ~]$ hadoop distcp -update -skipcrccheck hdfs://wh-ns/data/hive/warehouse/test.db/tbldate hdfs://yy1:8020/apps/hive/warehouse/test.db/tbldate

彻底解决方案：

在cloudera manager的hdfs-site.xml 的 HDFS 客户端高级配置代码段（安全阀）【注：对于HDFS来说:YARN相关于HDFS的客户端对象】，加入hdpclusterns1相关的地址解析配置，并重启resourcemanager

问题相关url:

https://community.hortonworks.com/content/supportkb/49528/job-submitted-from-external-client-fails-with-erro.html

https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_admin_distcp_data_cluster_migrate.html#concept_hcs_srr_sr

https://community.hortonworks.com/articles/181990/hive-partitions-on-different-namespaces-in-a-feder.html

https://zh.hortonworks.com/blog/increasing-hadoop-storage-scale-4x/

9 备注（警告）

ITPub博客

HDFS Federation应用案例hadoop2 with hadoop3

最新文章