ITPub博客

首页 > 大数据 > Hadoop > sqoop条件抽取报错distcp

sqoop条件抽取报错distcp

原创 Hadoop 作者:longer3281 时间:2018-07-06 16:28:20 0 删除 编辑
Sqoop抽取语句如下:
/usr/local/sqoop-1.4.6/bin/sqoop import \
  --connect jdbc:mysql://10.13.50.125:3306/longer \
  --username sqoop \
  --password XXXXXX \
  --query " select * from tabname where scheduled_deptime between UNIX_TIMESTAMP('$v_begin_date') and UNIX_TIMESTAMP('$v_end_date') and \$CONDITIONS " \
  --hive-database testdb\
  --hive-table tabname _test \
  --hive-import \
  --split-by dynamic_id \
  --target-dir /user/hive/sqptarget \
  --bindir /home/hadoop/new_binddir \
  --outdir /home/hadoop/new_binddir \
  --direct \
  -m 8
--------------------------------------------------------------------------------
报错如下(片段):
18/07/06 15:25:41 INFO common.FileUtils: Source is 76566465 bytes. (MAX: 33554432)
18/07/06 15:25:41 INFO common.FileUtils: Launch distributed copy (distcp) job.
18/07/06 15:25:41 ERROR metadata.Hive: Failed to move: java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCpOptions
Failed with exception java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCpOptions
18/07/06 15:25:41 ERROR exec.Task: Failed with exception java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCpOptions
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCpOptions
        at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:2895)
        at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:3205)
        at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1920)
        at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:364)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2084)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1755)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1463)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1181)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
        at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:474)
        at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:490)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
--------------------------------------------------------------------------
原因: hive 最后移动数据的时候,需要调用hadoop-distcp-X.X.X.jar,
解决方法:只需要把$HADOOP_HOME/share/hadoop/tools/lib/hadoop-distcp-x.x.x.jar 拷贝 $HIVE_HOME/lib下面,重启hive即可


来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/9606353/viewspace-2157457/,如需转载,请注明出处,否则将追究法律责任。

请登录后发表评论 登录
全部评论

注册时间:2009-05-22

  • 博文量
    51
  • 访问量
    147286