ITPub博客

首页 > 大数据 > 数据分析 > Sqoop导入数据异常处理

Sqoop导入数据异常处理

原创 数据分析 作者:z597011036 时间:2019-01-30 13:37:18 0 删除 编辑

1.错误提示(没有json.jar包)

19/01/30 11:59:48 INFO manager.DirectMySQLManager: Beginning mysqldump fast path import

19/01/30 11:59:48 INFO mapreduce.ImportJobBase: Beginning import of t3

Exception in thread "main" java.lang.NoClassDefFoundError: org/json/JSONObject

at org.apache.sqoop.util.SqoopJsonUtil.getJsonStringforMap(SqoopJsonUtil.java:43)

at org.apache.sqoop.SqoopOptions.writeProperties(SqoopOptions.java:780)

at org.apache.sqoop.mapreduce.JobBase.putSqoopOptionsToConfiguration(JobBase.java:392)

at org.apache.sqoop.mapreduce.JobBase.createJob(JobBase.java:378)

at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:256)

at org.apache.sqoop.manager.DirectMySQLManager.importTable(DirectMySQLManager.java:92)

at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:513)

at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621)

at org.apache.sqoop.Sqoop.run(Sqoop.java:147)


解决方法:

参考文档:https://github.com/stleary/JSON-java

下载json包: https://search.maven.org/search?q=g:org.json%20AND%20a:json&core=gav

将下载的json.tar包上传到/opt/cloudera/parcels/CDH/lib/sqoop/lib中。


2.其它CDH节点没有mysqldump命令( 因为导入时加了--direct参数

19/01/30 13:59:29 INFO mapreduce.Job:  map 0% reduce 0%

19/01/30 13:59:33 INFO mapreduce.Job: Task Id : attempt_1545874390047_0006_m_000000_0, Status : FAILED

Error: java.io.IOException: Cannot run program "mysqldump": error=2, 没有那个文件或目录

at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)

at java.lang.Runtime.exec(Runtime.java:620)

at java.lang.Runtime.exec(Runtime.java:485)

at org.apache.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:405)

at org.apache.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:49)

at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)


解决方法:

[root@jingong01 ~]# scp /usr/bin/mysqldump root@192.168.7.32:/usr/bin/

mysqldump                                        100% 3102KB   3.0MB/s   00:00    

[root@jingong01 ~]#


3.Sqoop缺少hive包

19/01/30 14:15:50 WARN hive.TableDefWriter: Column CREATETIME had to be cast to a less precise type in Hive

19/01/30 14:15:50 WARN hive.TableDefWriter: Column UPDATETIME had to be cast to a less precise type in Hive

19/01/30 14:15:50 INFO hive.HiveImport: Loading uploaded data into Hive

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/shims/ShimLoader

at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:370)

at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:108)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:264)

at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:44)

at org.apache.sqoop.hive.HiveImport.getHiveArgs(HiveImport.java:392)

at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:379)

at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337)

at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)

at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:530)

at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621)

at org.apache.sqoop.Sqoop.run(Sqoop.java:147)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)

at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)


解决方法:

[root@jingong01 ~]# cp -a  /opt/cloudera/parcels/CDH/lib/hive/lib/hive-shims* /opt/cloudera/parcels/CDH/lib/sqoop/lib/              --将hive下面的hive-shims包全部考到sqoop下面。


4.在hive中已创建表,在导入时加了创建表参数

19/01/30 14:30:20 INFO hive.HiveImport: WARN: The method class org.apache.commons.logging.impl.SLF4JLogFactory#release() was invoked.

19/01/30 14:30:20 INFO hive.HiveImport: WARN: Please see http://www.slf4j.org/codes.html#release for an explanation.

19/01/30 14:30:21 ERROR tool.ImportTool: Import failed: java.io.IOException: Hive exited with status 64

at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:384)

at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337)

at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)

at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:530)

at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621)

at org.apache.sqoop.Sqoop.run(Sqoop.java:147)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)


解决方法:

[hdfs@jingong01 ~]$ sqoop import --connect jdbc:mysql://172.16.8.93:3306/db_stktag --username wangying --password wangying --table t3 --target-dir /user/tong/123 --hive-import --create-hive-table --num-mappers 1 --hive-table TT3 -m 1 --split-by date --direct        --去掉红色参数


来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/25854343/viewspace-2565234/,如需转载,请注明出处,否则将追究法律责任。

请登录后发表评论 登录
全部评论

注册时间:2014-03-31

  • 博文量
    31
  • 访问量
    269701