Sqoop2 从 mysql 导入 hdfs 失败，原因是：java.lang.ClassCastException：java.math.BigInteger 无法转换为 java.lang.Long答案

【问题标题】：Sqoop2 failed import from mysql to hdfs with "Caused by: java.lang.ClassCastException: java.math.BigInteger cannot be cast to java.lang.Long"Sqoop2 从 mysql 导入 hdfs 失败，原因是：java.lang.ClassCastException：java.math.BigInteger 无法转换为 java.lang.Long
【发布时间】：2017-10-05 17:28:03
【问题描述】：

我想用Sqoop2 将数据从Mysql 导入到HDFS

我的桌子看起来像

CREATE TABLE `campaign` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(128) NOT NULL,
   PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8

我创建了 2 个链接

sqoop:000> show link
+----+-------+--------------+------------------------+---------+
| Id | Name  | Connector Id |     Connector Name     | Enabled |
+----+-------+--------------+------------------------+---------+
| 2  | hdfs  | 3            | hdfs-connector         | true    |
| 7  | mysql | 1            | generic-jdbc-connector | true    |
+----+-------+--------------+------------------------+---------+

我已经用

创建了工作

Table SQL statement: SELECT id, name FROM campaign WHERE ${CONDITIONS}

我也试过这个 sql 语句 - 它也没有工作

Table SQL statement: SELECT cast(id as UNSIGNED INTEGER) id, name FROM campaign WHERE ${CONDITIONS}

我的工作是这样的：

sqoop:000> show job
+----+-----------------+----------------+--------------+---------+
| Id |      Name       | From Connector | To Connector | Enabled |
+----+-----------------+----------------+--------------+---------+
| 4  | campaign_import | 1              | 3            | true    |
+----+-----------------+----------------+--------------+---------+

我已经成功运行了这项工作

sqoop:000> start job -j 4

然后我在映射器上发现了错误

原因：java.lang.ClassCastException：java.math.BigInteger 不能被强制转换为 java.lang.Long at

错误的全文看起来像

Error: org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs during extractor run at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:99) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.ClassCastException: java.math.BigInteger cannot be cast to java.lang.Long at org.apache.sqoop.connector.common.SqoopIDFUtils.toCSVFixedPoint(SqoopIDFUtils.java:133) at org.apache.sqoop.connector.common.SqoopIDFUtils.toCSV(SqoopIDFUtils.java:588) at org.apache.sqoop.connector.idf.CSVIntermediateDataFormat.toCSV(CSVIntermediateDataFormat.java:116) at org.apache.sqoop.connector.idf.CSVIntermediateDataFormat.setObjectData(CSVIntermediateDataFormat.java:87) at org.apache.sqoop.job.mr.SqoopMapper$SqoopMapDataWriter.writeArrayRecord(SqoopMapper.java:125) at org.apache.sqoop.connector.jdbc.GenericJdbcExtractor.extract(GenericJdbcExtractor.java:96) at org.apache.sqoop.connector.jdbc.GenericJdbcExtractor.extract(GenericJdbcExtractor.java:38) at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:95) ... 7 more

看起来id 类型（bigint）到campaign 表的问题，但我不知道如何在不更改此字段类型的情况下解决它:)

也许，有人知道吗？

【问题讨论】：

标签： java mysql hadoop sqoop2

【解决方案1】：

当我将 Table SQL 语句更改为时，它对我有用

SELECT cast(id as SIGNED INTEGER) id, name FROM campaign WHERE ${CONDITIONS}

我仔细看了看

1) 使用 Table SQL 语句：SELECT id, name FROM campaign WHERE ${CONDITIONS} 我发现了错误：

原因：java.lang.ClassCastException：java.math.BigInteger 不能被强制转换为 java.lang.Long

2) 使用 Table SQL 语句：SELECT cast(id as UNSIGNED INTEGER) id, name FROM campaign WHERE ${CONDITIONS} 我发现错误：

原因：java.lang.ClassCastException：java.lang.Long 无法强制转换到 java.lang.Integer

所以问题在于投射，我需要将 BIGINT 投射到 SIGNED INTEGER

【讨论】：