Spark EMR 作业杰克逊错误 - java.lang.NoSuchMethodError - AnnotatedMember.getType()Lcom/fasterxml/jackson/databind/JavaType

Question

我知道我们已经在这里回答了类似的问题。但由于某种原因，没有一个选项对我有用。

以下是我得到的错误：

User class threw exception: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.introspect.AnnotatedMember.getType()Lcom/fasterxml/jackson/databind/JavaType;
at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector._fullSerializationType(JaxbAnnotationIntrospector.java:1550)
at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector._findContentAdapter(JaxbAnnotationIntrospector.java:1502)
at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.findSerializationContentConverter(JaxbAnnotationIntrospector.java:914)
at com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.findSerializationContentConverter(AnnotationIntrospectorPair.java:388)
at com.fasterxml.jackson.databind.ser.std.StdSerializer.findConvertingContentSerializer(StdSerializer.java:266)
at com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.createContextual(AsArraySerializerBase.java:197)

以下是我添加的依赖项。我尝试删除所有杰克逊依赖项并单独添加所需的版本。

dependencies {
compile (group: 'org.scala-lang', name: 'scala-library', version: '2.11.8') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}

compile (group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.3.0') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}
compile (group: 'org.apache.spark', name: 'spark-sql_2.11', version: '2.3.0') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}

compile (group: 'com.amazonaws', name: 'aws-java-sdk', version: '1.11.633') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}

compile group: 'org.apache.httpcomponents', name: 'httpclient', version: '4.5.2'
compile group: 'com.sun.mail', name: 'javax.mail', version: '1.6.2'

testCompile group: 'junit', name: 'junit', version: '4.12'

compile group: 'net.liftweb', name: 'lift-json_2.11', version: '2.6.3'
compile group: 'net.sf.json-lib', name: 'json-lib', version: '2.4', classifier: 'jdk15'
compile group: 'net.sf.oval', name: 'oval', version: '1.31'

compile (group: 'com.bedatadriven', name: 'jackson-datatype-jts', version: '2.4') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}

compile (group: 'com.fasterxml.jackson.datatype', name: 'jackson-datatype-joda', version: '2.11.2') {
    exclude group: 'com.fasterxml.jackson.core'
}

compile group: 'com.fasterxml.jackson.core', name: 'jackson-core', version: '2.11.2'
compile (group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.11.2') {
    exclude group: 'com.fasterxml.jackson.core'
}

compile group: 'com.fasterxml.jackson.core', name: 'jackson-annotations', version: '2.11.2'

compile (group: 'com.fasterxml.jackson.module', name: 'jackson-module-jaxb-annotations', version: '2.11.2') {
    exclude group: 'com.fasterxml.jackson.core'
}

compile (group: 'com.fasterxml.jackson.dataformat', name: 'jackson-dataformat-csv', version: '2.11.2') {
     exclude group: 'com.fasterxml.jackson.core'
 }

// Some custom Jars

compile (group: 'org.springframework', name: 'spring-core', version: '2.5.6') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}

}

当我在本地系统中运行 spark 作业而不是在 aws emr 集群中时，此配置有效。

Answer 1

Spark 提供 Jackson 本身，而不是您期望的版本。该错误可能是由于类路径中有 2 个冲突的 Jackson 版本引起的。

你有两个选择：

• 强制具有某些属性的特定 Jackson 版本（例如，请参阅 Classpath resolution between spark uber jar and spark-submit --jars when similar classes exist in both）
• 更新您的构建定义以使用 Spark 提供的相同 Jackson 版本

Spark 2.3.0 与 Jackson 2.6.7.1 一起提供（可以在这里查看，例如：https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11/2.3.0）。