杰克逊投掷错误

Question

尝试在 STORM Trident 中将 XML 转换为 JSON 我的代码在 Storm 0.9.1 版本中运行良好，现在我们升级到 Storm 0.10.0 的最新版本，如果您对原因有任何想法，我们会收到以下错误让我知道？

11752 [Thread-56-b-1] ERROR b.s.util - Async loop died!
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.JavaType.isReferenceType()Z
    at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findDefaultDeserializer(BasicDeserializerFactory.java:1522) ~[StormTridentXmlNew-0.0.1-SNAPSHOT.jar:?]
    at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.findStdDeserializer(BeanDeserializerFactory.java:167) ~[StormTridentXmlNew-0.0.1-SNAPSHOT.jar:?]
    at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:132) ~[StormTridentXmlNew-0.0.1-SNAPSHOT.jar:?]
    at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:403) ~[StormTridentXmlNew-0.0.1-SNAPSHOT.jar:?]
    at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:352) ~[StormTridentXmlNew-0.0.1-SNAPSHOT.jar:?]
    at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264) ~[StormTridentXmlNew-0.0.1-SNAPSHOT.jar:?]
    at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244) ~[StormTridentXmlNew-0.0.1-SNAPSHOT.jar:?]
    at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142) ~[StormTridentXmlNew-0.0.1-SNAPSHOT.jar:?]
    at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:461) ~[StormTridentXmlNew-0.0.1-SNAPSHOT.jar:?]
    at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:3804) ~[StormTridentXmlNew-0.0.1-SNAPSHOT.jar:?]
    at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3698) ~[StormTridentXmlNew-0.0.1-SNAPSHOT.jar:?]
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2714) ~[StormTridentXmlNew-0.0.1-SNAPSHOT.jar:?]
    at com.tda.trident.bolt.HdfsStateConvertor$HdfsFileOptions.execute(HdfsStateConvertor.java:278) ~[StormTridentXmlNew-0.0.1-SNAPSHOT.jar:?]
    at com.tda.trident.bolt.HdfsStateConvertor.updateState(HdfsStateConvertor.java:462) ~[StormTridentXmlNew-0.0.1-SNAPSHOT.jar:?]
    at com.tda.trident.bolt.HdfsUpdater.updateState(HdfsUpdater.java:15) ~[StormTridentXmlNew-0.0.1-SNAPSHOT.jar:?]
    at com.tda.trident.bolt.HdfsUpdater.updateState(HdfsUpdater.java:9) ~[StormTridentXmlNew-0.0.1-SNAPSHOT.jar:?]
    at storm.trident.planner.processor.PartitionPersistProcessor.finishBatch(PartitionPersistProcessor.java:98) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
    at storm.trident.planner.SubtopologyBolt.finishBatch(SubtopologyBolt.java:152) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
    at storm.trident.topology.TridentBoltExecutor.finishBatch(TridentBoltExecutor.java:253) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
    at storm.trident.topology.TridentBoltExecutor.checkFinish(TridentBoltExecutor.java:286) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
    at storm.trident.topology.TridentBoltExecutor.execute(TridentBoltExecutor.java:360) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
    at backtype.storm.daemon.executor$fn__6214$tuple_action_fn__6216.invoke(executor.clj:670) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
    at backtype.storm.daemon.executor$mk_task_receiver$fn__6137.invoke(executor.clj:426) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
    at backtype.storm.disruptor$clojure_handler$reify__5713.onEvent(disruptor.clj:58) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
    at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
    at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
    at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
    at backtype.storm.daemon.executor$fn__6214$fn__6227$fn__6278.invoke(executor.clj:808) ~[storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
    at backtype.storm.util$async_loop$fn__543.invoke(util.clj:475) [storm-core-0.10.0.2.3.2.0-2950.jar:0.10.0.2.3.2.0-2950]
    at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
    at java.lang.Thread.run(Thread.java:744) [?:1.7.0_45]

我在构建 POM 时的 maven 阴影输出如下所示。

[INFO] --- maven-shade-plugin:1.4:shade (default) @ StormTridentXmlNew ---
[INFO] Including org.apache.hadoop:hadoop-auth:jar:2.7.1.2.3.2.0-2950 in the shaded jar.
[INFO] Including org.apache.curator:curator-framework:jar:2.7.1 in the shaded jar.
[INFO] Including org.apache.storm:storm-hdfs:jar:0.10.0.2.3.2.0-2950 in the shaded jar.
[INFO] Including org.yaml:snakeyaml:jar:1.11 in the shaded jar.
[INFO] Including com.hmsonline:storm-jms:jar:0.2.2 in the shaded jar.
[INFO] Including org.apache.geronimo.specs:geronimo-jms_1.1_spec:jar:1.1.1 in the shaded jar.
[INFO] Including com.tibco.ems:tibjms:jar:5.1 in the shaded jar.
[INFO] Including org.springframework:spring-beans:jar:4.1.3.RELEASE in the shaded jar.
[INFO] Including org.springframework:spring-core:jar:4.1.3.RELEASE in the shaded jar.
[INFO] Including commons-logging:commons-logging:jar:1.2 in the shaded jar.
[INFO] Including org.springframework:spring-context:jar:4.1.3.RELEASE in the shaded jar.
[INFO] Including org.springframework:spring-aop:jar:4.1.3.RELEASE in the shaded jar.
[INFO] Including aopalliance:aopalliance:jar:1.0 in the shaded jar.
[INFO] Including org.springframework:spring-expression:jar:4.1.3.RELEASE in the shaded jar.
[INFO] Including org.springframework:spring-jms:jar:4.1.3.RELEASE in the shaded jar.
[INFO] Including org.springframework:spring-messaging:jar:4.1.3.RELEASE in the shaded jar.
[INFO] Including org.springframework:spring-tx:jar:4.1.3.RELEASE in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.6.5 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.6.5 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.6.5 in the shaded jar.
[INFO] Including com.fasterxml.jackson.dataformat:jackson-dataformat-xml:jar:2.6.5 in the shaded jar.
[INFO] Including com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.6.5 in the shaded jar.
[INFO] Including org.codehaus.woodstox:stax2-api:jar:3.1.4 in the shaded jar.
[INFO] Including org.codehaus.woodstox:woodstox-core-asl:jar:4.4.1 in the shaded jar.
[INFO] Including javax.xml.stream:stax-api:jar:1.0-2 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-client:jar:2.7.1 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.7.1 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.7.1 in the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-client:jar:2.7.1 from the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-server-common:jar:2.7.1 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.7.1 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-api:jar:2.7.1 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.1 in the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-common:jar:2.7.1 from the shaded jar.
[INFO] Including javax.xml.bind:jaxb-api:jar:2.2.2 in the shaded jar.
[INFO] Including javax.activation:activation:jar:1.1 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-client:jar:1.9 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.7.1 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-annotations:jar:2.7.1 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-common:jar:2.7.1 in the shaded jar.
[INFO] Including com.google.guava:guava:jar:11.0.2 in the shaded jar.
[INFO] Including commons-cli:commons-cli:jar:1.2 in the shaded jar.
[INFO] Including org.apache.commons:commons-math3:jar:3.1.1 in the shaded jar.
[INFO] Including xmlenc:xmlenc:jar:0.52 in the shaded jar.
[INFO] Including commons-httpclient:commons-httpclient:jar:3.1 in the shaded jar.
[INFO] Including commons-codec:commons-codec:jar:1.4 in the shaded jar.
[INFO] Including commons-io:commons-io:jar:2.4 in the shaded jar.
[INFO] Including commons-net:commons-net:jar:3.1 in the shaded jar.
[INFO] Including commons-collections:commons-collections:jar:3.2.1 in the shaded jar.
[INFO] Including javax.servlet:servlet-api:jar:2.5 in the shaded jar.
[INFO] Including org.mortbay.jetty:jetty:jar:6.1.26 in the shaded jar.
[INFO] Including org.mortbay.jetty:jetty-util:jar:6.1.26 in the shaded jar.
[INFO] Including javax.servlet.jsp:jsp-api:jar:2.1 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-core:jar:1.9 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-json:jar:1.9 in the shaded jar.
[INFO] Including org.codehaus.jettison:jettison:jar:1.1 in the shaded jar.
[INFO] Including com.sun.xml.bind:jaxb-impl:jar:2.2.3-1 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-server:jar:1.9 in the shaded jar.
[INFO] Including asm:asm:jar:3.1 in the shaded jar.
[INFO] Including log4j:log4j:jar:1.2.17 in the shaded jar.
[INFO] Including net.java.dev.jets3t:jets3t:jar:0.9.0 in the shaded jar.
[INFO] Including org.apache.httpcomponents:httpclient:jar:4.1.2 in the shaded jar.
[INFO] Including org.apache.httpcomponents:httpcore:jar:4.1.2 in the shaded jar.
[INFO] Including com.jamesmurty.utils:java-xmlbuilder:jar:0.4 in the shaded jar.
[INFO] Including commons-lang:commons-lang:jar:2.6 in the shaded jar.
[INFO] Including commons-configuration:commons-configuration:jar:1.6 in the shaded jar.
[INFO] Including commons-digester:commons-digester:jar:1.8 in the shaded jar.
[INFO] Excluding commons-beanutils:commons-beanutils:jar:1.7.0 from the shaded jar.
[INFO] Excluding commons-beanutils:commons-beanutils-core:jar:1.8.0 from the shaded jar.
[INFO] Excluding org.slf4j:slf4j-api:jar:1.7.10 from the shaded jar.
[INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.10 from the shaded jar.
[INFO] Including org.apache.avro:avro:jar:1.7.4 in the shaded jar.
[INFO] Including com.thoughtworks.paranamer:paranamer:jar:2.3 in the shaded jar.
[INFO] Including org.xerial.snappy:snappy-java:jar:1.0.4.1 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:2.5.0 in the shaded jar.
[INFO] Including com.google.code.gson:gson:jar:2.2.4 in the shaded jar.
[INFO] Including com.jcraft:jsch:jar:0.1.42 in the shaded jar.
[INFO] Including org.apache.curator:curator-client:jar:2.7.1 in the shaded jar.
[INFO] Including org.apache.curator:curator-recipes:jar:2.7.1 in the shaded jar.
[INFO] Including com.google.code.findbugs:jsr305:jar:3.0.0 in the shaded jar.
[INFO] Including org.apache.htrace:htrace-core:jar:3.1.0-incubating in the shaded jar.
[INFO] Including org.apache.zookeeper:zookeeper:jar:3.4.6 in the shaded jar.
[INFO] Including io.netty:netty:jar:3.7.0.Final in the shaded jar.
[INFO] Including org.apache.commons:commons-compress:jar:1.4.1 in the shaded jar.
[INFO] Including org.tukaani:xz:jar:1.0 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-hdfs:jar:2.7.1 in the shaded jar.
[INFO] Including commons-daemon:commons-daemon:jar:1.0.13 in the shaded jar.
[INFO] Including io.netty:netty-all:jar:4.0.23.Final in the shaded jar.
[INFO] Including xerces:xercesImpl:jar:2.9.1 in the shaded jar.
[INFO] Including org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 in the shaded jar.
[INFO] Replacing original artifact with shaded artifact.

Answer 1

看起来您有许多版本的 jackson jar，并且您的类路径在运行时引用了 2.6 之前的一些 Jackson jar。

类 ResolvedType 从 2.6 版开始引入 isReferenceType()。请参考以下链接：

http://googleweblight.com/?lite_url=http://fasterxml.github.io/jackson-core/javadoc/2.6/com/fasterxml/jackson/core/type/ResolvedType.html&ei=ochB8Ipc&lc=en-IN&geid=10&s=1&m=71&ts=1454581842&sig=ALL1Aj7I2EAu3AoKmOu6FKbtFTTuKU02Bg#a-isReferenceType()

请确保您在运行时引用正确的版本或更高版本。如果不需要，您可以删除/删除较旧的 Jackson jar。

希望这会有所帮助。

Answer 2

storm 的最新版本在其 lib 文件夹中有 jackson-core-2.3.1.jar，而我使用 maven 创建的 fat jar 有另一个版本的 jackson 依赖项。这就是为什么当我在风暴集群中提交我的 jar 时，它会在运行时抛出这些 No such method 错误..

我将我的 POM 更改为使用相同版本的 jackson 依赖项，storm 正在使用，一切正常。

Answer 3

这个问题似乎来自类路径中杰克逊类的多个版本。您可以删除不需要的版本。在撰写本文时。 jackson 的当前版本是 2.7.1。