【发布时间】：2016-02-19 06:15:52
【问题描述】：
Spark 环境的新手（对 Maven 来说相当新），所以我正在努力解决如何正确发送我需要的依赖项。
看起来 Spark 1.5.1 有一个它尝试使用的 guava-14.0.1 依赖项，并且 isPrimitive 是在 15+ 中添加的。确保我的 uber-jar 获胜的正确方法是什么？我在 spark-defaults.conf 中尝试过 spark.executor.extraClassPath 无济于事。
复制到这个[问题]：Spark 1.5.1 + Scala 2.10 + Kafka + Cassandra = Java.lang.NoSuchMethodError: 但本质上是针对 Maven（还没有代表发表评论）
将我的依赖项剥离到这个：
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>18.0</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-compress</artifactId>
<version>1.10</version>
</dependency>
<dependency>
<groupId>com.esotericsoftware.kryo</groupId>
<artifactId>kryo</artifactId>
<version>2.21</version>
</dependency>
<dependency>
<groupId>org.objenesis</groupId>
<artifactId>objenesis</artifactId>
<version>2.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.5.0</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.10</artifactId>
<version>1.5.0-M2</version>
</dependency>
使用以下所有依赖项为我的 JAR 着色：
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<excludes>
<exclude>org.apache.hadoop:*</exclude>
<exclude>org.apache.hbase:*</exclude>
</excludes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
<filter>
<artifact>org.apache.spark:spark-network-common_2.10</artifact>
<excludes>
<exclude>com.google.common.base.*</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<!-- merge multiple reference.conf files into one -->
<resource>reference.conf</resource>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
这是我跑步时的惊人爆炸
./spark-submit --master local --class <my main class> <my shaded jar>
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.reflect.TypeToken.isPrimitive()Z
at com.datastax.driver.core.TypeCodec.<init>(TypeCodec.java:142)
at com.datastax.driver.core.TypeCodec.<init>(TypeCodec.java:136)
at com.datastax.driver.core.TypeCodec$BlobCodec.<init>(TypeCodec.java:609)
at com.datastax.driver.core.TypeCodec$BlobCodec.<clinit>(TypeCodec.java:606)
at com.datastax.driver.core.CodecRegistry.<clinit>(CodecRegistry.java:147)
at com.datastax.driver.core.Configuration$Builder.build(Configuration.java:259)
at com.datastax.driver.core.Cluster$Builder.getConfiguration(Cluster.java:1135)
at com.datastax.driver.core.Cluster.<init>(Cluster.java:111)
at com.datastax.driver.core.Cluster.buildFrom(Cluster.java:178)
at com.datastax.driver.core.Cluster$Builder.build(Cluster.java:1152)
at com.datastax.spark.connector.cql.DefaultConnectionFactory$.createCluster(CassandraConnectionFactory.scala:85)
at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:155)
【问题讨论】：
标签： java scala maven apache-spark spark-cassandra-connector