SBT 程序集插件错误输出答案

【问题标题】：SBT Assembly plugin Errors outSBT 程序集插件错误输出
【发布时间】：2015-12-01 21:07:19
【问题描述】：

我写了下面的sbt文件

name := "Test"

version := "1.0"

scalaVersion := "2.11.7"

libraryDependencies ++= Seq(
  "org.apache.hadoop" % "hadoop-client" % "2.7.1",
  "org.apache.spark" % "spark-core_2.10" % "1.3.0",
  "org.apache.avro" % "avro" % "1.7.7",
  "org.apache.avro" % "avro-mapred" % "1.7.7"
)

mainClass := Some("com.test.Foo")

我在项目文件夹中也有以下 assembly.sbt 文件

resolvers += Resolver.url("bintray-sbt-plugins", url("http://dl.bintray.com/sbt/sbt-plugin-releases"))(Resolver.ivyStylePatterns)
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.0")

当我执行sbt assembly 时，我会收到一大串错误

[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /Users/abhishek.srivastava/.ivy2/cache/com.esotericsoftware.kryo/kryo/bundles/kryo-2.21.jar:com/esotericsoftware/minlog/Log$Logger.class
[error] /Users/abhishek.srivastava/.ivy2/cache/com.esotericsoftware.minlog/minlog/jars/minlog-1.2.jar:com/esotericsoftware/minlog/Log$Logger.class
[error] deduplicate: different file contents found in the following:
[error] /Users/abhishek.srivastava/.ivy2/cache/com.esotericsoftware.kryo/kryo/bundles/kryo-2.21.jar:com/esotericsoftware/minlog/Log.class
[error] /Users/abhishek.srivastava/.ivy2/cache/com.esotericsoftware.minlog/minlog/jars/minlog-1.2.jar:com/esotericsoftware/minlog/Log.class
[error] deduplicate: different file contents found in the following:

【问题讨论】：

标签： scala sbt sbt-assembly

【解决方案1】：

我能够解决问题。实际上没有必要构建一个胖 jar，因为“spark-submit”工具无论如何都会在类路径中包含所有内容。

因此构建jar文件的正确方法是

name := "Test"

version := "1.0"

scalaVersion := "2.11.7"

libraryDependencies ++= Seq(
  "org.apache.hadoop" % "hadoop-client" % "2.7.1" % "provided",
  "org.apache.spark" % "spark-core_2.10" % "1.3.0" % "provided",
  "org.apache.avro" % "avro" % "1.7.7" % "provided",
  "org.apache.avro" % "avro-mapred" % "1.7.7" % "provided"
)

mainClass := Some("com.test.Foo")

【讨论】：

【解决方案2】：

1。使用 MergeStrategy，见sbt-assembly

2。排除重复的jar，如：

    lazy val hbaseLibSeq = Seq(
    ("org.apache.hbase" % "hbase" % hbaseVersion).
      excludeAll(
        ExclusionRule(organization = "org.slf4j"),
        ExclusionRule(organization = "org.mortbay.jetty"),
        ExclusionRule(organization = "javax.servlet")),
    ("net.java.dev.jets3t" % "jets3t" % "0.6.1" % "provided").
      excludeAll(ExclusionRule(organization = "javax.servlet"))
  )

3。使用提供的范围

显示依赖树：

➜ 猫 ~/.sbt/0.13/plugins/plugins.sbt

addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.7.5") addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")

➜ 猫 ~/.sbt/0.13/global.sbt

net.virtualvoid.sbt.graph.Plugin.graphSettings

➜ sbt 依赖图

【讨论】：