Spark app [error] (*:assembly) deduplicate: 不同的文件内容在下面找到：

Question

有谁知道如何避免以下重复文件错误？

我有一个 Spark 应用程序，该应用程序一直在 InteliJ 社区版中运行，直到昨晚，并且出于不明原因，来自 esotericsoftware 的两个 jar 现在正在引发错误。当我现在执行 sbt 程序集时，我收到以下错误消息。

   [warn] Merging 'META-INF\taglib.tld' with strategy 'discard'
java.lang.RuntimeException: deduplicate: different file contents found in the following:
C:\Users\osadmin\.ivy2\cache\com.esotericsoftware.kryo\kryo\bundles\kryo-2.21.jar:com/esotericsoftware/minlog/Log$Logger.class
C:\Users\osadmin\.ivy2\cache\com.esotericsoftware.minlog\minlog\jars\minlog-1.2.jar:com/esotericsoftware/minlog/Log$Logger.class
        at sbtassembly.Plugin$Assembly$.sbtassembly$Plugin$Assembly$$applyStrategy$1(Plugin.scala:253)
        at sbtassembly.Plugin$Assembly$$anonfun$15.apply(Plugin.scala:270)
        at sbtassembly.Plugin$Assembly$$anonfun$15.apply(Plugin.scala:267)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
        at sbtassembly.Plugin$Assembly$.applyStrategies(Plugin.scala:272)
        at sbtassembly.Plugin$Assembly$.x$4$lzycompute$1(Plugin.scala:172)
        at sbtassembly.Plugin$Assembly$.x$4$1(Plugin.scala:170)
        at sbtassembly.Plugin$Assembly$.stratMapping$lzycompute$1(Plugin.scala:170)
        at sbtassembly.Plugin$Assembly$.stratMapping$1(Plugin.scala:170)
        at sbtassembly.Plugin$Assembly$.inputs$lzycompute$1(Plugin.scala:214)
        at sbtassembly.Plugin$Assembly$.inputs$1(Plugin.scala:204)
        at sbtassembly.Plugin$Assembly$.apply(Plugin.scala:230)
        at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:373)
        at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:370)
        at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
        at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
        at sbt.std.Transform$$anon$4.work(System.scala:63)
        at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
        at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
        at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
        at sbt.Execute.work(Execute.scala:235)
        at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
        at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
        at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] C:\Users\osadmin\.ivy2\cache\com.esotericsoftware.kryo\kryo\bundles\kryo-2.21.jar:com/esotericsoftware/minlog/Log$Logger.class
[error] C:\Users\osadmin\.ivy2\cache\com.esotericsoftware.minlog\minlog\jars\minlog-1.2.jar:com/esotericsoftware/minlog/Log$Logger.class

我正在使用 sbt 并且确实有一个如下的合并策略。

libraryDependencies ++= Seq("org.apache.spark" % "spark-core_2.10" % "1.5.2" % "provided" exclude("org.apache.hadoop", "hadoop-client"),
  ("org.mongodb" % "mongo-java-driver" % "3.2.2")
    .exclude("commons-beanutils", "commons-beanutils-core")
    .exclude("commons-beanutils", "commons-beanutils") ,
  ("org.mongodb.mongo-hadoop" % "mongo-hadoop-core" % "1.4.2")
    .exclude("commons-beanutils", "commons-beanutils-core")
    .exclude("commons-beanutils", "commons-beanutils"),
  ("com.stratio.datasource" % "spark-mongodb_2.10" % "0.11.0"),
  ("org.apache.spark" % "spark-sql_2.10" % "1.5.2" % "provided"),
  ("org.apache.spark" % "spark-hive_2.10" % "1.5.2" % "provided"),
  ("org.apache.spark" % "spark-streaming_2.10" % "1.5.2")
)

assemblySettings

jarName in assembly := "mongotest.jar"

val meta = """META.INF(.)*""".r

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
  case PathList("org", "apache", xs @ _*) => MergeStrategy.last
  case PathList("plugin.properties") => MergeStrategy.last
  case meta(_) => MergeStrategy.discard
  case x => old(x)
}
}


assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)

Answer 1

今天经过大量搜索，我在 Stackoverflow 中找到了答案。问题是 Spark-streaming 依赖项缺少“提供”。这是另一个问题的网址，其中包含 4e6 的明确答案。 error while running sbt assembly : sbt deduplication error