使用 sbt-assembly 构建多项目 fat jars答案

【问题标题】：Build multi-project fat jars with sbt-assembly使用 sbt-assembly 构建多项目 fat jars
【发布时间】：2018-11-18 08:17:40
【问题描述】：

我有多个项目，主模块称为根、消费者和生产者模块，其依赖项依赖于核心模块。核心模块包含与配置相关的类。我想用 sbt-assembly 为消费者和生产者构建 2 个单独的主类。但是，当我尝试像 sbt consumer/assembly 那样单独构建它们或通过运行 sbt assembly 完全构建它们时，会出现这样的错误，并且 sbt 无法编译整个项目：

[error] (consumer / update) java.lang.IllegalArgumentException: a module is not authorized to depend on itself: demotest#demotest_2.11;0.1
[error] (producer / update) java.lang.IllegalArgumentException: a module is not authorized to depend on itself: demotest#demotest_2.11;0.1

我应该在依赖项中更改哪些内容才能正确打包 jar？我应该单独运行它们还是一起运行它们？

lazy val overrides = Seq("com.fasterxml.jackson.core" % "jackson-core" % "2.9.5",
  "com.fasterxml.jackson.core" % "jackson-databind" % "2.9.5",
  "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.9.5")

lazy val commonSettings = Seq(
  name := "DemoTest",
  version := "0.1",
  scalaVersion := "2.11.8",
  resolvers += "Spark Packages Repo" at "http://dl.bintray.com/spark-packages/maven",
  dependencyOverrides ++= overrides
)

lazy val assemblySettings = Seq(
  assemblyMergeStrategy in assembly := {
    case PathList("org","aopalliance", xs @ _*) => MergeStrategy.last
    case PathList("javax", "inject", xs @ _*) => MergeStrategy.last
    case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
    case PathList("javax", "activation", xs @ _*) => MergeStrategy.last
    case PathList("org", "apache", xs @ _*) => MergeStrategy.last
    case PathList("com", "google", xs @ _*) => MergeStrategy.last
    case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last
    case PathList("com", "codahale", xs @ _*) => MergeStrategy.last
    case PathList("com", "yammer", xs @ _*) => MergeStrategy.last
    case PathList("org", "slf4j", xs @ _*) => MergeStrategy.last
    case PathList("org", "neo4j", xs @ _*) => MergeStrategy.last
    case PathList("com", "typesafe", xs @ _*) => MergeStrategy.last
    case PathList("net", "jpountz", xs @ _*) => MergeStrategy.last
    case PathList("META-INF", xs @ _*) => MergeStrategy.discard
    case "about.html" => MergeStrategy.rename
    case "META-INF/ECLIPSEF.RSA" => MergeStrategy.last
    case "META-INF/mailcap" => MergeStrategy.last
    case "META-INF/mimetypes.default" => MergeStrategy.last
    case "plugin.properties" => MergeStrategy.last
    case "log4j.properties" => MergeStrategy.last
    case x =>
      val oldStrategy = (assemblyMergeStrategy in assembly).value
      oldStrategy(x)
  }
)

lazy val sparkVersion = "2.2.0"

lazy val commonDependencies = Seq(
  "org.apache.kafka" %% "kafka" % "1.1.0",
  "org.apache.spark" %% "spark-core" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.apache.spark" %% "spark-streaming" % sparkVersion,
  "org.apache.spark" %% "spark-streaming-kafka-0-10" % sparkVersion,
  "neo4j-contrib" % "neo4j-spark-connector" % "2.1.0-M4",
  "com.typesafe" % "config" % "1.3.0",
  "org.neo4j.driver" % "neo4j-java-driver" % "1.5.1",
  "com.opencsv" % "opencsv" % "4.1",
  "com.databricks" %% "spark-csv" % "1.5.0",
  "com.github.tototoshi" %% "scala-csv" % "1.3.5",
  "org.elasticsearch" %% "elasticsearch-spark-20" % "6.2.4"
)

lazy val root = (project in file("."))
  .aggregate(consumer, producer)



lazy val core = (project in file("core"))
  .settings(
    commonSettings,
    libraryDependencies ++= commonDependencies
  )

lazy val consumer = (project in file("consumer"))
  .settings(
    commonSettings,
    assemblySettings,
    libraryDependencies ++= commonDependencies)
  .settings(
    mainClass in assembly := Some("consumer.SparkConsumer"),
    assemblyJarName in assembly := "demo_consumer.jar"
  ).dependsOn(core)

lazy val producer = (project in file("producer"))
  .settings(
    commonSettings,
    assemblySettings,
    libraryDependencies ++= commonDependencies
  ).settings(
  mainClass in assembly := Some("producer.KafkaCheckinsProducer"),
  assemblyJarName in assembly := "demo_producer.jar"
).dependsOn(core)

【问题讨论】：

commonSettings 和 commonDependencies 在 core 中，然后也包含在 producer 和 consumer 中。既然这两个依赖core，那岂不是多余？您是否尝试将它们从 producer 和 consumer 中删除？
删除后我得到另一个错误：(consumer / update) sbt.librarymanagement.ResolveException: unresolved dependency: demotest#demotest_2.12;0.1: not found (producer / update) sbt.librarymanagement.ResolveException: unresolved dependency: demotest#demotest_2.12;0.1: not found

标签： scala sbt sbt-assembly multi-project

【解决方案1】：

问题出在这一行：

lazy val commonSettings = Seq(
  name := "DemoTest",  // should not be defined at this level!
)

这个name应该为root定义：

lazy val root = (project in file("."))
  .aggregate(core, producer, consumer)
  .settings(
    name := "demotest"
  )

子模块将使用“core”、“producer”和“consumer”作为它们自己的名称，除非在它们的设置中没有覆盖name。

还请注意，在您的示例中，.aggregate 不包括 core，但确实应该。

【讨论】：