Spark 2.2 的 SBT 编译失败

Question

我刚开始在 HDP 2.6 上使用 Spark 2.2，我在尝试进行 sbt 编译时遇到了问题

错误

[信息] 更新文件 /home/maria_dev/structuredstreaming/project/build.properties：将 sbt.version 设置为 1.3.0 [信息] 从 /home/maria_dev/structuredstreaming/project 加载项目定义 [信息] 获取工件 [信息] 获取的工件 [错误] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts：获取工件时出错： [错误] https://repo1.maven.org/maven2/com/squareup/okhttp3/okhttp-urlconnection/3.7.0/okhttp-urlconnection-3.7.0.jar: 下载错误: Caught java.net.UnknownHostException: repo1.maven.org (repo1.maven.org) 同时下载https://repo1.maven.org/maven2/com/squareup/okhttp3/okhttp-urlconnection/3.7.0/okhttp-urlconnection-3.7.0.jar

build.sbt 文件如下

buid.sbt

scalaVersion := "2.11.8"

resolvers ++= Seq(
 "Conjars" at "http://conjars.org/repo",
 "Hortonworks Releases" at "http://repo.hortonworks.com/content/groups/public"
)

publishMavenStyle := true

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "2.2.0.2.6.3.0-235",
  "org.apache.spark" %% "spark-sql" % "2.2.0.2.6.3.0-235",
  "org.apache.phoenix" % "phoenix-spark2" % "4.7.0.2.6.3.0-235",
  "org.apache.phoenix" % "phoenix-core" % "4.7.0.2.6.3.0-235",
  "org.apache.kafka" % "kafka-clients" % "0.10.1.2.6.3.0-235",
  "org.apache.spark" %% "spark-streaming" % "2.0.2" % "provided",
  "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.0.2",
  "org.apache.spark" %% "spark-sql-kafka-0-10" % "2.0.2" % "provided",
  "com.typesafe" % "config" % "1.3.1",
  "com.typesafe.play" %% "play-json" % "2.7.2",
  "com.solarmosaic.client" %% "mail-client" % "0.1.0",
  "org.json4s" %% "json4s-jackson" % "3.2.10",
  "org.apache.logging.log4j" % "log4j-api-scala_2.11" % "11.0",
  "com.databricks" %% "spark-avro" % "3.2.0",
  "org.elasticsearch" %% "elasticsearch-spark-20" % "5.0.0-alpha5",
  "io.spray" %%  "spray-json" % "1.3.3"
)

retrieveManaged := true

fork in run := true

Answer 1

看起来 coursier 正在尝试从被阻止的 repo1.maven.org 获取依赖项。 Scala-Metals 人解释一下here。基本上，您必须通过设置如下所示的mirror.properties 文件来设置指向您公司代理服务器的全局 Coursier 配置：

central.from=https://repo1.maven.org/maven2
central.to=http://mycorporaterepo.com:8080/nexus/content/groups/public

根据您的操作系统，它将是：

• Windows：C:\Users\\AppData\Roaming\Coursier\config\mirror.properties
• Linux：~/.config/coursier/mirror.properties
• MacOS：~/Library/Preferences/Coursier/mirror.properties

您可能还需要设置 SBT 以使用代理下载依赖项。为此，您需要编辑此文件：

~/.sbt/repositories

设置如下：

[repositories]
  local
  maven-central: http://mycorporaterepo.com:8080/nexus/content/groups/public

这两个设置的组合应该可以满足您将 SBT 指向正确位置所需的一切。