spark 2.4.3 的 build.sbt 未解决的依赖项

Question

我是 scala 的新手，并尝试使用 Scala version 2.11.12 和 spark version 2.4.3 和 JDK 1.8.0_212 在集群上运行 Spark 应用程序。我的 build.sbt 看起来像这样 -

name := "LearningScala"
version := "1.0"
scalaVersion := "2.11.12"
val sparkVersion = "2.4.3"

//resolvers += "Maven Dependencies" at "https://repo1.maven.org/maven2/"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion
)

如您所见，我也尝试使用resolvers，但没有任何效果。错误日志说以下是未解决的依赖项 -

[error] sbt.librarymanagement.ResolveException: unresolved dependency: org.codehaus.jackson#jackson-core-asl;1.9.13: configuration not found in org.codehaus.jackson#jackson-core-asl;1.9.13: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.parquet#parquet-hadoop;1.10.1 compile
[error] unresolved dependency: org.codehaus.jackson#jackson-mapper-asl;1.9.13: configuration not found in org.codehaus.jackson#jackson-mapper-asl;1.9.13: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.parquet#parquet-hadoop;1.10.1 compile
[error] unresolved dependency: commons-httpclient#commons-httpclient;3.1: configuration not found in commons-httpclient#commons-httpclient;3.1: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.hadoop#hadoop-common;2.6.5 compile
[error] unresolved dependency: commons-collections#commons-collections;3.2.2: configuration not found in commons-collections#commons-collections;3.2.2: 'master(compile)'. Missing configuration: 'compile'. It was required from commons-configuration#commons-configuration;1.6 compile
[error] unresolved dependency: log4j#log4j;1.2.17: configuration not found in log4j#log4j;1.2.17: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.hadoop#hadoop-auth;2.6.5 runtime
[error] unresolved dependency: commons-lang#commons-lang;2.6: configuration not found in commons-lang#commons-lang;2.6: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.orc#orc-mapreduce;1.5.5 compile
[error] unresolved dependency: javax.inject#javax.inject;1: configuration not found in javax.inject#javax.inject;1: 'master(compile)'. Missing configuration: 'compile'. It was required from com.google.inject#guice;3.0 compile
[error] unresolved dependency: org.xerial.snappy#snappy-java;1.1.7.3: configuration not found in org.xerial.snappy#snappy-java;1.1.7.3: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.parquet#parquet-hadoop;1.10.1 compile
[error] unresolved dependency: org.lz4#lz4-java;1.4.0: configuration not found in org.lz4#lz4-java;1.4.0: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.spark#spark-core_2.11;2.4.3 compile
[error] unresolved dependency: com.thoughtworks.paranamer#paranamer;2.8: configuration not found in com.thoughtworks.paranamer#paranamer;2.8: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.spark#spark-core_2.11;2.4.3 runtime
[error] unresolved dependency: javax.ws.rs#javax.ws.rs-api;2.0.1: configuration not found in javax.ws.rs#javax.ws.rs-api;2.0.1: 'master(compile)'. Missing configuration: 'compile'. It was required from org.glassfish.jersey.containers#jersey-container-servlet;2.22.2 compile
[error] unresolved dependency: javax.annotation#javax.annotation-api;1.2: configuration not found in javax.annotation#javax.annotation-api;1.2: 'master(compile)'. Missing configuration: 'compile'. It was required from org.glassfish.jersey.core#jersey-server;2.22.2 compile
[error] unresolved dependency: javax.validation#validation-api;1.1.0.Final: configuration not found in javax.validation#validation-api;1.1.0.Final: 'master(compile)'. Missing configuration: 'compile'. It was required from org.glassfish.jersey.core#jersey-server;2.22.2 compile
[error] unresolved dependency: io.dropwizard.metrics#metrics-json;3.1.5: configuration not found in io.dropwizard.metrics#metrics-json;3.1.5:

有人用 2.11 和 2.4.3 吗？我的 spark shell 显示了这些值，我可以手动导入这样的包 -

$ spark-shell
scala> scala> import org.apache.spark._
import org.apache.spark._

scala> import org.apache.spark.sql._
import org.apache.spark.sql._

那为什么 build.sbt 无法获取呢？

Answer 1

这是我试图解决这个问题的方法： 我先删除了~/.ivy2 目录，然后开始从2.2.0 升级Spark 版本，同时保留scalaVersion := 2.11.12。完美运行的 Spark 最终版本是2.4.3。

虽然我尝试了试错法，但我认为之前版本2.4.3无法下载的一些依赖是它之前的版本下载的。

我尝试并成功运行的 Spark 版本列表 -

• 2.2.0
• 2.2.1
• 2.3.0
• 2.3.1
• 2.4.0
• 2.4.3

我知道这很奇怪，但对我有用。我认为问题在于已经下载到~/.ivy2 目录中的缓存包。仅删除 .ivy2 中的 cache/ 将无济于事。

Answer 2

在解析器之前删除评论。另外，尝试以这种方式添加repo

resolvers ++= Seq(
  DefaultMavenRepository,
  Resolver.bintrayRepo("typesafe", "releases"),
  Resolver.sonatypeRepo("releases"),
  Resolver.mavenLocal
)