spark sbt 获取工件时出错 https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/2.6.5/hadoop-hdfs-2.6.5.jar 校验和错误答案

【问题标题】：spark sbt Error fetching artifacts https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/2.6.5/hadoop-hdfs-2.6.5.jar wrong checksumspark sbt 获取工件时出错 https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/2.6.5/hadoop-hdfs-2.6.5.jar 校验和错误
【发布时间】：2020-09-26 10:08:07
【问题描述】：

当我在 intellij 创建我的第一个基于 spark scala 的项目时，我的 sbt 设置工作正常。 sbt如下

name := "sample1"
version := "0.1"
//running in cluster
//scalaVersion := "2.11.0"
//val sparkVersion = "2.1.1"

//running in local
scalaVersion := "2.11.12"
val sparkVersion = "2.3.2"

//sbt when running in cluster using spark-submit

/*
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-mllib" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided"
)
*/

//sbt when running locally

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.apache.spark" %% "spark-mllib" % sparkVersion,
  "org.apache.spark" %% "spark-streaming" % sparkVersion
)

但是当我使用新的 sbt 设置创建一个新项目时，我遇到了错误。尝试更改版本但没有帮助。新项目的新 sbt 是

name := "ranger_policy"

version := "0.1"

scalaVersion := "2.11.11"


libraryDependencies ++= Seq(
  "org.apache.spark" % "spark-core_2.11" % "2.2.0",
  "org.apache.spark" % "spark-sql_2.11" % "2.2.0",
  "org.apache.spark" % "spark-mllib_2.10" % "1.1.0"
)

我的错误是

[error] stack trace is suppressed; run 'last update' for the full output
[error] stack trace is suppressed; run 'last ssExtractDependencies' for the full output
[error] (update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/2.6.5/hadoop-hdfs-2.6.5.jar: wrong checksum: C:\Users\vidushi.jaiswal\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\apache\hadoop\hadoop-hdfs\2.6.5\hadoop-hdfs-2.6.5.jar (expected SHA-1 8bd0f95e29b9ba7960b4239a7f3706b37b183652 in C:\Users\vidushi.jaiswal\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\apache\hadoop\hadoop-hdfs\2.6.5\.hadoop-hdfs-2.6.5.jar__sha1, got 495045c7fe5110559fa09d7f2381cd3483189a7)
[error] (ssExtractDependencies) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/2.6.5/hadoop-hdfs-2.6.5.jar: wrong checksum: C:\Users\vidushi.jaiswal\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\apache\hadoop\hadoop-hdfs\2.6.5\hadoop-hdfs-2.6.5.jar (expected SHA-1 8bd0f95e29b9ba7960b4239a7f3706b37b183652 in C:\Users\vidushi.jaiswal\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\apache\hadoop\hadoop-hdfs\2.6.5\.hadoop-hdfs-2.6.5.jar__sha1, got 495045c7fe5110559fa09d7f2381cd3483189a7)
[error] Total time: 247 s (04:07), completed Jun 7, 2020 5:35:48 PM
[info] shutting down sbt server

【问题讨论】：

为什么要在第二个 SBT 文件中使用不同版本的 Spark 依赖项？以及针对您的 Spark 依赖项的不同 Scala 版本？您应该再读一次这些依赖项的含义。也许还可以开始阅读有关 SBT 和 %% 操作员的相关信息。另外，为什么有人会使用旧版 Spark 而不是最新版？

标签： scala apache-spark sbt

【解决方案1】：

我在通过 sbt 导入 Spark SQL 包时也遇到了同样的问题。有多个 spark-sql 文件夹，即 spark-sql 2.12 和 spark-sql 2.11。当我删除它们并刷新 build.sbt 时，问题得到了解决。

在这个路径 C:\Users\vidushi.jaiswal\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\apache\hadoop\hadoop-hdfs\

对于同一个包，您可能有多个不同版本的文件夹。删除所有这些并刷新 build.sbt。所有依赖项都将重新下载。

【讨论】：