【问题标题】:Why is my build.sbt looking for version 2.11 of hadoop-streaming?为什么我的 build.sbt 正在寻找 2.11 版的 hadoop-streaming?
【发布时间】:2017-01-12 11:32:20
【问题描述】:

我正在做关于 Apache Spark 的复数课程,有一次他们要求我们设置对 Hadoop-streaming 的依赖。我已将其添加到我的 build.sbt 文件中,但得到的结果出乎意料:

构建.sbt

name := "SparkPlayground"

version := "1.0"

scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0" % "provided"
libraryDependencies += "com.github.scala-incubator.io" %% "scala-io-core" % "0.4.3"
libraryDependencies += "com.github.scala-incubator.io" %% "scala-io-file" % "0.4.3"
libraryDependencies += "org.apache.hadoop" %% "hadoop-streaming" % "2.7.0"

错误信息

SBT 'SparkPlayground' project refresh failed
    Error:Error:Error while importing SBT project:<br/>...<br/><pre>[info] Resolving org.scala-sbt#task-system;0.13.8 ...

[info] Resolving org.scala-sbt#tasks;0.13.8 ...

[info] Resolving org.scala-sbt#tracking;0.13.8 ...

[info] Resolving org.scala-sbt#cache;0.13.8 ...

[info] Resolving org.scala-sbt#testing;0.13.8 ...

[info] Resolving org.scala-sbt#test-agent;0.13.8 ...

[info] Resolving org.scala-sbt#test-interface;1.0 ...

[info] Resolving org.scala-sbt#main-settings;0.13.8 ...

[info] Resolving org.scala-sbt#apply-macro;0.13.8 ...

[info] Resolving org.scala-sbt#command;0.13.8 ...

[info] Resolving org.scala-sbt#logic;0.13.8 ...

[info] Resolving org.scala-sbt#precompiled-2_8_2;0.13.8 ...

[info] Resolving org.scala-sbt#precompiled-2_9_2;0.13.8 ...

[info] Resolving org.scala-sbt#precompiled-2_9_3;0.13.8 ...

[trace] Stack trace suppressed: run 'last *:update' for the full output.

[trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the full output.

[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.hadoop#hadoop-streaming_2.11;2.6.0: not found

[error] (*:ssExtractDependencies) sbt.ResolveException: unresolved dependency: org.apache.hadoop#hadoop-streaming_2.11;2.6.0: not found

[error] Total time: 13 s, completed Sep 5, 2016 2:05:47 AM

从错误消息看来,sbt 出于某种原因正在寻找 hadoop-streaming_2.11,但我不知道这个 2.11 来自哪里。我对 Scala 和 sbt 还很陌生,所以我猜我在某处犯了一些愚蠢的错字

【问题讨论】:

    标签: scala hadoop apache-spark sbt


    【解决方案1】:

    “如果你使用 groupID %% artifactID % revision 而不是 groupID % artifactID % revision (区别是 groupID 后面的双 %%),sbt 会将你项目的 Scala 版本添加到工件名称中。”

    来自SBT manual

    所以你应该在这里使用 %。

    【讨论】:

    • 那么为什么这不会破坏我的其他依赖项?他们真的都附加了一个scala版本吗?这看起来很奇怪。
    • 你可以访问mvnrepository.com,会找到你的其他依赖,格式为xx_2.10。
    • Hadoop-streaming 是一个 java 库,不依赖于 scala。然而,需要为您正在使用的 scala 版本编译 Scala 库(2.11.8 的 2.11 部分)
    猜你喜欢
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2017-03-04
    • 1970-01-01
    • 2015-05-25
    • 2018-06-03
    • 1970-01-01
    相关资源
    最近更新 更多