如何构建未发布的 Spark RC 版本并在 sbt 项目中使用它？答案

【问题标题】：How to build unreleased Spark RC version and use it in sbt project?如何构建未发布的 Spark RC 版本并在 sbt 项目中使用它？
【发布时间】：2017-10-28 22:20:22
【问题描述】：

我正在 Spark v2.2.0-rc1 上构建一个 sbt 应用程序。因为，这个版本的 Spark 在 Maven 存储库上不可用，所以我从 Spark 的 GitHub 存储库下载它并尝试在本地发布它，这样我就可以使用 Spark v2.2.0-rc1 编译我的应用程序。但是，在执行sbt publishLocal 时，我遇到了以下异常：

$ sbt publishLocal

...
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.postgresql#postgresql;9.4.1207.jre7: several problems occurred while resolving dependency: org.postgresql#postgresql;9.4.1207.jre7 {test=[default(compile)]}:
[warn]  org.postgresql#postgresql;9.4.1207.jre7!postgresql.pom(pom.original) origin location must be absolute: file:/home/user/.m2/repository/org/postgresql/postgresql/9.4.1207.jre7/postgresql-9.4.1207.jre7.pom
[warn]  org.postgresql#postgresql;9.4.1207.jre7!postgresql.pom(pom.original) origin location must be absolute: file:/home/user/.m2/repository/org/postgresql/postgresql/9.4.1207.jre7/postgresql-9.4.1207.jre7.pom
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[info] :: delivering :: org.apache.spark#spark-streaming-flume-assembly_2.11;2.2.0 :: 2.2.0 :: release :: Sun May 28 14:05:05 GMT 2017
[warn] 
[warn]  Note: Unresolved dependencies path:
[warn]      org.postgresql:postgresql:9.4.1207.jre7 ((com.typesafe.sbt.pom.MavenHelper) MavenHelper.scala#L76)
[warn]        +- org.apache.spark:spark-sql_2.11:2.2.0
[info]  delivering ivy file to /home/user/Softwares/spark-2.2.0-rc1/external/flume-assembly/target/scala-2.11/ivy-2.2.0.xml

...

[info]  published ivy to /home/user/.ivy2/local/org.apache.spark/spark-catalyst_2.11/2.2.0/ivys/ivy.xml
sbt.ResolveException: unresolved dependency: org.postgresql#postgresql;9.4.1207.jre7: several problems occurred while resolving dependency: org.postgresql#postgresql;9.4.1207.jre7 {test=[default(compile)]}:
    org.postgresql#postgresql;9.4.1207.jre7!postgresql.pom(pom.original) origin location must be absolute: file:/home/user/.m2/repository/org/postgresql/postgresql/9.4.1207.jre7/postgresql-9.4.1207.jre7.pom
    org.postgresql#postgresql;9.4.1207.jre7!postgresql.pom(pom.original) origin location must be absolute: file:/home/user/.m2/repository/org/postgresql/postgresql/9.4.1207.jre7/postgresql-9.4.1207.jre7.pom

    at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:313)
    at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:191)
    at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:168)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156)
    at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:133)
    at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:57)
    at sbt.IvySbt$$anon$4.call(Ivy.scala:65)
    at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
    at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
    at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
    at xsbt.boot.Using$.withResource(Using.scala:10)
    at xsbt.boot.Using$.apply(Using.scala:9)
    at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
    at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
    at xsbt.boot.Locks$.apply0(Locks.scala:31)
    at xsbt.boot.Locks$.apply(Locks.scala:28)
    at sbt.IvySbt.withDefaultLogger(Ivy.scala:65)
    at sbt.IvySbt.withIvy(Ivy.scala:128)
    at sbt.IvySbt.withIvy(Ivy.scala:125)
    at sbt.IvySbt$Module.withModule(Ivy.scala:156)
    at sbt.IvyActions$.updateEither(IvyActions.scala:168)
    at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1481)
    at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1477)
    at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$121.apply(Defaults.scala:1512)
    at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$121.apply(Defaults.scala:1510)
    at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:37)
    at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1515)
    at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1509)
    at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:60)
    at sbt.Classpaths$.cachedUpdate(Defaults.scala:1532)
    at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1459)
    at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1411)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
    at sbt.std.Transform$$anon$4.work(System.scala:63)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.Execute.work(Execute.scala:237)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
[error] (sql/*:update) sbt.ResolveException: unresolved dependency: org.postgresql#postgresql;9.4.1207.jre7: several problems occurred while resolving dependency: org.postgresql#postgresql;9.4.1207.jre7 {test=[default(compile)]}:
[error]     org.postgresql#postgresql;9.4.1207.jre7!postgresql.pom(pom.original) origin location must be absolute: file:/home/user/.m2/repository/org/postgresql/postgresql/9.4.1207.jre7/postgresql-9.4.1207.jre7.pom
[error]     org.postgresql#postgresql;9.4.1207.jre7!postgresql.pom(pom.original) origin location must be absolute: file:/home/user/.m2/repository/org/postgresql/postgresql/9.4.1207.jre7/postgresql-9.4.1207.jre7.pom
[error] Total time: 240 s, completed May 28, 2017 2:08:01 PM

但是，当我运行mvn install -DskipTests=true 时，它运行得非常好，但sbt publishLocal 给出了异常。因为，我必须使用 sbt 构建我的应用程序，我希望 spark v2.2.0 位于 .ivy2 文件夹而不是 .m2 文件夹中。

谁能帮我解决这个问题？

【问题讨论】：

标签： maven apache-spark sbt

【解决方案1】：

使用Maven构建Spark（官方方式）如下：

./build/mvn -Phadoop-2.7,yarn,mesos,hive,hive-thriftserver -DskipTests clean install

一旦您在 Maven 存储库中获得了 jar，请在 build.sbt 中定义一个解析器以使用 Maven 的本地存储库。

resolvers += Resolver.mavenLocal

参见 sbt 官方文档中的Resolvers。

这并不能回答您在使用 PostgreSQL JDBC 驱动程序时遇到的问题，但我相信这不太需要。

【讨论】：