【问题标题】:Spark 1.2.1 fails to compile the assembly projectSpark 1.2.1 编译组装项目失败
【发布时间】:2015-04-21 14:39:48
【问题描述】:

刚下载Spark 1.2.1,在汇编项目中编译失败,报错:

The requested profile "hadoop-2.6" could not be activated because it does not exist.
[ERROR] Failed to execute goal on project spark-assembly_2.10: Could not resolve dependencies for project org.apache.spark:spark-assembly_2.10:pom:1.2.1: Failure to find org.apache.spark:spark-hive-thriftserver_2.11:jar:1.2.1

这里是环境:

  1. Hadoop 2.6.0 -- 通过 brew 安装
  2. Hive 0.14.0 -- 通过 brew 安装
  3. Spark 1.2.1 作为 tgz 下载,因为 Brew 抱怨 Beeline 是一个常见的二进制文件
  4. Scala 2.11 -- 通过 brew 安装
  5. sbt 0.13.7 -- 通过 brew 安装

我正在使用以下参数编译火花分布: mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -Phive -Phive-thriftserver -Dscala-2.11 -DskipTests clean package

Reactor Summary:
[INFO] 
[INFO] Spark Project Parent POM .......................... SUCCESS [  3.525 s]
[INFO] Spark Project Core ................................ SUCCESS [02:56 min]
[INFO] Spark Project Bagel ............................... SUCCESS [ 17.102 s]
[INFO] Spark Project GraphX .............................. SUCCESS [ 45.246 s]
[INFO] Spark Project ML Library .......................... SUCCESS [01:22 min]
[INFO] Spark Project Tools ............................... SUCCESS [ 11.457 s]
[INFO] Spark Project Networking .......................... SUCCESS [  6.121 s]
[INFO] Spark Project Shuffle Streaming Service ........... SUCCESS [  5.642 s]
[INFO] Spark Project Streaming ........................... SUCCESS [01:19 min]
[INFO] Spark Project Catalyst ............................ SUCCESS [01:27 min]
[INFO] Spark Project SQL ................................. SUCCESS [01:19 min]
[INFO] Spark Project Hive ................................ SUCCESS [01:20 min]
[INFO] Spark Project Assembly ............................ FAILURE [  0.396 s]
[INFO] Spark Project External Twitter .................... SKIPPED
[INFO] Spark Project External Flume ...................... SKIPPED
[INFO] Spark Project External Flume Sink ................. SKIPPED
[INFO] Spark Project External MQTT ....................... SKIPPED
[INFO] Spark Project External ZeroMQ ..................... SKIPPED
[INFO] Spark Project Examples ............................ SKIPPED
[INFO] Spark Project REPL ................................ SKIPPED
[INFO] Spark Project YARN Parent POM ..................... SKIPPED
[INFO] Spark Project YARN Stable API ..................... SKIPPED
[INFO] Spark Project YARN Shuffle Service ................ SKIPPED
[INFO] Spark Project Hive Thrift Server .................. SKIPPED

我错过了什么吗?我不想使用 brew 来安装 Apache-Spark,因为我必须取消链接 hive,我也想使用它。

谢谢!

【问题讨论】:

    标签: hadoop apache-spark hive apache-spark-1.2


    【解决方案1】:

    尝试使用hadoop-2.4 配置文件,但将其他 Hadoop 版本保持为 2.6.0:

    mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 -Phive -Phive-thriftserver -Dscala-2.11 -DskipTests clean package
    

    来源:https://mail-archives.apache.org/mod_mbox/spark-user/201412.mbox/%3CCAMAsSdLVO73-YRa8-k_SXN5kkDKML-nKpPhgnQ3TgxUVnCp=bg@mail.gmail.com%3E

    【讨论】:

    • 谢谢,但没用。同样的错误,问题似乎在于缺少依赖项。找不到 org.apache.spark:spark-hive-thriftserver_2.11:jar:1.2.1
    猜你喜欢
    • 2014-04-23
    • 1970-01-01
    • 2019-08-02
    • 2017-07-29
    • 2017-07-25
    • 1970-01-01
    • 2019-02-12
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多