为什么运行时依赖在 gradle 中不起作用？答案

【问题标题】：Why runtime dependency doesn't work in gradle?为什么运行时依赖在 gradle 中不起作用？
【发布时间】：2015-05-21 03:31:42
【问题描述】：

我有一个简单的 spark 应用程序和 gradle 2.3，spark 指南说 spark 库不需要捆绑，所以我在 build.gradle 中使用“运行时”依赖项，如下所示：

apply plugin: 'java'
apply plugin: 'idea'
apply plugin: 'scala'
apply plugin: 'maven'


repositories {

    mavenCentral()
}

dependencies {
    compile 'org.scala-lang:scala-library:2.10.5'
    runtime 'org.apache.spark:spark-core_2.10:1.3.1'
    runtime 'org.apache.spark:spark-streaming_2.10:1.3.1'
    compile 'com.datastax.spark:spark-cassandra-connector_2.10:1.2.0-rc3'

    testCompile group: 'junit', name: 'junit', version: '4.11'
}

但是，当我运行“类”任务时，我遇到了错误。这意味着编译找不到罐子。我也试过'provided'和'providedCompile'，结果是“找不到方法provided()/providedCompile()”

[ant:scalac] /Users/grant/programming/ideaprojects/scalaTest/src/main/scala/com/grant/cassandra/DemoApp.scala:3: error: object Logging is not a member of package org.apache.spark
[ant:scalac] import org.apache.spark.{Logging, SparkContext, SparkConf}
[ant:scalac]        ^
[ant:scalac] /Users/grant/programming/ideaprojects/scalaTest/src/main/scala/com/grant/cassandra/DemoApp.scala:5: error: not found: type Logging
[ant:scalac] trait DemoApp extends App with Logging {
[ant:scalac]                                ^
[ant:scalac] /Users/grant/programming/ideaprojects/scalaTest/src/main/scala/com/grant/cassandra/DemoApp.scala:14: error: not found: type SparkConf
[ant:scalac]   val conf = new SparkConf(true)
[ant:scalac]                  ^
[ant:scalac] /Users/grant/programming/ideaprojects/scalaTest/src/main/scala/com/grant/cassandra/DemoApp.scala:21: error: not found: type SparkContext
[ant:scalac]   lazy val sc = new SparkContext(conf)
[ant:scalac]                     ^
[ant:scalac] /Users/grant/programming/ideaprojects/scalaTest/src/main/scala/com/grant/cassandra/WordCountDemo.scala:3: error: object SparkContext is not a member of package org.apache.spark
[ant:scalac] import org.apache.spark.SparkContext._
[ant:scalac]                         ^
[ant:scalac] error: bad symbolic reference. A signature in CassandraConnector.class refers to type Logging
[ant:scalac] in package org.apache.spark which is not available.
[ant:scalac] It may be completely missing from the current classpath, or the version on
[ant:scalac] the classpath might be incompatible with the version used when compiling CassandraConnector.class.
[ant:scalac] error: bad symbolic reference. A signature in CassandraConnector.class refers to type SparkConf
[ant:scalac] in package org.apache.spark which is not available.
[ant:scalac] It may be completely missing from the current classpath, or the version on
[ant:scalac] the classpath might be incompatible with the version used when compiling CassandraConnector.class.
[ant:scalac] /Users/grant/programming/ideaprojects/scalaTest/src/main/scala/com/grant/spark/PairRDDTest.scala:3: error: object SparkConf is not a member of package org.apache.spark
[ant:scalac] import org.apache.spark.SparkConf
[ant:scalac]        ^
[ant:scalac] /Users/grant/programming/ideaprojects/scalaTest/src/main/scala/com/grant/spark/PairRDDTest.scala:4: error: object SparkContext is not a member of package org.apache.spark
[ant:scalac] import org.apache.spark.SparkContext
[ant:scalac]        ^
[ant:scalac] /Users/grant/programming/ideaprojects/scalaTest/src/main/scala/com/grant/spark/PairRDDTest.scala:5: error: object rdd is not a member of package org.apache.spark
[ant:scalac] import org.apache.spark.rdd.PairRDDFunctions
[ant:scalac]                         ^
[ant:scalac] /Users/grant/programming/ideaprojects/scalaTest/src/main/scala/com/grant/spark/PairRDDTest.scala:10: error: not found: type SparkConf
[ant:scalac]         val conf = new SparkConf
[ant:scalac]                        ^
[ant:scalac] /Users/grant/programming/ideaprojects/scalaTest/src/main/scala/com/grant/spark/PairRDDTest.scala:14: error: not found: type SparkContext
[ant:scalac]         val sc = new SparkContext(conf)
[ant:scalac]                      ^
[ant:scalac] /Users/grant/programming/ideaprojects/scalaTest/src/main/scala/com/grant/spark/PairRDDTest.scala:18: error: not found: type PairRDDFunctions
[ant:scalac]         val func = new PairRDDFunctions(rdd)
[ant:scalac]                        ^
[ant:scalac] /Users/grant/programming/ideaprojects/scalaTest/src/main/scala/com/grant/spark/SparkMain.scala:3: error: object SparkConf is not a member of package org.apache.spark
[ant:scalac] import org.apache.spark.{SparkConf, SparkContext}
[ant:scalac]        ^
[ant:scalac] /Users/grant/programming/ideaprojects/scalaTest/src/main/scala/com/grant/spark/SparkMain.scala:7: error: not found: type SparkConf
[ant:scalac]       val conf = new SparkConf
[ant:scalac]                      ^
[ant:scalac] /Users/grant/programming/ideaprojects/scalaTest/src/main/scala/com/grant/spark/SparkMain.scala:11: error: not found: type SparkContext
[ant:scalac]       val sc = new SparkContext(conf)
[ant:scalac]                    ^
[ant:scalac] 16 errors found

【问题讨论】：

标签： gradle apache-spark

【解决方案1】：

provided/providedCompile 配置在某些插件或您的构建脚本创建它们之前不存在。您可以使用来自nebula-plugins 的插件，也可以按您的方式进行操作

configurations {
  provided
}
sourceSets {
  main {
    compileClasspath += [configurations.provided]
  }
}
dependencies {
  provided 'org.apache.hadoop:hadoop-core:2.5.0-mr1-cdh5.3.0'
  compile ...
  testCompile 'org.apache.mrunit:mrunit:1.0.0'
}
jar {
  doFirst {
    into('lib') { from configurations.runtime }
  }
}
idea {
  module {
    scopes.PROVIDED.plus += [configurations.provided]
  }
}

此示例还将库从 compile 添加到 JAR 的 lib 文件夹中，以便将它们作为 Hadoop 作业运行。对于 Spark，您可能需要创建带阴影的 JAR 或“uber” JAR。这将添加编译依赖项（未提供）。

【讨论】：

感谢您的回答。我求助于 SBT，它有组装插件来帮助构建一个“超级”jar。然而，其他问题发生了，叹息
Gradle 有一些#fatjar 插件。您可以在插件门户plugins.gradle.org/search?term=fatjar 上搜索它们

【解决方案2】：

查看 gradle 文档 (https://docs.gradle.org/current/userguide/artifact_dependencies_tutorial.html) 我认为运行时意味着它仅在运行时需要。我相信如果您将其切换为已编译，它也将包含在运行时中（措辞有点令人困惑，但如果您查看 testCompile 的措辞，它要么意味着测试所包含的所有内容都包含在生产代码中 - 这将非常非标准 - 或者这意味着默认情况下 testCompile 包含所有编译元素，就像运行时一样）。

【讨论】：