Scala Oracle JDBC

Question

我在 Scala 中收到 Module not found 错误。我正在尝试与 Oracle 建立 jdbc 连接，加入两个表，然后将其打印出来。

我的 scala 文件是

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.sql.SQLContext

object sparkJDBC {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("Simple      
        Application").setMaster("local[2]").set("spark.executor.memory","1g")
    val sc = new SparkContext(conf)
    var sqlContext = new SQLContext(sc)
    val chrttype = sqlContext.load("jdbc", 
      Map("url" -> "jdbc:oracle:thin:gductv1/gductv1@//localhost:1521/XE",
      "dbtable" -> "chrt_typ"))
    val clntlvl1  = sqlContext.load("jdbc", 
      Map("url" -> "jdbc:oracle:thin:gductv1/gductv1@//localhost:1521/XE", 
      "dbtable" -> "clnt_lvl1"))
    val join2 =  
      chrttyp.join(clntlvl1,chrttyp.col("chrt_typ_key")===clntlvl1("lvl1_key"))
    join2.foreach(println)
    join2.printSchema()
    }
}

我的 build.sbt 文件是

   name := "sparkJDBC"
   version := "0.1"
   scalaVersion := "2.11.7"

   libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" 
   libraryDependencies += "org.apache.tika" % "tika-core" % "1.11"
   libraryDependencies += "org.apache.tika" % "tika-parsers" % "1.11"
   libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.7.1" 
   libraryDependencies += "org.apache.spark" % "spark-sql" % "1.0.0"

错误文件是

[warn]  module not found: org.apache.spark#spark-sql;1.0.0
[warn] ==== local: tried
[warn]   C:\Users\.ivy2\local\org.apache.spark\spark-sql\1.0.0\ivys\ivy.xml
[warn] ==== public: tried
[warn]   https://repo1.maven.org/maven2/org/apache/spark/spark-sql/1.0.0/spark-sql-1.0.0.pom
[info] Resolving jline#jline;2.12.1 ...
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.apache.spark#spark-sql;1.0.0: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::

[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-sql;1.0.0: not found

请帮我弄清楚是什么原因造成的。

Answer 1

spark-sql 是一个 Scala 库，类似于 spark-core，因此您需要在组名和工件名称之间使用 %% 以同样的方式。在它们之间使用 % 仅用于 Java 库。请参阅http://www.scala-sbt.org/0.13/docs/Cross-Build.html 了解您需要%% 的原因。

Answer 2

为了确保有正确的依赖关系，您可以使用 mvnrepository 之类的网站：https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.10/1.0.0

libraryDependencies += "org.apache.spark" % "spark-sql_2.10" % "1.0.0"