在 spark 中使用 stanford nlp，错误“找不到类 java.util.function.Function - 继续使用存根。”答案

【问题标题】：using stanford nlp in spark, error " Class java.util.function.Function not found - continuing with a stub."在 spark 中使用 stanford nlp，错误“找不到类 java.util.function.Function - 继续使用存根。”
【发布时间】：2019-06-08 12:17:52
【问题描述】：

我需要在 spark 1.6 中进行一些文本预处理。从Simplest method for text lemmatization in Scala and Spark 得到答案，需要import java.util.Properties。但是通过运行abt编译和组装，我得到了以下错误：

[warn] Class java.util.function.Function not found - continuing with a stub.
[warn] Class java.util.function.Function not found - continuing with a stub.
[warn] Class java.util.function.Function not found - continuing with a stub.
[error] Class java.util.function.Function not found - continuing with a stub.
[error] Class java.util.function.Function not found - continuing with a stub.
[warn] four warnings found
[error] two errors found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 52 s, completed Feb 10, 2016 2:11:12 PM

代码如下：

 // ref https://stackoverflow.com/questions/30222559/simplest-methodfor-text-lemmatization-in-scala-and-spark?rq=1

 def plainTextToLemmas(text: String): Seq[String] = {

 import java.util.Properties

 import edu.stanford.nlp.ling.CoreAnnotations._
 import edu.stanford.nlp.pipeline._

 import scala.collection.JavaConversions._
 import scala.collection.mutable.ArrayBuffer
 //  val stopWords = Set("stopWord")

 val props = new Properties()
 props.put("annotators", "tokenize, ssplit, pos, lemma")
 val pipeline = new StanfordCoreNLP(props)
 val doc = new Annotation(text)
 pipeline.annotate(doc)
 val lemmas = new ArrayBuffer[String]()
 val sentences = doc.get(classOf[SentencesAnnotation])
 for (sentence <- sentences;
     token <- sentence.get(classOf[TokensAnnotation])) {
 val lemma = token.get(classOf[LemmaAnnotation])
 if (lemma.length > 2) {
    lemmas += lemma.toLowerCase
 }
}
   lemmas
}

我的sbt文件如下：

scalaVersion := "2.11.7"

crossScalaVersions := Seq("2.10.5", "2.11.0-M8")

libraryDependencies ++= Seq(
  "org.apache.spark" % "spark-core_2.10" % "1.6.0" % "provided",
  "org.apache.spark" % "spark-mllib_2.10" % "1.6.0" % "provided",
  "org.apache.spark" % "spark-sql_2.10" % "1.6.0" % "provided",
  "com.github.scopt" % "scopt_2.10" % "3.3.0",
 )
    libraryDependencies ++= Seq(
  "edu.stanford.nlp" % "stanford-corenlp" % "3.5.2",
  "edu.stanford.nlp" % "stanford-corenlp" % "3.5.2" classifier "models"
  //   "edu.stanford.nlp" % "stanford-corenlp" % "3.5.2" classifier "models-chinese"
  //   "edu.stanford.nlp" % "stanford-corenlp" % "3.5.2" classifier "models-german"
  //   "edu.stanford.nlp" % "stanford-corenlp" % "3.5.2" classifier "models-spanish"
  //"com.google.code.findbugs" % "jsr305" % "2.0.3"
)

根据该站点的建议，我将java lib版本从1.7更改为1.8，问题仍然存在。

【问题讨论】：

奇怪，无法重新创建它，我想您可以通过将其添加到您的 sbt 依赖项来使用后端端口，看看会发生什么：libraryDependencies += "net.sf.m-m-m" %"mmm-util-backport-java.util.function" %"1.0.1"
感谢@GameOfThrows。我通过将我的 java home 链接到 java 1.8 解决了这个问题。以前，java home 链接到 java 1.7。虽然项目 SDK 链接到 java 1.8，但可能（基于我的观察和推论）sbt compile 命令使用默认的 java home 设置运行。合理吗？

标签： java scala apache-spark nlp

【解决方案1】：

问题是通过将java home设置为java 8解决的。以前我在java home还是7的时候将项目SDK更改为java 8，所以在sbt编译时它无法工作。

【讨论】：