使用保存的 Spark mllib 决策树二元分类模型对新数据进行预测答案

【问题标题】：Use saved Spark mllib decision tree binary classification model to predict on new data使用保存的 Spark mllib 决策树二元分类模型对新数据进行预测
【发布时间】：2019-04-03 22:57:10
【问题描述】：

我使用的是 Spark 2.2.0 版和 Scala 2.11.8 版。我使用以下代码创建并保存了一个决策树二元分类模型：

package...
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.mllib.tree.DecisionTree
import org.apache.spark.mllib.tree.model.DecisionTreeModel
import org.apache.spark.mllib.util.MLUtils
import org.apache.spark.sql.SparkSession


object DecisionTreeClassification {

def main(args: Array[String]): Unit = {

val sparkSession = SparkSession.builder
  .master("local[*]")
  .appName("Decision Tree")
  .getOrCreate()
// Load and parse the data file.
val data = MLUtils.loadLibSVMFile(sparkSession.sparkContext, "path/to/file/xyz.txt")
// Split the data into training and test sets (20% held out for testing)
val splits = data.randomSplit(Array(0.8, 0.2))
val (trainingData, testData) = (splits(0), splits(1))

// Train a DecisionTree model.
//  Empty categoricalFeaturesInfo indicates all features are continuous.
val numClasses = 2
val categoricalFeaturesInfo = Map[Int, Int]()
val impurity = "gini"
val maxDepth = 5
val maxBins = 32

val model = DecisionTree.trainClassifier(trainingData, numClasses, categoricalFeaturesInfo,
  impurity, maxDepth, maxBins)

// Evaluate model on test instances and compute test error
val labelAndPreds = testData.map { point =>
  val prediction = model.predict(point.features)
  (point.label, prediction) 
}
val testErr = labelAndPreds.filter(r => r._1 != r._2).count().toDouble / testData.count()
println(s"Test Error = $testErr")
println(s"Learned classification tree model:\n ${model.toDebugString}")

// Save and load model
model.save(sparkSession.sparkContext, "target/tmp/myDecisionTreeClassificationModel")
val sameModel = DecisionTreeModel.load(sparkSession.sparkContext, "target/tmp/myDecisionTreeClassificationModel")
// $example off$

sparkSession.sparkContext.stop()
}  
}

现在，我想使用这个保存的模型预测新数据的标签（0 或 1）。我是 Spark 新手，谁能告诉我该怎么做？

【问题讨论】：

标签： scala apache-spark apache-spark-mllib

【解决方案1】：

我找到了这个问题的答案，所以我想如果有人正在寻找类似问题的答案，我应该分享它

要对新数据进行预测，只需在停止 spark 会话之前添加几行：

 val newData = MLUtils.loadLibSVMFile(sparkSession.sparkContext, "path/to/file/abc.txt")

 val newDataPredictions = newData.map 
    { point =>
      val newPrediction = model.predict(point.features)
      (point.label, newPrediction)
    }
    newDataPredictions.foreach(f => println("Predicted label", f._2))

【讨论】：