如何使用 Apache Spark ML 库对随机森林执行网格搜索

Question

我想在 Apache Spark 中对我的随机森林模型执行网格搜索。但我找不到这样做的例子。有没有关于样本数据的示例，我可以使用 Grid Search 进行超参数调整？

Answer 1

from pyspark.ml import Pipeline
from pyspark.ml.classification import RandomForestClassifier
from pyspark.ml.evaluation import BinaryClassificationEvaluator
from pyspark.ml.tuning import CrossValidator, ParamGridBuilder


rf = RandomForestClassifier(labelCol="indexedLabel", featuresCol="indexedFeatures", numTrees=10)
pipeline = Pipeline(stages=[rf])
paramGrid = ParamGridBuilder().addGrid(rf.numTrees, [10, 30]).build()

crossval = CrossValidator(estimator=pipeline,
                          estimatorParamMaps=paramGrid,
                          evaluator=BinaryClassificationEvaluator(),
                          numFolds=2) 

cvModel = crossval.fit(training_df)

超参数和网格在 addGrid 方法中定义