【问题标题】:Spark ML insert/fit custom OneHotEncoder into a PipelineSpark ML 将自定义 OneHotEncoder 插入/拟合到管道中
【发布时间】:2020-06-10 21:48:26
【问题描述】:

假设我在应用常规 OneHotEncoder 的数据框中有一些特征/列,以及我需要应用我的自定义 OneHotEncoder 的一个(let,n-th)列。然后我需要使用 VectorAssembler 来组装这些特征,并放入 Pipeline,最后拟合我的 trainData 并从我的 testData 中获取预测,例如:

val sIndexer1 = new StringIndexer().setInputCol("my_feature1").setOutputCol("indexed_feature1")
// ... let, n-1 such sIndexers for n-1 features
val featureEncoder = new OneHotEncoderEstimator().setInputCols(Array(sIndexer1.getOutputCol), ...).
      setOutputCols(Array("encoded_feature1", ... ))

// **need to insert output from my custom OneHotEncoder function (please see below)**
// (which takes the n-th feature as input) in a way that matches the VectorAssembler below

val vectorAssembler = new VectorAssembler().setInputCols(featureEncoder.getOutputCols + ???).
      setOutputCol("assembled_features")

...

val pipeline = new Pipeline().setStages(Array(sIndexer1, ...,featureEncoder, vectorAssembler, myClassifier))
val model = pipeline.fit(trainData)
val predictions = model.transform(testData)

如何修改 vectorAssembler 的构建,以便它可以接收来自自定义 OneHotEncoder 的输出? 问题是我想要的 oheEncodingTopN() 不能/不应引用“实际”数据帧,因为它将是管道的一部分(应用于 trainData/testData)。

注意:

我测试了自定义 OneHotEncoder(参见 link)在例如训练数据。基本上,oheEncodingTopN 在输入列上应用 OneHotEncoding,但仅针对前 N 个频繁值(例如 N = 50),并将所有其余不频繁值放在一个虚拟列中(例如,“默认”) ,例如:

val oheEncoded = oheEncodingTopN(df, "my_featureN", 50)

import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.functions.{col, lit, when}
import org.apache.spark.sql.Column


def flip(col: Column): Column = when(col === 1, lit(0)).otherwise(lit(1))

def oheEncodingTopN(df: DataFrame, colName: String, n: Int): DataFrame = {
  df.createOrReplaceTempView("data")
  val topNDF = spark.sql(s"select $colName, count(*) as count from data group by $colName order by count desc limit $n")

  val pivotTopNDF = topNDF.
    groupBy(colName).
    pivot(colName).
    count().
    withColumn("default", lit(1))

  val joinedTopNDF = df.join(pivotTopNDF, Seq(colName), "left").drop(colName)

  val oheEncodedDF = joinedTopNDF.
    na.fill(0, joinedTopNDF.columns).
    withColumn("default", flip(col("default")))

   oheEncodedDF

}

【问题讨论】:

    标签: scala apache-spark machine-learning one-hot-encoding


    【解决方案1】:

    我认为最简洁的方法是创建您自己的扩展 spark ML Transformer 的类,这样您就可以像使用任何其他 Transformer(如 OneHotEncoder)一样使用它。你的班级应该是这样的:

    import org.apache.spark.ml.Transformer
    import org.apache.spark.ml.param.Param
    import org.apache.spark.ml.param.ParamMap
    import org.apache.spark.ml.util.Identifiable
    import org.apache.spark.sql.types._
    import org.apache.spark.sql.functions._
    import org.apache.spark.sql.{DataFrame, Dataset, Column}
    
    class OHEncodingTopN(n :Int, override val uid: String) extends Transformer {
      final val inputCol= new Param[String](this, "inputCol", "The input column")
      final val outputCol = new Param[String](this, "outputCol", "The output column")
    
     ; def setInputCol(value: String): this.type = set(inputCol, value)
    
      def setOutputCol(value: String): this.type = set(outputCol, value)
    
      def this(n :Int) = this(n, Identifiable.randomUID("OHEncodingTopN"))
    
      def copy(extra: ParamMap): OHEncodingTopN = {
        defaultCopy(extra)
      }
    
      override def transformSchema(schema: StructType): StructType = {
        // Check that the input type is what you want if needed 
        //     val idx = schema.fieldIndex($(inputCol))
        //     val field = schema.fields(idx)
        //     if (field.dataType != StringType) {
        //       throw new Exception(s"Input type ${field.dataType} did not match input type StringType")
        //     }
        // Add the return field
        schema.add(StructField($(outputCol), IntegerType, false))
      }
      def flip(col: Column): Column = when(col === 1, lit(0)).otherwise(lit(1))
    
      def transform(df: Dataset[_]): DataFrame = {
          df.createOrReplaceTempView("data")
          val colName = $(inputCol)
          val topNDF = df.sparkSession.sql(s"select $colName, count(*) as count from data group by $colName order by count desc limit $n")
    
          val pivotTopNDF = topNDF.
            groupBy(colName).
            pivot(colName).
            count().
            withColumn("default", lit(1))
    
          val joinedTopNDF = df.join(pivotTopNDF, Seq(colName), "left").drop(colName)
    
          val oheEncodedDF = joinedTopNDF.
            na.fill(0, joinedTopNDF.columns).
            withColumn("default", flip(col("default")))
    
           oheEncodedDF
      }
    }
    

    现在在 OHEncodingTopN 对象上,您应该能够调用 .getOuputCol 来执行您想要的操作。祝你好运。

    编辑:我刚刚复制粘贴到转换方法中的方法应稍作修改,以便输出具有 setOutputCol 中给定名称的 Vector 类型列。

    【讨论】:

    • 我试过但失败了..需要在 setOutputCol 中“匹配”名称:我试过:final val outputCol = new Param[String](this, "outputCol", "encoded_TopN") def setOutputCol(value: String): this.type = set(outputCol, value) .. def transform(df: Dataset[_]): DataFrame = { ... // not sure how to get the encodedColumns and return it as a column of type Vector? }
    猜你喜欢
    • 2018-12-18
    • 1970-01-01
    • 2017-08-24
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2016-03-14
    • 1970-01-01
    • 2016-05-23
    相关资源
    最近更新 更多