【问题标题】:Creating a Random Feature Array in Spark DataFrames在 Spark DataFrames 中创建随机特征数组
【发布时间】:2018-08-22 07:32:50
【问题描述】:

在创建ALS 模型时,我们可以提取一个userFactors DataFrame 和一个itemFactors DataFrame。这些 DataFrame 包含一个带有 Array 的列。

我想生成一些随机数据并将其合并到userFactors DataFrame。

这是我的代码:

 val df1: DataFrame  = Seq((123, 456, 4.0), (123, 789, 5.0), (234, 456, 4.5), (234, 789, 1.0)).toDF("user", "item", "rating")
val model1 = (new ALS()
 .setImplicitPrefs(true)
 .fit(df1))

val iF = model1.itemFactors
val uF = model1.userFactors

然后我使用 VectorAssembler 和这个函数创建一个随机 DataFrame:

def makeNew(df: DataFrame, rank: Int): DataFrame = {
    var df_dummy = df
    var i: Int = 0
    var inputCols: Array[String] = Array()
    for (i <- 0 to rank) {
       df_dummy = df_dummy.withColumn("feature".concat(i.toString), rand())
       inputCols = inputCols :+ "feature".concat(i.toString)
      }
    val assembler = new VectorAssembler()
      .setInputCols(inputCols)
      .setOutputCol("userFeatures")
    val output = assembler.transform(df_dummy)
    output.select("user", "userFeatures")
  }

然后我使用新的用户 ID 创建 DataFrame 并添加随机向量和偏差:

val usersDf: DataFrame = Seq(567), (678)).toDF("user")
var usersFactorsNew: DataFrame = makeNew(usersDf, 20)

当我合并两个 DataFrame 时,问题就出现了。

usersFactorsNew.union(uF) 产生错误:

 org.apache.spark.sql.AnalysisException: Union can only be performed on tables with the compatible column types. struct<type:tinyint,size:int,indices:array<int>,values:array<double>> <> array<float> at the second column of the second table;;

如果我打印架构,uF DataFrame 具有 Array[Float] 类型的特征向量,usersFactorsNew DataFrame 作为Vector 类型的特征向量。

我的问题是如何将Vector 的类型更改为数组以执行联合。

我尝试写这个udf,但收效甚微:

val toArr: org.apache.spark.ml.linalg.Vector => Array[Double] = _.toArray
val toArrUdf = udf(toArr)

也许VectorAssembler 不是此任务的最佳选择。但是,目前,这是我找到的唯一选择。我很想得到一些更好的建议。

【问题讨论】:

    标签: arrays scala apache-spark dataframe vector


    【解决方案1】:

    你所有的过程都是正确的。甚至udf 功能也能成功运行。您需要做的就是将makeNew函数的最后一部分更改为

    def makeNew(df: DataFrame, rank: Int): DataFrame = {
      var df_dummy = df
      var i: Int = 0
      var inputCols: Array[String] = Array()
      for (i <- 0 to rank) {
        df_dummy = df_dummy.withColumn("feature".concat(i.toString), rand())
        inputCols = inputCols :+ "feature".concat(i.toString)
      }
      val assembler = new VectorAssembler()
        .setInputCols(inputCols)
        .setOutputCol("userFeatures")
      val output = assembler.transform(df_dummy)
      output.select(col("id"), toArrUdf(col("userFeatures")).as("features"))
    }
    

    你应该是完美的,这样当你这样做时(我创建了带有 id 列的 userDf 和 不是用户列)

    val usersDf: DataFrame = Seq((567), (678)).toDF("id")
    var usersFactorsNew: DataFrame = makeNew(usersDf, 20)
    usersFactorsNew.union(uF).show(false)
    

    你应该得到

    +---+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    |id |features                                                                                                                                                                                                                                                                                                                                                                                                                                  |
    +---+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    |567|[0.8259185719733708, 0.327713892339658, 0.049547223031371046, 0.056661808506210054, 0.5846626163454274, 0.038497936270104005, 0.8970865088803417, 0.8840660648882804, 0.837866669938156, 0.9395263094918058, 0.09179528484355126, 0.4915430644129799, 0.11083447052043116, 0.5122858182953718, 0.4302683812966408, 0.3862741815833828, 0.6189322403095068, 0.3000371006293433, 0.09331299668168902, 0.7421838728601371, 0.855867963988993]|
    |678|[0.7686514248005568, 0.5473580740023187, 0.072945344124282, 0.36648594574355287, 0.9780202082328863, 0.5289221651923784, 0.3719451099963028, 0.2824660794505932, 0.4873197501260199, 0.9364676464120849, 0.011539929543513794, 0.5240615794930654, 0.6282546154521298, 0.995256022569878, 0.6659179561266975, 0.8990775317754092, 0.08650071017556926, 0.5190186149992805, 0.056345335742325475, 0.6465357505620791, 0.17913532817943245] |
    |123|[0.04177388548851013, 0.26762014627456665, -0.19617630541324615, 0.34298020601272583, 0.19632814824581146, -0.2748605012893677, 0.07724890112876892, 0.4277132749557495, 0.1927199512720108, -0.40271613001823425]                                                                                                                                                                                                                        |
    |234|[0.04139673709869385, 0.26520395278930664, -0.19440513849258423, 0.3398836553096771, 0.1945556253194809, -0.27237895131111145, 0.07655145972967148, 0.42385169863700867, 0.19098000228405, -0.39908021688461304]                                                                                                                                                                                                                          |
    +---+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    

    【讨论】:

      【解决方案2】:

      您可以直接使用UDF,而不是创建一个虚拟数据框并使用VectorAssembler 生成随机特征向量。 ALS 模型中的 userFactors 将返回 Array[Float],因此 UDF 的输出应该匹配。

      val createRandomArray = udf((rank: Int) => {
        Array.fill(rank)(Random.nextFloat)
      })
      

      请注意,这将给出区间 [0.0, 1.0] 中的数字(与问题中使用的 rand() 相同),如果需要其他数字,请根据需要进行修改。

      使用等级 3 和 userDf:

      val usersFactorsNew = usersDf.withColumn("userFeatures", createRandomArray(lit(3)))
      

      会给出如下的数据框(当然是随机特征值)

      +----+----------------------------------------------------------+
      |user|userFeatures                                              |
      +----+----------------------------------------------------------+
      |567 |[0.6866711267486822,0.7257031656127676,0.983562255688249] |
      |678 |[0.7013908820314967,0.41029552817665327,0.554591149586789]|
      +----+----------------------------------------------------------+
      

      现在应该可以将此数据帧与uF 数据帧连接起来。


      UDF 不起作用的原因应该是因为它是一个Array[Double] while you need anArray[Float]for theunion. It should be possible to fix with amap(_.toFloat)`。

      val toArr: org.apache.spark.ml.linalg.Vector => Array[Float] = _.toArray.map(_.toFloat)
      val toArrUdf = udf(toArr)
      

      【讨论】:

        猜你喜欢
        • 2018-02-11
        • 1970-01-01
        • 1970-01-01
        • 2016-02-25
        • 1970-01-01
        • 2016-04-09
        • 2019-07-25
        • 1970-01-01
        • 2019-10-20
        相关资源
        最近更新 更多