如何将数据框转换为列表（Scala）？

Question

我想将包含 Double 值的 Dataframe 转换为 List，以便我可以使用它进行计算。您有什么建议，以便我可以采用正确的类型 List（即 Double）？

我的做法是这样的：

var newList = myDataFrame.collect().toList 

但它返回一个类型 List[org.apache.spark.sql.Row] 我不知道它到底是什么！

是否可以忘记该步骤，只需将我的 Dataframe 传递到函数中并从中进行计算？ （例如，我想将其第二列的第三个元素与特定的 double 进行比较。是否可以直接从我的 Dataframe 中这样做？）

我必须不惜一切代价了解如何每次都创建正确的类型列表！

编辑：

输入数据框：

+---+---+ 
|_c1|_c2|
+---+---+ 
|0  |0  | 
|8  |2  | 
|9  |1  | 
|2  |9  | 
|2  |4  | 
|4  |6  | 
|3  |5  | 
|5  |3  | 
|5  |9  | 
|0  |1  | 
|8  |9  | 
|1  |0  | 
|3  |4  |
|8  |7  | 
|4  |9  | 
|2  |5  | 
|1  |9  | 
|3  |6  |
+---+---+

转换后的结果：

List((0,0), (8,2), (9,1), (2,9), (2,4), (4,6), (3,5), (5,3), (5,9), (0,1), (8,9), (1,0), (3,4), (8,7), (4,9), (2,5), (1,9), (3,6))

但是 List 中的每个元素都必须是 Double 类型。

Answer 1

你可以将你需要的coulmn转换成Double并转换成RDD和collect它

如果您有无法解析的数据，则可以在将其转换为双精度之前使用 udf 进行清理

val stringToDouble = udf((data: String) => {
  Try (data.toDouble) match {
    case Success(value) => value
    case Failure(exception) => Double.NaN
  }
})

 val df = Seq(
   ("0.000","0"),
   ("0.000008","24"),
   ("9.00000","1"),
   ("-2","xyz"),
   ("2adsfas","1.1.1")
 ).toDF("a", "b")
  .withColumn("a", stringToDouble($"a").cast(DoubleType))
  .withColumn("b", stringToDouble($"b").cast(DoubleType))

在此之后，您将获得输出为

+------+----+
|a     |b   |
+------+----+
|0.0   |0.0 |
|8.0E-6|24.0|
|9.0   |1.0 |
|-2.0  |NaN |
|NaN   |NaN |
+------+----+

获取Array[(Double, Double)]

val result = df.rdd.map(row => (row.getDouble(0), row.getDouble(1))).collect()

结果将是Array[(Double, Double)]

Answer 2

#Convert DataFrame to DataSet using case class & then convert it to list

#It'll return the list of type of your class object.All the variables inside the #class(mapping to fields in your table)will be pre-typeCasted) Then you won't need to #type cast every time.

#Please execute below code to check it-
#Sample to check & verify(scala)-

val wa = Array("one","two","two")
val wr = sc.parallelize(wa,3).map(x=>(x,"x",1))
val wdf = wr.toDF("a","b","c")
case class wc(a:String,b:String,c:Int)
val myList= wds.collect.toList
myList.foreach(x=>println(x))
myList.foreach(x=>println(x.a.getClass,x.b.getClass,x.c.getClass))

Answer 3

myDataFrame.select("_c1", "_c2").collect().map(each => (each.getAs[Double]("_c1"), each.getAs[Double]("_c2"))).toList