如何在 PySpark 中将 Vector 类型的列转换为数组/字符串类型？

Question

在 pyspark 中考虑以下数据框：

+-----------+
|      Col A|
+-----------+
| [0.5, 0.6]|                  
| [0.7, 0.8]|                   
| [1.1, 1.5]|                                 
+-----------+

Col A 的类型是向量，我怎样才能创建一个新列，它的值是 Col A 但类型是数组或字符串？

df:

+-----------+-----------+
|Col A      |new_column |
+-----------+-----------+
| [0.5, 0.6]|  0.5, 0.6 |               
| [0.7, 0.8]|  0.7, 0.8 |            
| [1.1, 1.5]|  1.1, 1.5 |                         
+-----------+-----------+

提前致谢！

Answer 1

如果您只想将 Vector 转换为 Array[Double]，使用 UDF 非常简单：

import org.apache.spark.ml.linalg.DenseVector
val toArr: Any => Array[Double] = _.asInstanceOf[DenseVector].toArray
val toArrUdf = udf(toArr)
val dataWithFeaturesArr = dataWithFeatures.withColumn("A_arr",toArrUdf('COl A'))

Answer 2

可能的解决方案是：

scala> output.show
+---+---------+
| id|vectorCol|
+---+---------+
|  0|[1.2,1.3]|
|  1|[2.2,2.3]|
|  2|[3.2,3.3]|
+---+---------+


scala> output.printSchema
root
 |-- id: integer (nullable = false)
 |-- vectorCol: vector (nullable = true)


scala> import org.apache.spark.ml.linalg.DenseVector
import org.apache.spark.ml.linalg.DenseVector

scala> val toArr: Any => Array[Double] = _.asInstanceOf[DenseVector].toArray
toArr: Any => Array[Double] = <function1>

scala> val toArrUdf = udf(toArr)
toArrUdf: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(<function1>,ArrayType(DoubleType,false),None)

scala> val df1 = output.withColumn("features_arr",toArrUdf('vectorCol))

scala> df1.show
+---+---------+------------+
| id|vectorCol|features_arr|
+---+---------+------------+
|  0|[1.2,1.3]|  [1.2, 1.3]|
|  1|[2.2,2.3]|  [2.2, 2.3]|
|  2|[3.2,3.3]|  [3.2, 3.3]|
+---+---------+------------+

scala> df1.printSchema
root
 |-- id: integer (nullable = false)
 |-- vectorCol: vector (nullable = true)
 |-- features_arr: array (nullable = true)
 |    |-- element: double (containsNull = false)

在link. 中可以看到 pyspark 中的一个可能实现

如果有帮助请告诉我！！