使用 Scala 和 Spark 将 Array[Array[String]] 转换为列中的字符串

Question

这是我的数据框：

+--------------------+                          
|    NewsId|             newsArr|            transArr|
+----------+--------------------+--------------------+
|        26|[Republicans, Sto...|[[R, IH0, P, AH1,...|
|        29|[ISIS, Claims, Re...|[[AY1, S, AH0], [...|
|       474|[Concert, for, Tr...|[[K, AA1, N, S, E...|
|       964|[How, a, Fractiou...|[[HH, AW1], [AH0]...|
|      1677|[Review:, ‘Kong:,...|[[n/a], [n/a], [S...|
|      1697|[The, Rice-Size, ...|[[DH, AH0], [n/a]...|
|      1806|[Populists, Appea...|[[P, AA1, Y, AH0,...|
|      1950|[Uber, Board, Sta...|[[Y, UW1, B, ER0]...|
|      2040|[Health, Bill’s, ...|[[HH, EH1, L, TH]...|
|      2214|[Unmasking, the, ...|[[n/a], [DH, AH0]...|

我想将“transArr”列单元格变成这样的字符串：

+--------------------+                          
|    NewsId|             newsArr|      transArr|
+----------+--------------------+--------------+
|        26|[Republicans, Sto...|R IH0 P AH1...|
|        29|[ISIS, Claims, Re...|AY1 S AH0...  |
|       474|[Concert, for, Tr...|K AA1 N S E...|
|       964|[How, a, Fractiou...|HH AW1 AH0... |
|      1677|[Review:, ‘Kong:,...|n/a n/a S...  |
|      1697|[The, Rice-Size, ...|DH AH0 n/a... |
|      1806|[Populists, Appea...|P AA1 Y AH0...|
|      1950|[Uber, Board, Sta...|Y UW1 B ER0...|
|      2040|[Health, Bill’s, ...|HH EH1 L TH...|
|      2214|[Unmasking, the, ...|n/a DH AH0... |

有没有相对简单的解决方案？

Answer 1

使用concat_ws & flatten，检查下面的代码。

scala> df.printSchema
root
 |-- data: array (nullable = true)
 |    |-- element: array (containsNull = true)
 |    |    |-- element: string (containsNull = true)

scala> df
.withColumn(
     "flatten",
     concat_ws(" ",flatten($"data"))
)
.show(false)

+------------+-------+
|data        |flatten|
+------------+-------+
|[[abc, cdf]]|abc cdf|
+------------+-------+

Answer 2

使用concat_ws:

import spark.implicits._
val df: DataFrame = Seq(
  ("a1", Array("2", "3", "5")),
  ("b2", Array("1", "6", "23")),
  ("b1", Array("df", "l2", "14")),
  ("c1", Array("te", "3pa", "gw"))
).toDF("key", "values")
df.show()
val newDF = df.withColumn("values", concat_ws(" ", col("values")))
newDF.show()
newDF.printSchema()

输出：

+---+-------------+
|key|       values|
+---+-------------+
| a1|    [2, 3, 5]|
| b2|   [1, 6, 23]|
| b1| [df, l2, 14]|
| c1|[te, 3pa, gw]|
+---+-------------+

+---+---------+
|key|   values|
+---+---------+
| a1|    2 3 5|
| b2|   1 6 23|
| b1| df l2 14|
| c1|te 3pa gw|
+---+---------+

root
 |-- key: string (nullable = true)
 |-- values: string (nullable = false)