在单词列表pyspark中过滤列值的行

Question

我正在使用 pyspark 数据框。我有一个专栏words（array<string>）如下图：

+---+--------------------------------------------------------------------------------+
| id|                                                                           words|                      
----+--------------------------------------------------------------------------------+
|012|[content, type, multipart, alternative, boundaries, nextpart, nextpart, drama,..|
|013|[received, from, am5eur02ht120, eop, eur02, prod, protection, outlook by, pro...|
|014|[data, care, much, important, information, summer, care, send, faraway, forget..|

我还有一个单词列表：

list = ["protection", "content", "received"]

我想过滤列表中具有任何值的行。

预期输出：

+---+--------------------------------------------------------------------------------+
| id|                                                                           words|                      
----+--------------------------------------------------------------------------------+
|012|[content, type, multipart, alternative, boundaries, nextpart, nextpart, drama,..|
|013|[received, from, am5eur02ht120, eop, eur02, prod, protection, outlook by, pro...|

Answer 1

我认为你可以使用array_intersect 函数和size 函数来实现你需要的东西，像这样（未经测试，不确定这个F.lit(list) 的确切代码）：

>>> import pyspark.sql.functions as F

>>> df.show()
+----------------+
|           words|
+----------------+
|[content, word2]|
|      [111, 222]|
+----------------+

>>> list_col = F.array(*[F.lit(cl) for cl in list])
>>> df.filter(F.size(F.array_intersect(F.col("words"), list_col)) > 0).show()
+----------------+
|           words|
+----------------+
|[content, word2]|
+----------------+