循环 3 次，每次向 spark DF 中的新列添加一个新值

Question

我想为 pysaprk DF 中的每一行创建 3 行。我想添加一个名为 loopVar=(val1,val2,val3) 的新列。必须在每个循环中添加三个不同的值作为值。知道我该怎么做吗？

原文：

    a   b   c
    1   2   3
    1   2   3

 Condition 1: loop = 1  and b is not null then loopvar =va1
 Condition 2: loop = 2  and b is not null then loopvar =va2
 Condition 3: loop = 3  and c is not null then loopvar =va3

输出：

    a   b   c  loopvar
    1   2   3  val1
    1   2   3  vall
    1   2   3  val2
    1   2   3  val2
    1   2   3  val3
    1   2   3  val3

Answer 1

使用crossJoin：

df = spark.createDataFrame([[1,2,3], [1,2,3]]).toDF('a','b','c')
df.show()
+---+---+---+
|  a|  b|  c|
+---+---+---+
|  1|  2|  3|
|  1|  2|  3|
+---+---+---+

df2 = spark.createDataFrame([['val1'], ['val2'], ['val3']]).toDF('loopvar')
df2.show()
+-------+
|loopvar|
+-------+
|   val1|
|   val2|
|   val3|
+-------+

df3 = df.crossJoin(df2)
df3.show()
+---+---+---+-------+
|  a|  b|  c|loopvar|
+---+---+---+-------+
|  1|  2|  3|   val1|
|  1|  2|  3|   val2|
|  1|  2|  3|   val3|
|  1|  2|  3|   val1|
|  1|  2|  3|   val2|
|  1|  2|  3|   val3|
+---+---+---+-------+