在 PySpark 中查找两个数据帧之间的变化

Question

我有两个数据框，比如说 dfA 和 dfB。

dfA:
IdCol | Col2  | Col3
id1   | val2  | val3

dfB:
IdCol | Col2  | Col3
id1   | val2  | val4

两个数据框在 IdCol 中连接。我想每行比较它们并保持列不同以及它们在另一个数据框中的值。例如，从上述两个数据框中，我想要一个结果：

dfChanges:
RowId | Col  | dfA_value | dfB_value |
id1   | Col3 | val_3     | val_4     |

我有点坚持如何做到这一点。任何人都可以提供方向吗？ 提前致谢

编辑

我的尝试是这样的。但它不是很清楚或具有良好的性能。有更好的方法吗？

dfChanges = None

#for all column excpet id
for colName in dfA.column[1:]:

    #Select whole columns of id and targeted column 
    #from both datasets and subtract to find differences
    changedRows = dfA.select(['IdCol',colName]).subtract(dfB.select(['IdCol',colName]))

    #Join with dfB to take the value of targeted column from there
    temp = changedRows.join(dfB.select(col('IdCol'),col(colName).alias("dfB_value")),dfA.IdCol == dfB.IdCol, 'inner'). \
    drop(dfB.IdCol)

    #Proper Rename columns
    temp = temp.withColumnRenamed(colname,"dfA_value")
    temp = temp.withColumn("Col",lit(colName))

    #Append to a single dataframe
    if (dfChanges is None):
        dfChanges = temp
    else:
        dfChanges = dfChanges.union(temp)

Answer 1

通过 id 加入两个数据框：

dfA = spark.createDataFrame(
    [("id1", "val2", "val3")], ("Idcol1", "Col2", "Col3")
)

dfB = spark.createDataFrame(
    [("id1", "val2", "val4")], ("Idcol1", "Col2", "Col3")
)

dfAB = dfA.alias("dfA").join(dfB.alias("dfB"), "idCol1")

重塑：

from pyspark.sql.functions import col, struct

ids = ["Idcol1"]

vals = [struct(
    col("dfA.{}".format(c)).alias("dfA_value"),
    col("dfB.{}".format(c)).alias("dfB_value")
).alias(c) for c in dfA.columns if c not in ids]

和melt（定义为here）

(melt(dfAB.select(ids + vals), ids, [c for c in dfA.columns if c not in ids])
    .where(col("value.dfA_value") != col("value.dfB_value"))
    .select(ids + ["variable" , "value.dfA_value", "value.dfB_value"])
    .show())

+------+--------+---------+---------+                                           
|Idcol1|variable|dfA_value|dfB_value|
+------+--------+---------+---------+
|   id1|    Col3|     val3|     val4|
+------+--------+---------+---------+