比较 2 个数据帧答案

【问题标题】：To Compare 2 dataframes比较 2 个数据帧
【发布时间】：2021-12-14 00:02:38
【问题描述】：

我正在尝试比较 2 个数据帧并使用 pyspark 获取不匹配的行值。给定 df1 和 df2 格式如下

df1 :
+-----+---------+----------+--------+------+------+
|   id|firstname|middlename|lastname|gender|salary|
+-----+---------+----------+--------+------+------+
|42114|   Robert|          |Williams|     M|  5000|    
|40288|  Michael|      Rose|        |     M|  4000|   
|39192|    Maria|      Anne|   Jones|     F|  4000|
|36636|    James|          |   Smith|     M|  3000|       
|     |      Jen|      Mary|  Browln|     F|    -1|  
+-----+---------+----------+--------+------+------+

df2:
+-----+---------+----------+--------+------+------+
|   id|firstname|middlename|lastname|gender|salary|
+-----+---------+----------+--------+------+------+
|42114|   Robert|          |Williams|     M|  6000|    
|40288|  Michael|      Rose|        |     M|  4000|   
|39192|    Maria|      Anne|   Jones|     M|  4000|
|36636|    James|          |   Smith|     M|  3000|       
|     |      Jen|      Mary|  Browln|     F|    -1|  
+-----+---------+----------+--------+------+------+

当存在不匹配时例如，第一行薪水列不匹配。所以 df1 和 df2 的值都需要并排写入

Output

+-----+---------+----------+--------+------+------++-----+---------+----------+--------+-
|   id|firstname|middlename|lastname|gender|salary|df1_gender|df2_gender|df1_salary|df2_salay
+-----+---------+----------+--------+------+------+
|42114|   Robert|          |Williams|     M |        |    |   |         | 6000.    | 50000
|
|39192|    Maria|      Anne|   Jones|       |  4000   |  F  |  M  |      |          |
+-----+---------+----------+--------+------+------+

conditions_ = [when(df1[c] != df2[c], lit(c)).otherwise("") for c in df1.columns if c != 'id']
select_expr = [
    col("id"),
    [df2[c] for c in df2.columns if c != 'id'],
    array_remove(array(*conditions_), "").alias("column_names")
]
df1.join(df2, "id").select(*select_expr).show()

我是这样的。有人可以帮我解决这个问题吗

【问题讨论】：

由于不匹配是不可预测的，所有列都可能不同（我想除了 ID），所以我们最终会发现所有列都是重复的，我理解正确吗？
你找到答案了吗？

标签： dataframe apache-spark pyspark

【解决方案1】：

如果我理解正确，您希望保留至少一个不匹配的行。我会这样做：

from pyspark.sql.functions import *
import functools

# a function that prefixes all columns but 'id' with the dataframe name
def rename_columns(df, name):
    renamed_cols = [col(c).alias(name + "_" + c) for c in df.columns if c != "id"]
    return df.select(["id"] + renamed_cols)

df1_join = rename_columns(df1, "df1").na.fill("")
df2_join = rename_columns(df2, "df2").na.fill("")

# a list of mismatch conditions
condition_list = [col("df2_" + c) != col("df1_" + c) for c in df2.columns if c != "id"]
# merging the list above into one condition
condition = functools.reduce(lambda a, b: a or b, condition_list)

result = df1_join.join(df2_join, ['id']).where(condition)
result.show()

+-----+-------------+--------------+------------+----------+----------+-------------+--------------+------------+----------+----------+
|   id|df1_firstname|df1_middlename|df1_lastname|df1_gender|df1_salary|df2_firstname|df2_middlename|df2_lastname|df2_gender|df2_salary|
+-----+-------------+--------------+------------+----------+----------+-------------+--------------+------------+----------+----------+
|42114|       Robert|          null|    Williams|         M|      5000|       Robert|          null|    Williams|         M|      6000|
|39192|        Maria|          Anne|       Jones|         F|      4000|        Maria|          Anne|       Jones|         M|      4000|
+-----+-------------+--------------+------------+----------+----------+-------------+--------------+------------+----------+----------+

如果“并排”显示列对您很重要，您可以像这样重新调整数据框的形状：

columns = [["df1_"+c, "df2_"+c]  for c in df1.columns if c != 'id']
ordered_columns = functools.reduce(lambda a, b: a+b, columns)
result = result.select(['id'] + ordered_columns)

【讨论】：