pyspark：根据其他记录获取列

Question

我有一个看起来像这样的数据框

membershipAccountNbr            cntryRetailChannelCustId
111590058               1010015900581000010101
214100897               1010041008972100010101
104100897               1010041008971000010101

还有一个是这样的：

membershipAccountNbr    parentMembershipNbr
111590058                   111590058
214100897                   104100897

我的目标是让它看起来像：

membershipAccountNbr parentMembershipNbr parentCustId
111590058               111590058    1010015900581000010101
214100897               104100897    1010041008971000010101

我尝试使用连接，但它们给出了模棱两可的错误。我是 Pyspark 的新手，所以请帮忙。

Answer 1

假设df1 是，

+--------------------+------------------------+
|membershipAccountNbr|cntryRetailChannelCustId|
+--------------------+------------------------+
|           111590058|    10100159005810000...|
|           214100897|    10100410089721000...|
|           104100897|    10100410089710000...|
+--------------------+------------------------+

还有df2，

+--------------------+-------------------+
|membershipAccountNbr|parentMembershipNbr|
+--------------------+-------------------+
|           111590058|          111590058|
|           214100897|          104100897|
+--------------------+-------------------+

然后你跑，

df1.join(df2, on="membershipAccountNbr", how="right").select(
    col("membershipAccountNbr"),
    col("parentMembershipNbr"),
    col("cntryRetailChannelCustId").alias("parentCustId"),
).show()

结果会是这样，

+--------------------+-------------------+--------------------+
|membershipAccountNbr|parentMembershipNbr|        parentCustId|
+--------------------+-------------------+--------------------+
|           111590058|          111590058|10100159005810000...|
|           214100897|          104100897|10100410089721000...|
+--------------------+-------------------+--------------------+