比较来自两个不同 pyspark 数据帧的两个不同列答案

【问题标题】：Compare two different columns from two different pyspark dataframe比较来自两个不同 pyspark 数据帧的两个不同列
【发布时间】：2021-07-21 14:11:29
【问题描述】：

我正在尝试比较两个不同数据帧中的两个不同列，如果找到匹配项，则返回值 1 else None -

df1 =

df2 =

df1 (Expected_Output) =

我试过下面的代码-

def getImpact(row):
match = df2.filter(df2.second_key == row)
if match.count() > 0:
    return 1
return None

udf_sol = udf(lambda x: getImpact(x), IntegerType())
df1 = df1.withcolumn('impact',udf_sol(df1.first_key))

但低于错误 - TypeError: cannot pickle '_thread.RLock' object

谁能帮我实现如上所示的预期输出？

谢谢

【问题讨论】：

标签： python pyspark apache-spark-sql

【解决方案1】：

假设 first_key 和 second_key 是唯一的，您可以选择跨数据框连接 -

更多例子和解释可以找到here

from pyspark import SparkContext
from pyspark.sql import SQLContext
from functools import reduce
import pyspark.sql.functions as F

from pyspark.sql import Window


data_list1 = [
    ("abcd","Key1")
    ,("jkasd","Key2")
    ,("oigoa","Key3")
    ,("ad","Key4")
    ,("bas","Key5")
    ,("lkalsjf","Key6")
    ,("bsawva","Key7")
]

data_list2 = [
    ("cashj","Key1",10)
    ,("ax","Key11",12)
    ,("safa","Key5",21)
    ,("safasf","Key6",78)
    ,("vasv","Key3",4)
    ,("wgaga","Key8",0)
    ,("saasfas","Key7",10)
]

sparkDF1 = sql.createDataFrame(data_list1,['data','first_key'])
sparkDF2 = sql.createDataFrame(data_list2,['temp_data','second_key','frinks'])


>>> sparkDF1
+-------+---------+
|   data|first_key|
+-------+---------+
|   abcd|     Key1|
|  jkasd|     Key2|
|  oigoa|     Key3|
|     ad|     Key4|
|    bas|     Key5|
|lkalsjf|     Key6|
| bsawva|     Key7|
+-------+---------+

>>> sparkDF2
+---------+----------+------+
|temp_data|second_key|frinks|
+---------+----------+------+
|    cashj|      Key1|    10|
|       ax|     Key11|    12|
|     safa|      Key5|    21|
|   safasf|      Key6|    78|
|     vasv|      Key3|     4|
|    wgaga|      Key8|     0|
|  saasfas|      Key7|    10|
+---------+----------+------+

#### Joining the dataframes on common columns 
finalDF = sparkDF1.join(
                sparkDF2
             ,(sparkDF1['first_key'] == sparkDF2['second_key'])
            ,'left'
).select(sparkDF1['*'],sparkDF2['frinks']).orderBy('frinks')


### Identifying impact if the frinks value is Null or Not
finalDF = finalDF.withColumn('impact',F.when(F.col('frinks').isNull(),0).otherwise(1))

>>> finalDF.show()

+-------+---------+------+------+
|   data|first_key|frinks|impact|
+-------+---------+------+------+
|  jkasd|     Key2|  null|     0|
|     ad|     Key4|  null|     0|
|  oigoa|     Key3|     4|     1|
|   abcd|     Key1|    10|     1|
| bsawva|     Key7|    10|     1|
|    bas|     Key5|    21|     1|
|lkalsjf|     Key6|    78|     1|
+-------+---------+------+------+

【讨论】：

【解决方案2】：

将 numpy 导入为 np

df1['final']= np.where(df1['first_key']==df2['second_key'],'1','None')

【讨论】：