pyspark中的case语句问题

Question

我正在使用下面的代码来获取一个数据帧，但出现错误。如果将从代码中删除 case 语句，那么它将正常工作。

from pyspark.sql import functions as F
ivr_icm_cfs_lookup = spark.table("sample_table").where(F.col("tier") == "Servicing").select("fig","fig_bu","tier") .withColumn("fig",F.regexp_replace(F.col("fig"),"[^0-9a-zA-Z]",' ')).withColumn("fig_bu",F.when((F.col("fig_bu") == 'OPEN' & F.col("tier") == 'Academy', F.upper("tier")).otherwise(F.col("fig_bu"))))

Answer 1

先生，您的括号有问题。立即尝试

from pyspark.sql import functions as F
ivr_icm_cfs_lookup = spark.table("sample_table").where(F.col("tier") == "Servicing").select("fig","fig_bu","tier") .withColumn("fig",F.regexp_replace(F.col("fig"),"[^0-9a-zA-Z]",' ')).withColumn("fig_bu",F.when((F.col("fig_bu") == 'OPEN') & (F.col("tier") == 'Academy'), F.upper("tier")).otherwise(F.col("fig_bu")))

如果这真的不适合你，那么你可以通过将数据框注册为表然后在 sql 中编写你的案例来尝试 spark sql。

from pyspark.sql import functions as F
ivr_icm_cfs_lookup = spark.table("sample_table").where(F.col("tier") == "Servicing").select("fig","fig_bu","tier") .withColumn("fig",F.regexp_replace(F.col("fig"),"[^0-9a-zA-Z]",' '))

ivr_icm_cfs_lookup.createOrReplaceTempView('ivr_icm_cfs_lookup_table')
ivr_icm_cfs_lookup_df = spark.sql('''
select *, case when fig_bu='OPEN' and tier='Academy' then upper(tier) else fig_bu END as fig_bu_2 from ivr_icm_cfs_lookup_table
'''

这样你就可以添加所需的列了。