将字符串格式的不同日期格式解析为日期格式pyspark when 子句

Question

需要将此格式转换为

name,code,DATE_invoice
Ram,E01,09/29/2018
Mara,E02,07/14/2017
Test,E03,01/01/18

这个：

name,code,DATE_invoice
Ram,E01,2018-09-29
Mara,E02,2017-07-14
Test,E03,2018-01-01

Answer 1

如果该列已经是日期，这应该可以完成工作：

df = df.withColumn('DATE_invoice', date_format(col("DATE_invoice"), "yyyy-MM-dd")))

Answer 2

要解析不同的数据格式，您可以使用to_date 和coalesce

您可以对数据集中的多个模式使用相同的方法，可以在here找到示例

数据准备

input_str = """
Ram,E01,09/29/2018,
Mara,E02,07/14/2017,
Test,E03,01/01/18
""".split(",")

input_values = list(map(lambda x: x.strip() if x.strip() != 'null' else None, input_str))

cols = list(map(lambda x: x.strip() if x.strip() != 'null' else None, "name,code,DATE_invoice".split(",")))
            
n = len(input_values)
n_col = 3

input_list = [tuple(input_values[i:i+n_col]) for i in range(0,n,n_col)]

sparkDF = sql.createDataFrame(input_list, cols)

sparkDF.show()

+----+----+------------+
|name|code|DATE_invoice|
+----+----+------------+
| Ram| E01|  09/29/2018|
|Mara| E02|  07/14/2017|
|Test| E03|    01/01/18|
+----+----+------------+

迄今为止和合并

sql.sql("set spark.sql.legacy.timeParserPolicy=LEGACY")

sparkDF.withColumn('p1',F.to_date(F.col('DATE_invoice'),"MM/dd/yyyy"))\
       .withColumn('p2',F.to_date(F.col('DATE_invoice'),"MM/dd/yy"))\
       .withColumn('DATE_invoice_parsed',F.coalesce(F.col('p1'),F.col('p2')))\
       .drop(*['p1','p2'])\
       .show(truncate=False)

+----+----+------------+-------------------+
|name|code|DATE_invoice|DATE_invoice_parsed|
+----+----+------------+-------------------+
|Ram |E01 |09/29/2018  |2018-09-29         |
|Mara|E02 |07/14/2017  |2017-07-14         |
|Test|E03 |01/01/18    |0018-01-01         |
+----+----+------------+-------------------+