pyspark 将 json 数组转换为多列

Question

我正在使用以下代码从 api 读取数据，其中有效负载为 json 格式，使用 pyspark 在 azure databricks 中。所有字段都定义为字符串，但一直运行到 json_tuple 要求所有参数都是字符串错误。

架构：

root
 |-- Payload: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- ActiveDate: string (nullable = true)
 |    |    |-- BusinessId: string (nullable = true)
 |    |    |-- BusinessName: string (nullable = true)

JSON：

 {
    "Payload": 
    [
        {
            "ActiveDate": "2008-11-25",
            "BusinessId": "5678",
            "BusinessName": "ACL"
        },
        {
            "ActiveDate": "2009-03-22",
            "BusinessId": "6789",
            "BusinessName": "BCL"
        }
    ]
}

PySpark：

from pyspark.sql import functions as F
df = df.select(F.col('Payload'), F.json_tuple(F.col('Payload'), 'ActiveDate', 'BusinessId', 'BusinessName') \.alias('ActiveDate', 'BusinessId', 'BusinessName'))
df.write.format("delta").mode("overwrite").saveAsTable("delta_payload")

错误：

AnalysisException: cannot resolve 'json_tuple(`Payload`, 'ActiveDate', 'BusinessId', 'BusinessName')' due to data type mismatch: json_tuple requires that all arguments are strings;

Answer 1

从您的架构看来，JSON 已被解析，因此 Payload 属于 ArrayType 而不是包含 JSON 的 StringType，因此出现错误。

您可能需要explode 而不是json_tuple：

>>> from pyspark.sql.functions import explode
>>> df = spark.createDataFrame([{
...     "Payload":
...     [
...         {
...             "ActiveDate": "2008-11-25",
...             "BusinessId": "5678",
...             "BusinessName": "ACL"
...         },
...         {
...             "ActiveDate": "2009-03-22",
...             "BusinessId": "6789",
...             "BusinessName": "BCL"
...         }
...     ]
... }])
>>> df.schema
StructType(List(StructField(Payload,ArrayType(MapType(StringType,StringType,true),true),true)))
>>> df.select(explode("Payload").alias("x")).select("x.ActiveDate", "x.BusinessName", "x.BusinessId").show()
+----------+------------+----------+
|ActiveDate|BusinessName|BusinessId|
+----------+------------+----------+
|2008-11-25|         ACL|      5678|
|2009-03-22|         BCL|      6789|
+----------+------------+----------+