使用 pyspark 将数据框列转换为嵌套的 JSON 结构答案

【问题标题】：Converting a dataframe columns into nested JSON structure using pyspark使用 pyspark 将数据框列转换为嵌套的 JSON 结构
【发布时间】：2021-08-26 23:55:15
【问题描述】：

我对 pyspark 还很陌生

到目前为止我尝试过的代码

ConceptGriddf = spark.sql("""
          SELECT  DataID
                 ,collect_list(struct(ConceptGridName AS Title,
                                      named_struct("ZoneName",ZoneName
                                                   ,"Zone", Zone
                                                   ,"ZoneScore", ZoneScore)as Zones
                                                   ))ConceptGrid
          FROM OutputTable 
          GROUP BY DataID """)

if dbutils.widgets.get("Debug") == 'yes':
  display(ConceptGriddf.limit(10))

桌子

DataID ConceptGridName Zone ZoneName ZoneScore
1       Reserved_89    y    Shared    0.58115    
1       Reserved_89    x    Unshared  0.4939

这是我当前在数据框中的 JSON 输出

"ConceptGrid": [
    {
        "Title": "Reserved_89",
        "Zones": {
            "ZoneName": "Shared",
            "Zone": "y",
            "ZoneScore": 0.58115
        }
    },
    {
        "Title": "Reserved_89",
        "Zones": {
            "ZoneName": "Unshared",
            "Zone": "x",
            "ZoneScore": 0.4939
        }
    }
  ]

相反，我希望它看起来像这样，但我无法使用我在这里使用的功能来实现

"ConceptGrid": [
      {
          "Title":"Reserved_89",
          "Zones":[
              {
              "ZoneName":"Shared",
              "Zone":"y",
              "ZoneScore":0.58115
              },
              {
                "ZoneName":"Unshared",
                "Zone":"x",
                "ZoneScore":0.4939
                }
            
            ]
        }
    ]

【问题讨论】：

标签： json apache-spark pyspark apache-spark-sql

【解决方案1】：

检查下面的代码。

 spark.sql("with data as (select ConceptGridName as Title,collect_list(struct(ZoneName,Zone,ZoneScore)) as zones from OutputTable group by ConceptGridName) select collect_list(struct(Title,zones)) as ConceptGrid from data").toJSON.show(false)
+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
|value                                                                                                                                                         |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
|{"ConceptGrid":[{"Title":"Reserved_89","zones":[{"ZoneName":"Shared","Zone":"y","ZoneScore":0.58115},{"ZoneName":"Unshared","Zone":"x","ZoneScore":0.4939}]}]}|
+--------------------------------------------------------------------------------------------------------------------------------------------------------------+

【讨论】：