需要从单列数据框中提取 Json 数据（嵌套数组） - 表以 Null 形式出现，模式读取 -Scala

Question

我正在尝试从数据框中提取如下数据。具有嵌套数组的 Json 数据完全在一列（_c1）中。我想将其拉出并将其创建为具有有效列名的单独数据框。一个示例记录如下。

|_c1                                                                                                                                                                                                                                                                                                                                                                   |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|{"Id":"31279605299","Type":"12121212","client":"Checklist _API","eventTime":"2020-03-17T15:50:30.640Z","eventType":"Event","payload":{"sourceApp":"ios","questionnaire":{"version":"1.0","question":"How to resolve ? ","fb":"Na"}}} 

我正在将其读取为架构，

val schema=StructType(Array(
      StructField("Id", StringType, false),
      StructField("Type", StringType, false),
      StructField("client", StringType, false),
      StructField("eventTime", StringType, false),
      StructField("eventType", StringType, false),
      StructField("payload", ArrayType(StructType(Array(
        StructField("sourceApp", StringType, false),
        StructField("questionnaire", ArrayType(StructType(Array(
          StructField("version", StringType, false),
          StructField("question", StringType, false),
          StructField("fb", StringType, false)))))
      ))))
    ))

      val json_paral = DF.select(from_json(col("_c1"),schema))
`
Structure comes out as below,
`

 |-- jsontostructs(_c1): struct (nullable = true)
 |    |-- Id: string (nullable = true)
 |    |-- Type: string (nullable = true)
 |    |-- client: string (nullable = true)
 |    |-- eventTime: string (nullable = true)
 |    |-- eventType: string (nullable = true)
 |    |-- payload: array (nullable = true)
 |    |    |-- element: struct (containsNull = true)
 |    |    |    |-- sourceApp: string (nullable = true)
 |    |    |    |-- questionnaire: array (nullable = true)
 |    |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |    |-- version: string (nullable = true)
 |    |    |    |    |    |-- question: string (nullable = true)
 |    |    |    |    |    |-- fb: string (nullable = true)

结构很好，但是当我检查数据框时，所有数据都显示为 NULL。读书好吗？也没有遇到任何解析问题。

Answer 1

请检查这是否有帮助-

1。加载数据

val data = """{"Id":"31279605299","Type":"12121212","client":"Checklist _API","eventTime":"2020-03-17T15:50:30.640Z","eventType":"Event","payload":{"sourceApp":"ios","questionnaire":{"version":"1.0","question":"How to resolve ? ","fb":"Na"}}} """

    val df = Seq(data).toDF("jsonCol")
    df.show(false)
    df.printSchema()

输出-

+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|jsonCol                                                                                                                                                                                                                              |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|{"Id":"31279605299","Type":"12121212","client":"Checklist _API","eventTime":"2020-03-17T15:50:30.640Z","eventType":"Event","payload":{"sourceApp":"ios","questionnaire":{"version":"1.0","question":"How to resolve ? ","fb":"Na"}}} |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

root
 |-- jsonCol: string (nullable = true)

2。提取json字符串以分隔文件

 df.select(json_tuple(col("jsonCol"), "Id", "Type", "client", "eventTime", "eventType", "payload"))
      .show(false)

输出-

+-----------+--------+--------------+------------------------+-----+----------------------------------------------------------------------------------------------+
|c0         |c1      |c2            |c3                      |c4   |c5                                                                                            |
+-----------+--------+--------------+------------------------+-----+----------------------------------------------------------------------------------------------+
|31279605299|12121212|Checklist _API|2020-03-17T15:50:30.640Z|Event|{"sourceApp":"ios","questionnaire":{"version":"1.0","question":"How to resolve ? ","fb":"Na"}}|
+-----------+--------+--------------+------------------------+-----+----------------------------------------------------------------------------------------------+

3。使用from_json(..)

val processed = df.select(
      expr("from_json(jsonCol, 'struct<Id:string,Type:string,client:string,eventTime:string, eventType:string," +
      "payload:struct<questionnaire:struct<fb:string,question:string,version:string>,sourceApp:string>>')")
        .as("json_converted"))
      processed.show(false)
      processed.printSchema()

输出-

+-------------------------------------------------------------------------------------------------------------+
|json_converted                                                                                               |
+-------------------------------------------------------------------------------------------------------------+
|[31279605299, 12121212, Checklist _API, 2020-03-17T15:50:30.640Z, Event, [[Na, How to resolve ? , 1.0], ios]]|
+-------------------------------------------------------------------------------------------------------------+

root
 |-- json_converted: struct (nullable = true)
 |    |-- Id: string (nullable = true)
 |    |-- Type: string (nullable = true)
 |    |-- client: string (nullable = true)
 |    |-- eventTime: string (nullable = true)
 |    |-- eventType: string (nullable = true)
 |    |-- payload: struct (nullable = true)
 |    |    |-- questionnaire: struct (nullable = true)
 |    |    |    |-- fb: string (nullable = true)
 |    |    |    |-- question: string (nullable = true)
 |    |    |    |-- version: string (nullable = true)
 |    |    |-- sourceApp: string (nullable = true)

Answer 2

我没有将其读取到架构中，而是尝试将其设置为一个值

val Df = json_DF.map(r => r.getString(0))

这会将数据作为字符串提取出来，下面的内容会将其提取出来，并将键作为列名。

val g1DF=spark.read.json(Df)

某些横向视图是否会爆炸嵌套以提取嵌套数组值。