使用嵌套数组和 StructType Spark Scala 展平 Parquet 文件

Question

我希望使用 Scala 在 Spark 中有效地动态展平镶木地板文件。我想知道实现这一目标的有效方法是什么。

parquet 文件包含多个深度级别的 Array 和 Struct 类型嵌套。 parquet 文件架构将来可能会更改，因此我无法对任何属性进行硬编码。所需的最终结果是一个扁平的分隔文件。

使用平面图和递归爆炸的解决方案会起作用吗？

示例架构：

|-- exCar: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- exCarOne: string (nullable = true)
 |    |    |-- exCarTwo: string (nullable = true)
 |    |    |-- exCarThree: string (nullable = true)
 |-- exProduct: string (nullable = true)
 |-- exName: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- exNameOne: string (nullable = true)
 |    |    |-- exNameTwo: string (nullable = true)
 |    |    |-- exNameThree: string (nullable = true)
 |    |    |-- exNameFour: string (nullable = true)
 |    |    |-- exNameCode: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- exNameCodeOne: string (nullable = true)
 |    |    |    |    |-- exNameCodeTwo: string (nullable = true)
 |    |    |-- exColor: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- exColorOne: string (nullable = true)
 |    |    |    |    |-- exColorTwo: string (nullable = true)
 |    |    |    |    |-- exWheelColor: array (nullable = true)
 |    |    |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |    |    |-- exWheelColorOne: string (nullable = true)
 |    |    |    |    |    |    |-- exWheelColorTwo: string (nullable = true)
 |    |    |    |    |    |    |--exWheelColorThree: string (nullable =true)
 |    |    |-- exGlass: string (nullable = true)
 |-- exDetails: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- exBill: string (nullable = true)
 |    |    |-- exAccount: string (nullable = true)
 |    |    |-- exLoan: string (nullable = true)
 |    |    |-- exRate: string (nullable = true)

所需的输出架构：

 exCar.exCarOne
 exCar.exCarTwo
 exCar.exCarThree
 exProduct
 exName.exNameOne
 exName.exNameTwo
 exName.exNameThree
 exName.exNameFour
 exName.exNameCode.exNameCodeOne
 exName.exNameCode.exNameCodeTwo
 exName.exColor.exColorOne
 exName.exColor.exColorTwo
 exName.exColor.exWheelColor.exWheelColorOne
 exName.exColor.exWheelColor.exWheelColorTwo
 exName.exColor.exWheelColor.exWheelColorThree
 exName.exGlass
 exDetails.exBill
 exDetails.exAccount
 exDetails.exLoan
 exDetails.exRate

Answer 1

有两件事需要做：

1) 将数组列从最外面的嵌套数组分解到位于内部的数组：分解exName（给你很多包含exColor的json行），然后分解exColor，然后你可以分解它访问exWheelColor等

2) 将嵌套的 json 投影到单独的列。