json文件嵌套列值在pyspark中为null

Question

json 文件： "products":[{"productId":"8d809e13-fdc5-4d15-9271- 953750f6d592","quantity":500,"soldPrice":104.852}, {"productId":"ec15ba1d-53b6-44b0-8a22- 1e498485f1b8","quantity":300,"soldPrice":94.3668}, {"productId":"e672483e-57a8-434a-bc42- ecf827c8a8d4","quantity":1000,"soldPrice":109.57034}], "shippingAddress":{"attention":"Khaleesi Frost","address":"493 Augustine Drive N ","city":"Miramar","state":"FL","zip":"33785"}

我可以使用explode 方法访问“产品”，因为它是ArrayType。但“shippingAddress”的值为 null。

`select(F.col("shippingAddress.attention").alias("shipping_address_attention"),
   F.col("shippingAddress.address").alias("shipping_address_address"))`

json 架构： StructField('shippingAddress' , StructType([ StructField('attention' , StringType(), True), StructField('address' , StringType(), True), StructField('city' , StringType(), True), StructField('state' , StringType(), True), StructField('zip' , IntegerType(), True) ]))

Answer 1

检查您是否有有效的 JSON here。

如果您使用的 JSON 与您在问题中发布的相同，则应将整个 JSON 包含在 {} 中。读取 JSON 时使用选项 multiline 为 True。

>>> from pyspark.sql.functions import *
>>> df = spark.read.option('multiline', True).json('file.json')
>>> df.printSchema()
root
 |-- products: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- productId: string (nullable = true)
 |    |    |-- quantity: long (nullable = true)
 |    |    |-- soldPrice: double (nullable = true)
 |-- shippingAddress: struct (nullable = true)
 |    |-- address: string (nullable = true)
 |    |-- attention: string (nullable = true)
 |    |-- city: string (nullable = true)
 |    |-- state: string (nullable = true)
 |    |-- zip: string (nullable = true)

>>> df.select(col("shippingAddress.attention").alias("shipping_address_attention"),col("shippingAddress.address").alias("shipping_address_address")).show(truncate=False)
+--------------------------+------------------------+
|shipping_address_attention|shipping_address_address|
+--------------------------+------------------------+
|Khaleesi Frost            |493 Augustine Drive N   |
+--------------------------+------------------------+

更新

您提到 zip 列类型为 IntegerType() 但在 JSON 中它是字符串，由于这种类型不匹配，您在列中得到 Null。所以在你的自定义模式中将类型更改为 StringType()。

你的架构应该是，

StructType([StructField('shippingAddress' , StructType([ StructField('attention' , StringType(), True), StructField('address' , StringType(), True),                                 
StructField('city' , StringType(), True), StructField('state' , StringType(), True), StructField('zip' , StringType(), True) ]))])