使用 Java 读取 CSV 文件包含 Spark 中的结构类型

Question

我正在尝试为程序编写测试用例。 为此，我正在读取包含以下格式数据的 CSV 文件。
account_number,struct_data 123456789,{"key1":"value","key2":"value2","keyn":"valuen"} 987678909,{"key1":"value0","key2":"value20","keyn":"valuen0"}
数百个这样的行。

我需要将第二列作为结构读取。但我得到了错误 struct type expected, string type found

我尝试转换为 StructType，然后得到错误为“StringType 无法转换为 StructType”。

我应该改变我的 CSV 的方式吗？我还能做什么？

Answer 1

我在 Scala Spark 中提供了我的解决方案，它可能会为您的查询提供一些见解

scala> val sdf = """{"df":[{"actNum": "1234123", "strType": [{"key1": "value1", "key2": "value2"}]}]}"""
sdf: String = {"df":[{"actNum": "1234123", "strType": [{"key1": "value1", "key2": "value2"}]}]}

scala> val erdf = spark.read.json(Seq(sdf).toDS).toDF().withColumn("arr", explode($"df")).select("arr.*")
erdf: org.apache.spark.sql.DataFrame = [actNum: string, strType: array<struct<key1:string,key2:string>>]

scala> erdf.show()
+-------+-----------------+
| actNum|          strType|
+-------+-----------------+
|1234123|[[value1,value2]]|
+-------+-----------------+


scala> erdf.printSchema
root
 |-- actNum: string (nullable = true)
 |-- strType: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- key1: string (nullable = true)
 |    |    |-- key2: string (nullable = true)

Answer 2

如果所有 json 记录具有相同的架构，您可以定义它并使用 sparks from_json() 函数来完成您的任务。

import org.apache.spark.sql.types.StructType

val df = Seq(
    (123456789, "{\"key1\":\"value\",\"key2\":\"value2\",\"keyn\":\"valuen\"}"),
    (987678909, "{\"key1\":\"value0\",\"key2\":\"value20\",\"keyn\":\"valuen0\"}")
    ).toDF("account_number", "struct_data")

val schema = new StructType()
  .add($"key1".string)
  .add($"key2".string)
  .add($"keyn".string)

 val df2 = df.withColumn("st", from_json($"struct_data", schema))

 df2.printSchema
 df2.show(false)

这个 sn-p 导致这个输出：

root
 |-- account_number: integer (nullable = false)
 |-- struct_data: string (nullable = true)
 |-- st: struct (nullable = true)
 |    |-- key1: string (nullable = true)
 |    |-- key2: string (nullable = true)
 |    |-- keyn: string (nullable = true)

+--------------+---------------------------------------------------+------------------------+
|account_number|struct_data                                        |st                      |
+--------------+---------------------------------------------------+------------------------+
|123456789     |{"key1":"value","key2":"value2","keyn":"valuen"}   |[value,value2,valuen]   |
|987678909     |{"key1":"value0","key2":"value20","keyn":"valuen0"}|[value0,value20,valuen0]|
+--------------+---------------------------------------------------+------------------------+