如何从 SPARK SCALA 中的 XML 模式中获取列名？

Question

我有一个要求，我需要处理包含 XML 的表中的列。我正在尝试根据标签将 XML 列转换为多个单独的列。我正在使用 spark_xml 类来执行相同的操作。 我参考了问题 parsing XML columns from PySpark Dataframe using UDF 但是这里全部在 Pyspark 中处理，我需要在 SCALA 中处理。

我已经到了可以制作解析列的地步了。 我需要分解数据并将 XML 标记转换为列名。 我需要 SCALA 中该问题的以下行的等效项

df2 = parsed.select(*parsed.columns[:-1],F.explode(F.col('parsed').getItem('visitor')))    

new_col_names = [s.split(':')[0] for s in payloadSchema['visitor'].simpleString().split('<')[-1].strip('>>').split(',')]

添加 XML

<?xml version="1.0" encoding="utf-8"?> <visitors> <visitor id="9615" age="68" sex="F" /> <visitor id="1882" age="34" sex="M" /> <visitor id="5987" age="23" sex="M" /> </visitors>

输出：

> # +---+--------------------+----+----+----+
> # | id|            visitors|_age| _id|_sex|
> # +---+--------------------+----+----+----+
> # |  1|<?xml version="1....|  68|9615|   F|
> # |  1|<?xml version="1....|  34|1882|   M|
> # |  1|<?xml version="1....|  23|5987|   M|
> # +---+--------------------+----+----+----+

Answer 1

使用org.json将xml转换为json。

示例 XML 数据

val xmlData = """<?xml version="1.0" encoding="utf-8"?> <visitors> <visitor id="9615" age="68" sex="F" /> <visitor id="1882" age="34" sex="M" /> <visitor id="5987" age="23" sex="M" /> </visitors>"""

UDF 函数

val parse = udf((value: String) => {
    import org.json._
    XML.toJSONObject(value).toString
  }
)

转换后的 json 数据的架构。

import org.apache.spark.sql.types._

val schema = DataType.fromJson("""{"type":"struct","fields":[{"name":"visitors","type":{"type":"struct","fields":[{"name":"visitor","type":{"type":"array","elementType":{"type":"struct","fields":[{"name":"age","type":"long","nullable":true,"metadata":{}},{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"sex","type":"string","nullable":true,"metadata":{}}]},"containsNull":true},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}}]}""").asInstanceOf[StructType]

scala> schema.printTreeString
root
 |-- visitors: struct (nullable = true)
 |    |-- visitor: array (nullable = true)
 |    |    |-- element: struct (containsNull = true)
 |    |    |    |-- age: long (nullable = true)
 |    |    |    |-- id: long (nullable = true)
 |    |    |    |-- sex: string (nullable = true)

 df
 .withColumn(
     "parsed_xml", 
     from_json(parse($"xml"),schema)
    )
 .select(
        $"id",
        $"xml",
        explode_outer($"parsed_xml.visitors.visitor").as("visitors")
    )
 .select(
     $"id",
     $"xml",
     $"visitors.*"
    )
 .show(false)

最终输出

+---+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---+----+---+
|id |xml                                                                                                                                                                               |age|id  |sex|
+---+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---+----+---+
|1  |<?xml version="1.0" encoding="utf-8"?> <visitors> <visitor id="9615" age="68" sex="F" /> <visitor id="1882" age="34" sex="M" /> <visitor id="5987" age="23" sex="M" /> </visitors>|68 |9615|F  |
|1  |<?xml version="1.0" encoding="utf-8"?> <visitors> <visitor id="9615" age="68" sex="F" /> <visitor id="1882" age="34" sex="M" /> <visitor id="5987" age="23" sex="M" /> </visitors>|34 |1882|M  |
|1  |<?xml version="1.0" encoding="utf-8"?> <visitors> <visitor id="9615" age="68" sex="F" /> <visitor id="1882" age="34" sex="M" /> <visitor id="5987" age="23" sex="M" /> </visitors>|23 |5987|M  |
+---+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---+----+---+