带有命名空间的 XML 到 Pandas DF - Python

Question

我需要将从 Wordpress 下载的 XML 转换为 Pandas DF 以进行一些数据分析。

XML 是自动生成的，并且有一些命名空间：

<rss version="2.0"
    xmlns:excerpt="http://wordpress.org/export/1.2/excerpt/"
    xmlns:content="http://purl.org/rss/1.0/modules/content/"
    xmlns:wfw="http://wellformedweb.org/CommentAPI/"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:wp="http://wordpress.org/export/1.2/"
>

并且该文件有数百个<item>s，每个<item>s 都有数百个字段。 一个例子：

<item>
        <title>MockTitle</title>
        <link>https://mock.url.io/?post_type=mock&#038;p=992</link>
        <pubDate>Fri, 30 Oct 2020 20:47:53 +0000</pubDate>

        (...)

        <wp:postmeta>
            <wp:meta_key><![CDATA[_customer_user]]></wp:meta_key>
            <wp:meta_value><![CDATA[0]]></wp:meta_value>
        </wp:postmeta>
        
        <wp:postmeta>
            <wp:meta_key><![CDATA[_payment_method]]></wp:meta_key>
            <wp:meta_value><![CDATA[paypal]]></wp:meta_value>
        </wp:postmeta>

        <wp:postmeta>
            <wp:meta_key><![CDATA[_payment_method_title]]></wp:meta_key>
            <wp:meta_value><![CDATA[PayPal]]></wp:meta_value>
        </wp:postmeta>

        <wp:postmeta>
            <wp:meta_key><![CDATA[_customer_ip_address]]></wp:meta_key>
            <wp:meta_value><![CDATA[1.1.1.15]]></wp:meta_value>
        </wp:postmeta>

        <wp:postmeta>
            <wp:meta_key><![CDATA[_customer_user_agent]]></wp:meta_key>
            <wp:meta_value><![CDATA[Mock Info about the agent]]></wp:meta_value>
        </wp:postmeta>

        (...)
</item>
<item>
...

我的代码可以轻松提取title 和pubDate 之类的元素，但我不知道如何通过wp:meta_key 搜索并实际提取出与该键对应的wp:meta_value。为了能够使用我想要的特定信息创建数据框，我错过了这一点。

到目前为止，这是我的代码：

import pandas as pd
import xml.etree.ElementTree as et 

xtree = et.parse("data.xml")
xroot = xtree.getroot()

for item in xroot: 
    order_number = item.find("./title").text
    date = item.find("./pubDate").text

#Don't know how to access the different wp:postmeta's and search by wp:meta_key...

Answer 1

使用 xml 的简化版本，以及带有 xpath 的 lxml：

rss = """<rss version="2.0"
    xmlns:excerpt="http://wordpress.org/export/1.2/excerpt/"
    xmlns:content="http://purl.org/rss/1.0/modules/content/"
    xmlns:wfw="http://wellformedweb.org/CommentAPI/"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:wp="http://wordpress.org/export/1.2/"
>
<items>
<item>
        <title>MockTitle</title>
        <link>https://mock.url.io/?post_type=mock&#038;p=992</link>
        <pubDate>Fri, 30 Oct 2020 20:47:53 +0000</pubDate>
        <wp:postmeta>
            <wp:meta_key><![CDATA[_customer_user]]></wp:meta_key>
            <wp:meta_value><![CDATA[0]]></wp:meta_value>
        </wp:postmeta>        
        <wp:postmeta>
            <wp:meta_key><![CDATA[_payment_method]]></wp:meta_key>
            <wp:meta_value><![CDATA[paypal]]></wp:meta_value>
        </wp:postmeta>
</item>
<item>
        <title>MockTitle2</title>
        <link>https://mock2.url.io/?post_type=mock&#038;p=992</link>
        <pubDate>Fri, 30 Oct 2220 20:47:53 +0000</pubDate>
        <wp:postmeta>
            <wp:meta_key><![CDATA[_customer_user]]></wp:meta_key>
            <wp:meta_value><![CDATA[02]]></wp:meta_value>
        </wp:postmeta>        
        <wp:postmeta>
            <wp:meta_key><![CDATA[_payment_method]]></wp:meta_key>
            <wp:meta_value><![CDATA[paypal2]]></wp:meta_value>
        </wp:postmeta>
</item>
</items>
</rss>"""

from lxml import etree
doc = etree.XML(rss)
ns = { (k if k else "xx"):(v) for k, v in doc.xpath('//namespace::*') }

#searching, for example, for "_payment_method"

for company in doc.xpath("//wp:meta_key[.='_payment_method']/following-sibling::wp:meta_value/text()", namespaces=ns):
    print(company)

输出：

paypal
paypal2