从嵌套的 xml 文件创建 pandas 数据框

Question

这是一个 xml 文件的一小部分。我想用这个创建一个数据库，每个标签都有唯一的列名和非重复数据。

尝试使用 lxml，到目前为止我能做的最好的事情是创建一个数据框，结果如下：

"    
SRCSGT
DATE    11112017
AGENCY  Department of Veterans Affairs
OFFICE  Canandaigua VAMC   
LOCATION    Department of Veterans Affairs Medical Center
ZIP 14424
etc, etc, "

xml

<?xml version="1.0" encoding="UTF-8"?>
<NOTICES>
  <SRCSGT>
    <DATE>11112017</DATE>
    <AGENCY><![CDATA[Department of Veterans Affairs]]></AGENCY>
    <OFFICE><![CDATA[Canandaigua VAMC]]></OFFICE>
    <LOCATION><![CDATA[Department of Veterans Affairs Medical Center]]></LOCATION>
    <ZIP>14424</ZIP>
    <CLASSCOD>H</CLASSCOD>
    <NAICS>238210</NAICS>
    <OFFADD><![CDATA[Department of Veterans Affairs;400 Fort Hill Ave.;Canandaigua NY  14424]]></OFFADD>
    <SUBJECT><![CDATA[H--3 YEAR TESTING/MAINTENANCE OF ELECTRICAL EQUIPMENT AT THE SYRACUSE VA MEDICAL CENTER AND THE ROME COMMUNITY BASED OUTPATIENT CLINIC  ]]></SUBJECT>
    <SOLNBR><![CDATA[9069]]></SOLNBR>
    <RESPDATE>11172017</RESPDATE>
    <ARCHDATE>12172017</ARCHDATE>
    <CONTACT><![CDATA[COiyiyS, JUhhiuN<a href="mailto:Juggyui@va.gov">CONTRACT SPECIALIST</a>]]></CONTACT>
    <DESC><![CDATA[This is a Sources Sought Notice. (a) The Government does not intend to award a contract on the basis of this Sources Sought or to otherwise pay for the information solicited.(b) Although "proposal," "offeror," contractor, and "offeror" may be used in this sources sought notice, any response will be treated as information only. It shall not be used as a proposal.Attachment(s) if applicable. ]]></DESC>
    <LINK><![CDATA[https://www.fbo.gov/spg/VA/CaVAMC532/CaVAMC532/9069/listing.html]]></LINK>
    <EMAIL>
      <ADDRESS><![CDATA[Jigjhgjas@va.gov]]></ADDRESS>
      <DESC><![CDATA[CONTRACT SPECIALIST]]></DESC>
    </EMAIL>
    <SETASIDE>N/A</SETASIDE>
    <RECOVERY_ACT>N</RECOVERY_ACT>
    <DOCUMENT_PACKAGES>
      <PACKAGE><![CDATA[Attachment]]></PACKAGE>
    </DOCUMENT_PACKAGES>
  </SRCSGT>
</NOTICES>

---

我写的代码

from lxml import etree as et
import pandas as pd

trees = et.parse('test.xml') #get xml file
root = trees.getroot() #get to root of file

tags = [] #list for holding all tags
datas = [] #list for holding all data in tags


for child in root: #root is a list of all elements in the xml file
    #print(child.tag)
    tt = child.tag #reads xml tag
    tags.append(tt)
    datas.append(child.text) #read xml tag data
    for c in child.findall('./'): # ./ finds children
        tt1 = c.tag
        tags.append(str(tt1))
        datas.append(c.text)
        for i in c.findall('./'): #each child node loads a new list of elements
            tt2 = i.tag
            tags.append(str(tt1)+ '_' + str(tt2))
            datas.append(i.text)
            for j in i.findall('./'):
                tt3 = j.tag
                tags.append(str(tt1)+ '_' + str(tt2) + '_' + str(tt3))
                datas.append(j.text)
                for k in j.findall('./'):
                    tt4 = k.tag
                    tags.append(str(tt1)+ '_' + str(tt2) + '_' + str(tt3) + '_' + str(tt4))
                    datas.append(k.text)

df = pd.DataFrame({"tags": tags,"values": datas})

---

想要的解决方案是这样的

 date agency office
1/1/10  A1    O1
1/1/10  A2    O2
1/1/10  A3    O3

所以基本上标签应该变成列标题并且必须被填充。列名不应重复，以便创建标准数据库表。

Answer 1

考虑嵌套的 xpath 循环，首先循环遍历每个 <SCRSGT> 节点，然后使用内部字典提取所有 SCRSGT 的子节点，该字典迭代地附加到 DataFrame 调用的列表中：

from lxml import etree as et
import pandas as pd

trees = et.parse('test.xml')

d = []
for srcsgt in trees.xpath('//SRCSGT'):     # ITERATE THROUGH ROOT'S CHILDREN
    inner = {}
    for elem in srcsgt.xpath('//*'):       # ITERATE THROUGH ROOT'S DESCENDANTS PER CHILD
        if len(elem.text.strip()) > 0:     # KEEP ONLY NODES WITH NON-ZERO LENGTH TEXT
            inner[elem.tag] = elem.text

    d.append(inner)

df = pd.DataFrame(d)

输出

print(df)

#             ADDRESS                          AGENCY  ARCHDATE CLASSCOD  \
# 0  Jigjhgjas@va.gov  Department of Veterans Affairs  12172017        H   

#                                              CONTACT      DATE  \
# 0  COiyiyS, JUhhiuN<a href="mailto:Juggyui@va.gov...  11112017   

#                   DESC                                               LINK  \
# 0  CONTRACT SPECIALIST  https://www.fbo.gov/spg/VA/CaVAMC532/CaVAMC532...   

#                                         LOCATION   NAICS  \
# 0  Department of Veterans Affairs Medical Center  238210   

#                                               OFFADD            OFFICE  \
# 0  Department of Veterans Affairs;400 Fort Hill A...  Canandaigua VAMC   

#       PACKAGE RECOVERY_ACT  RESPDATE SETASIDE SOLNBR  \
# 0  Attachment            N  11172017      N/A   9069   

#                                              SUBJECT    ZIP  
# 0  H--3 YEAR TESTING/MAINTENANCE OF ELECTRICAL EQ...  14424