Python pandas 解析 html 表以获取隐藏的值和链接答案

【问题标题】：Python pandas parse html table to get hidden values and linksPython pandas 解析 html 表以获取隐藏的值和链接
【发布时间】：2020-03-19 12:46:56
【问题描述】：

这是我尝试使用 Python 使用 Pandas 解析的页面中的 sn-p：

<!DOCTYPE html><html><head><title>website</title><link rel='stylesheet' type='text/css' href='css/global.css'><META HTTP-EQUIV='Content-Type' CONTENT='text/html; charset=UTF-8'></head><body><script src="analyticstracking.js"></script>

</h3><table class='gene'><tr><th>header1<br>info</th>
<th><a href='useful.php#cods'>header2</a><br>info</th><th><a href='useful.php#cods'>header3</a><br>info</th><th><a href='http://www.somelink' target=link onclick="trackOutboundLink('http://www.somelink'); return false;">header4</a><br><span class='td'>info</span></th>
<th><a href='http://www.somelink' target=link onclick="trackOutboundLink('http://www.somelink'); return false;">header5</a><br><span class='td'>info</span></th>
<th>header6<br>info</th><th>header7</th><th>header8</th><th><a href='useful.php'>header9<br>info</a></th></tr>



<tr class='even'><td class='center'><form action='get.php' method='GET'>
    <input type='hidden' name='acc' value='value1'><input type='submit' value='value1'></form></td>
    <td>stuff</td><td>stuff</td><td>stuff</td><td>stuff</td><td class='center'><span class='dm' title='some extra info'>stuff</span> </td><td>stuff</td><td><a href='http://www.link1' target=ref onclick="trackOutboundLink('http://www.link1'); return false;">link1</a><br><span class='td'><a href='http://www.link2' target=ref onclick="trackOutboundLink('http://www.link2'); return false;">link2</span><br><span class='td'><a href='http://www.link3' target=ref onclick="trackOutboundLink('http://www.link3'); return false;">link3</span><br></td><td style='white-space:nowrap;'> <span class='gen' title='extra_info'>stuff</span>  <span class='gen' title='extra_info2'>stuff2</span>   <a href='http://www.out' target='out' title='Link to out' onclick="trackOutboundLink('http://www.out'); return false;"><span class='dbs'>out</span></a> </td></tr>

<tr class='even'><td class='center'><form action='get.php' method='GET'>
    <input type='hidden' name='acc' value='value2'><input type='submit' value='value2'></form></td>
    <td>stuff2</td><td>stuff2</td><td>stuff2</td><td>stuff2</td><td class='center'><span class='dm' title='some extra info'>stuff2</span> </td><td>stuff2</td><td><a href='http://www.link4' target=ref onclick="trackOutboundLink('http://www.link5'); return false;">link4</a><br><span class='td'><a href='http://www.link5' target=ref onclick="trackOutboundLink('http://www.link5'); return false;">link5</span><br></td><td style='white-space:nowrap;'> <span class='gen' title='extra_info'>stuff</span>  <span class='gen' title='extra_info2'>stuff</span>   <a href='http://www.out2' target='out2' title='Link to out2' onclick="trackOutboundLink('http://www.out2'); return false;"><span class='dbs'>out2</span></a> </td></tr>

<tr class='odd'><td class='center'><form action='get.php' method='GET'>
    <input type='hidden' name='acc' value='value3'><input type='submit' value='value3'></form></td>
    <td>stuff3</td><td>stuff3</td><td>stuff3</td><td>stuff3</td><td class='center'><span class='dm' title='extrainfo'>stuff3</span> </td><td>stuff3</td><td><a href='http://www.link6' target=ref onclick="trackOutboundLink('http://www.link6'); return false;">link6</a></td><td style='white-space:nowrap;'> <span class='gen' title='extra_info'>stuff3</span>  <span class='gen' title='extra_info2'>stuff3</span>  </td></tr>

</table>

表格中有隐藏变量（表头6和表头9），鼠标悬停在上面可以看到信息：

当我尝试使用 Pandas 时，我得到以下信息：

with open ("/root/Downloads/adad.html", "r") as content_file:
    f = content_file.read()
dfs = pd.read_html(f)
dfs

我的愿望是获得以下内容：

[   header1info    header2info    header3info    header4info    header5info    header6info         header7          header8                header9info
0   value1         stuff          stuff          stuff          stuff          stuff(extra_info)   stuff            link1(http://link1)    stuff(extra_info) stuff2(extra_info2) out(http://out)
                                                                                                                    link2(http://link2)
                                                                                                                    link3(http://link3)
1   value2         stuff2         stuff2         stuff2         stuff2         stuff2              stuff2           link4(http://link4)    stuff(extra_info) stuff(extra_info2) out2(http://out)
                                                                                                                    link5(http://link5)  
2   value3         stuff3        stuff3          stuff3         stuff3         stuff3              stuff3           link6(http://link6)    stuff3(extra_info) stuff3(extra_info2)]

这可以使用 Pandas 实现吗？如果是，我怎样才能达到预期的输出？

抱歉，我不是 Pandas 方面的专家。我不确定是否还有其他方法可以解析信息。我唯一想到的就是拆分行并获取所需的信息，但您只能想象它是多么的挑剔......

【问题讨论】：

标签： python pandas python-2.x

【解决方案1】：

简短回答：否

pd.read_html() 只读取 html 上生成的文本，而不读取具有属性的元素。为了实现你想要的，你可能想要使用像 bs4 这样的 HTML 解析器，然后找到表 class='gene'，然后遍历其中的 <tr> 和 <td>。代码如下：

import pandas as pd
from bs4 import BeautifulSoup

source = r"""<!DOCTYPE html><html><head><title>website</title><link rel='stylesheet' type='text/css' href='css/global.css'><META HTTP-EQUIV='Content-Type' CONTENT='text/html; charset=UTF-8'></head><body><script src="analyticstracking.js"></script>

</h3><table class='gene'><tr><th>header1<br>info</th>
<th><a href='useful.php#cods'>header2</a><br>info</th><th><a href='useful.php#cods'>header3</a><br>info</th><th><a href='http://www.somelink' target=link onclick="trackOutboundLink('http://www.somelink'); return false;">header4</a><br><span class='td'>info</span></th>
<th><a href='http://www.somelink' target=link onclick="trackOutboundLink('http://www.somelink'); return false;">header5</a><br><span class='td'>info</span></th>
<th>header6<br>info</th><th>header7</th><th>header8</th><th><a href='useful.php'>header9<br>info</a></th></tr>



<tr class='even'><td class='center'><form action='get.php' method='GET'>
    <input type='hidden' name='acc' value='value1'><input type='submit' value='value1'></form></td>
    <td>stuff</td><td>stuff</td><td>stuff</td><td>stuff</td><td class='center'><span class='dm' title='some extra info'>stuff</span> </td><td>stuff</td><td><a href='http://www.link1' target=ref onclick="trackOutboundLink('http://www.link1'); return false;">link1</a><br><span class='td'><a href='http://www.link2' target=ref onclick="trackOutboundLink('http://www.link2'); return false;">link2</span><br><span class='td'><a href='http://www.link3' target=ref onclick="trackOutboundLink('http://www.link3'); return false;">link3</span><br></td><td style='white-space:nowrap;'> <span class='gen' title='extra_info'>stuff</span>  <span class='gen' title='extra_info2'>stuff2</span>   <a href='http://www.out' target='out' title='Link to out' onclick="trackOutboundLink('http://www.out'); return false;"><span class='dbs'>out</span></a> </td></tr>

<tr class='even'><td class='center'><form action='get.php' method='GET'>
    <input type='hidden' name='acc' value='value2'><input type='submit' value='value2'></form></td>
    <td>stuff2</td><td>stuff2</td><td>stuff2</td><td>stuff2</td><td class='center'><span class='dm' title='some extra info'>stuff2</span> </td><td>stuff2</td><td><a href='http://www.link4' target=ref onclick="trackOutboundLink('http://www.link5'); return false;">link4</a><br><span class='td'><a href='http://www.link5' target=ref onclick="trackOutboundLink('http://www.link5'); return false;">link5</span><br></td><td style='white-space:nowrap;'> <span class='gen' title='extra_info'>stuff</span>  <span class='gen' title='extra_info2'>stuff</span>   <a href='http://www.out2' target='out2' title='Link to out2' onclick="trackOutboundLink('http://www.out2'); return false;"><span class='dbs'>out2</span></a> </td></tr>

<tr class='odd'><td class='center'><form action='get.php' method='GET'>
    <input type='hidden' name='acc' value='value3'><input type='submit' value='value3'></form></td>
    <td>stuff3</td><td>stuff3</td><td>stuff3</td><td>stuff3</td><td class='center'><span class='dm' title='extrainfo'>stuff3</span> </td><td>stuff3</td><td><a href='http://www.link6' target=ref onclick="trackOutboundLink('http://www.link6'); return false;">link6</a></td><td style='white-space:nowrap;'> <span class='gen' title='extra_info'>stuff3</span>  <span class='gen' title='extra_info2'>stuff3</span>  </td></tr>

</table>"""

soup = BeautifulSoup(source, 'html.parser')

table = soup.findAll("table", {"class": "gene"})
trs = table[0].findAll("tr")

headers = []
for th in trs[0].findAll("th"):
    headers.append(th.text)
rows = []
for i in range(1, len(trs)):
    tds = []
    for td in trs[i].findAll("td"):
        a = td.findAll("a")
        spans = td.findAll("span")
        inputs = td.findAll("input")
        ret = ""
        if len(a) != 0 or len(spans) != 0 or len(inputs) != 0:
            if len(a) != 0:
                for link in a:
                    ret += link.text + '('+link['href']+') '
            if len(spans) != 0:
                for span in spans:
                    if span.has_attr('title'):
                        ret += span.text + '('+span['title']+') '
            if len(inputs) != 0:
                for inp in inputs:
                    if inp.has_attr('value'):
                        if inp.has_attr('type'):
                            if inp['type'] == "hidden":
                                ret +=  inp['value']
        else:
            ret = td.text if td.text != '' and td.text != '\n' else "NaN"
        tds.append(ret)
    rows.append(tds)
    
df = pd.DataFrame(rows, columns = headers)

df

【讨论】：

感谢您提供的答案。我会稍等片刻，看看是否有人可以在接受之前提出另一个答案。
当然，如果您不确定，可以查看official docs。无论如何，使用bs4解析器并不挑剔，您可以从结构清晰的html文件中提取信息（只要html是有效的），并且可能不需要10行代码。
您能否提供一个bs4 代码，该代码将与我的原始帖子一起使用并打印所需的输出？
几乎！值仍在为header1 打印NaN
让我们continue this discussion in chat.