尝试使用python从html代码中抓取数据[关闭]

Question

我一直在为网站制作网络爬虫，我想使用 .findall 或其他可行的方法从 html 表中提取所有节点编号，但我很难得到它，我经常遇到错误显然没有放置正确的标签。

谁能帮忙，html代码如下

</div>

<table class="dataTable" cellpadding="5" cellspacing="0" rules="all" border="1" id="ctl00_ContentPlaceHolder1_dgNodes" style="border-collapse:collapse;">
    <tr class="header noBreak">
        <td>&nbsp;</td><td><a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$dgNodes$ctl00$ctl00','')">Node Name</a></td><td><a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$dgNodes$ctl00$ctl01','')">Description</a></td><td><a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$dgNodes$ctl00$ctl02','')">MAC Address</a></td><td><a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$dgNodes$ctl00$ctl03','')"></a>
                <a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$dgNodes$ctl00$liNodeRoleHeader','')" id="ctl00_ContentPlaceHolder1_dgNodes_ctl00_liNodeRoleHeader">Node Role</a>
            </td><td><a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$dgNodes$ctl00$ctl04','')">Firmware</a></td><td>
                <a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$dgNodes$ctl00$lbUptimeHeader','')" id="ctl00_ContentPlaceHolder1_dgNodes_ctl00_lbUptimeHeader">Uptime</a>
            </td><td><a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$dgNodes$ctl00$ctl05','')">Users</a></td>
    </tr><tr onmouseover="this.className = 'highlightedRow';" onmouseout="this.className = 'normalRow';" onclick="GoToNodePage('522');" style="height:18px;">

我需要在代码的最后一行提取数字 522 和所有其他 gotonodepage 数字，但我无法弄清楚，感谢任何帮助。我也想把提取出来的数字放到以后使用的列表中。

r2 = s2.get(webpage)
bsobjswap = BeautifulSoup(r2.content)

listy = []
for link in bsobjswap.findall('tr'):
    if 'onclick' in link.attrs:
        listy.append(link)
print (listy)

错误是 对于 bsobjswap.findall('tr') 中的链接： TypeError: 'NoneType' 对象不可调用

Answer 1

试试这样的：

from bs4 import BeautifulSoup

xml = """<table class="dataTable" cellpadding="5" cellspacing="0" rules="all" border="1" id="ctl00_ContentPlaceHolder1_dgNodes" style="border-collapse:collapse;">
    <tr class="header noBreak">
        <td>&nbsp;</td><td><a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$dgNodes$ctl00$ctl00','')">Node Name</a></td><td><a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$dgNodes$ctl00$ctl01','')">Description</a></td><td><a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$dgNodes$ctl00$ctl02','')">MAC Address</a></td><td><a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$dgNodes$ctl00$ctl03','')"></a>
                <a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$dgNodes$ctl00$liNodeRoleHeader','')" id="ctl00_ContentPlaceHolder1_dgNodes_ctl00_liNodeRoleHeader">Node Role</a>
            </td><td><a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$dgNodes$ctl00$ctl04','')">Firmware</a></td><td>
                <a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$dgNodes$ctl00$lbUptimeHeader','')" id="ctl00_ContentPlaceHolder1_dgNodes_ctl00_lbUptimeHeader">Uptime</a>
            </td><td><a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$dgNodes$ctl00$ctl05','')">Users</a></td>
    </tr><tr onmouseover="this.className = 'highlightedRow';" onmouseout="this.className = 'normalRow';" onclick="GoToNodePage('522');" style="height:18px;">"""

soup = BeautifulSoup(xml)
print([i.get('onclick') for i in soup.findAll('tr', attrs={'onclick':True})])

这将返回["GoToNodePage('522');"]

例如，您可以从这里使用正则表达式提取数字

print([re.findall("\d+", i.get('onclick')) for i in soup.findAll('tr', attrs={'onclick':True})])

这将返回[['522']]