如何从 python 中的 <td> 表中抓取 url答案

【问题标题】：How to scrape url from <td> table in python如何从 python 中的 <td> 表中抓取 url
【发布时间】：2021-07-27 18:27:06
【问题描述】：

我正在使用 python 和 beautifulsoup 来刮一张桌子。我想从下面的 html 表中抓取 url：

<tbody>
   <tr>
      <td colspan="4" style="height:10px"></td>
   </tr>
   <tr class="header" id="a">
      <td class="w40 hidden-xs hidden-sm hidden-xxs"> </td>
      <td>A</td>
      <td><a class="fa fa-angle-up goToTop pull-right" href="#" onclick="$('html, body').animate({scrollTop: 0}, 1000);return false;" title="Scroll to top"></a></td>
      <td class="w40 hidden-xs hidden-sm hidden-xxs"> </td>
   </tr>
   <tr>
      <td class="w40 hidden-xs hidden-sm hidden-xxs"> </td>
      <td colspan="2"><a data-iata="ARD" data-lat="-8.13234" data-lon="124.597" href="https://www.flightradar24.com/data/airports/ard" title="Alor Island Airport"><img class="icon-airport" src="https://www.flightradar24.com/static/images/airport_pin_40_blue.png"/> Alor Island Airport <small>(ARD/WATM)</small> </a> </td>
      <td class="w40 hidden-xs hidden-sm hidden-xxs"> </td>
   </tr>
   <tr>
      <td class="w40 hidden-xs hidden-sm hidden-xxs"> </td>
      <td colspan="2"><a data-iata="AMQ" data-lat="-3.71026" data-lon="128.089096" href="https://www.flightradar24.com/data/airports/amq" title="Ambon Pattimura Airport"><img class="icon-airport" src="https://www.flightradar24.com/static/images/airport_pin_40_blue.png"/> Ambon Pattimura Airport <small>(AMQ/WAPP)</small> </a> <span class="pull-right">Rating: 79%</span> </td>
      <td class="w40 hidden-xs hidden-sm hidden-xxs"> </td>
   </tr>
   <tr>
      <td class="w40 hidden-xs hidden-sm hidden-xxs"> </td>
      <td colspan="2"><a data-iata="ABU" data-lat="-9.07444" data-lon="124.904404" href="https://www.flightradar24.com/data/airports/abu" title="Atambua Haliwen Airport"><img class="icon-airport" src="https://www.flightradar24.com/static/images/airport_pin_40_blue.png"/> Atambua Haliwen Airport <small>(ABU/WATA)</small> </a> </td>
      <td class="w40 hidden-xs hidden-sm hidden-xxs"> </td>
   </tr>
   <tr class="header" id="b">
      <td class="w40 hidden-xs hidden-sm hidden-xxs"> </td>
      <td>B</td>
      <td><a class="fa fa-angle-up goToTop pull-right" href="#" onclick="$('html, body').animate({scrollTop: 0}, 1000);return false;" title="Scroll to top"></a></td>
      <td class="w40 hidden-xs hidden-sm hidden-xxs"> </td>
   </tr>
   <tr>
      <td class="w40 hidden-xs hidden-sm hidden-xxs"> </td>
      <td colspan="2"><a data-iata="BXB" data-lat="-2.53224" data-lon="133.438797" href="https://www.flightradar24.com/data/airports/bxb" title="Babo Airport"><img class="icon-airport" src="https://www.flightradar24.com/static/images/airport_pin_40_blue.png"/> Babo Airport <small>(BXB/WASO)</small> </a> </td>
      <td class="w40 hidden-xs hidden-sm hidden-xxs"> </td>
   </tr>
   <tr>
      <td class="w40 hidden-xs hidden-sm hidden-xxs"> </td>
      <td colspan="2"><a data-iata="BJW" data-lat="-8.7125" data-lon="121.0625" href="https://www.flightradar24.com/data/airports/bjw" title="Bajawa Turelelo Soa Airport"><img class="icon-airport" src="https://www.flightradar24.com/static/images/airport_pin_40_blue.png"/> Bajawa Turelelo Soa Airport <small>(BJW/WATB)</small> </a> </td>
      <td class="w40 hidden-xs hidden-sm hidden-xxs"> </td>
   </tr>
   <tr>
      <td class="w40 hidden-xs hidden-sm hidden-xxs"> </td>
      <td colspan="2"><a data-iata="BPN" data-lat="-1.26827" data-lon="116.894402" href="https://www.flightradar24.com/data/airports/bpn" title="Balikpapan Sepinggan Airport"><img class="icon-airport" src="https://www.flightradar24.com/static/images/airpo,...

我想连同这个 url 一起抓取整个表格，这样我抓取的数据就变成了这个字典：

{"Alor Island Airport":"https://www.flightradar24.com/data/airports/ard",
"Ambon Pattimura Airport":"https://www.flightradar24.com/data/airports/amq",
"Atambua Haliwen Airport":"https://www.flightradar24.com/data/airports/abu",
"Babo Airport":"https://www.flightradar24.com/data/airports/bxb",
"Bajawa Turelelo Soa Airport":"https://www.flightradar24.com/data/airports/bjw",
"Balikpapan Sepinggan Airport":"https://www.flightradar24.com/data/airports/bpn"}

代码如下。

bs=BeautifulSoup(page.content, 'html.parser')
table_body=bs.find('tbody')
rows = table_body.find_all('tr')
for row in rows:
    cols=row.find_all('td')
    for link in cols:
        a = link.get("href")
        print(a)

但是我得到了None 在 Python 中有什么方法可以做到吗？

【问题讨论】：

标签： python xml csv beautifulsoup web-crawler

【解决方案1】：

您缺少一个循环。 href 包含在 a 标记中

下面的代码正确输出

from bs4 import BeautifulSoup
bs=BeautifulSoup(html, 'html.parser')
table_body=bs.find('tbody')
rows = table_body.find_all('tr')
for row in rows:
    cols=row.find_all('td')
    for col in cols:
        a_list = col.find_all('a')
        for a in a_list:
            href = a.get("href")
            print(href)

输出

#
https://www.flightradar24.com/data/airports/ard
https://www.flightradar24.com/data/airports/amq
https://www.flightradar24.com/data/airports/abu
#
https://www.flightradar24.com/data/airports/bxb
https://www.flightradar24.com/data/airports/bjw
https://www.flightradar24.com/data/airports/bpn

【讨论】：

【解决方案2】：

要从<td> 中抓取所有<href>，您可以使用CSS 选择器：tbody td a[data-iata][href]，这意味着“所有a 都具有data-iata 属性，该属性在td 下包含href tbody"

soup = BeautifulSoup(html, "html.parser")

for tag in soup.select("tbody td a[data-iata][href]"):
    print(tag["href"])

输出：

https://www.flightradar24.com/data/airports/ard
https://www.flightradar24.com/data/airports/amq
https://www.flightradar24.com/data/airports/abu
https://www.flightradar24.com/data/airports/bxb
https://www.flightradar24.com/data/airports/bjw
https://www.flightradar24.com/data/airports/bpn

【讨论】：