使用 Beautifulsoup4 查找 .nextsibling

Question

我正在尝试从 URL 获取表格的（某些）内容。 到目前为止，我已经设法获得了页面的两个所需内容，但还有第三个（第三列）我只想获得它的文本。问题是，底层链接存在于页面的其他地方（具有不同的文本），如果我想将表加载到 SQL 数据库中，第三列的内容将与前两列不匹配。

import urllib2
from bs4 import BeautifulSoup4
startURL = "http://some.url/website.html"
page = urllib2.urlopen(startURL).read()
soup = BeautifulSoup(page, "html.parser")
for links in soup.findAll("a"):
    if "href" in links.attrs:
        www = links.attrs.values()
        if not "https://" in www[0]:  # to exclude all non-relative links, e.g. external links
            if "view/" in www[0]: # To get only my desired links of column 1
                link_of_column1 = www[0]   # this is now my wanted link

好的，所以通过这段代码我可以得到第二列。我必须在哪里以及如何应用 .nextsibling() 函数才能获得下一个（第 3 列）列中的下一个链接？

编辑： 有人问我：网址是https://myip.ms/browse/web_hosting/World_Web_Hosting_Global_Statistics.html，我想从第 2 列和第 3 列获取内容，即“托管公司”（链接文本和链接）和“国家”（仅文本）。

编辑2： 我忘记的另一件事...如何提取其137,157 records 的信息？

Answer 1

首先使用id=web_hosting_tbl 属性找到包含所有信息的表。然后遍历表的所有行。但是，如果您查看页面源代码，您需要的行不是连续的，而是交替的，并且它们没有任何类名。另外，表格的第一行是标题行，所以我们必须跳过它。

获取所需行后（使用table.find_all('tr')[1::2]），找到所有列，然后从对应的列中获取所需信息。

代码：

import requests
from bs4 import BeautifulSoup

r = requests.get('https://myip.ms/browse/web_hosting/World_Web_Hosting_Global_Statistics.html')
soup = BeautifulSoup(r.text, 'lxml')

table = soup.find('table', id='web_hosting_tbl')
for row in table.find_all('tr')[1::2]:
    all_columns = row.find_all('td')
    name = all_columns[1].a.text
    link = all_columns[1].a['href']
    country = all_columns[2].a.text
    print(name, link, country, sep=' | ')

部分输出：

Godaddy.com, LLC | /view/web_hosting/2433/Godaddy_com_LLC.html | USA
Cloudflare, Inc | /view/web_hosting/4638/Cloudflare_Inc.html | USA
Amazon.com, Inc | /view/web_hosting/615/Amazon_com_Inc.html | USA
Ovh Sas | /view/web_hosting/7593/Ovh_Sas.html | France
Hetzner Online Ag | /view/web_hosting/45081/Hetzner_Online_Ag.html | Germany
Hostgator.com Llc | /view/web_hosting/26757/Hostgator_com_Llc.html | USA
Google Inc | /view/web_hosting/617/Google_Inc.html | USA
Bluehost Inc | /view/web_hosting/3886/Bluehost_Inc.html | USA
...

Answer 2

代码：（Python 3.6+，使用f-strings）

import urllib.parse
from collections import namedtuple
from datetime import datetime

import bs4
import requests

HostingCompany = namedtuple('HostingCompany',
                            ('name', 'country', 'websites', 'usage', 'usage_by_top', 'update_time'))


class MyIpLink:
    url_base = 'https://myip.ms'

    def __init__(self, tag: bs4.element.Tag, *, is_anchor=False):
        a_tag = tag.find('a')

        if is_anchor:  # treat `tag` as an anchor tag
            a_tag = tag

        self.text = tag.text.strip()
        self.url = urllib.parse.urljoin(self.url_base, a_tag['href'])

    def __repr__(self):
        return f'{self.__class__.__name__}(text={repr(self.text)}, url={repr(self.url)})'


url = 'https://myip.ms/browse/web_hosting/World_Web_Hosting_Global_Statistics.html'
html = requests.get(url).text
soup = bs4.BeautifulSoup(html, 'html.parser')

rows = soup.select('#web_hosting_tbl > tbody > tr')[::2]  # skips "more info" rows
companies = []

for row in rows:
    tds = row.find_all('td')

    name = MyIpLink(tds[1])
    country = MyIpLink(tds[2])
    websites = [MyIpLink(a, is_anchor=True) for a in tds[3].find_all('a')]
    usage = MyIpLink(tds[4])
    usage_by_top = MyIpLink(tds[5])
    update_time = datetime.strptime(tds[6].text.strip(), '%d %b %Y, %H:%M')

    company = HostingCompany(name, country, websites, usage, usage_by_top, update_time)
    companies.append(company)

import pprint
pprint.pprint(companies)

print(companies[0].name.text)
print(companies[0].name.url)
print(companies[0].country.text)

输出：

[HostingCompany(name=MyIpLink(text='Godaddy.com, LLC', url='https://myip.ms/view/web_hosting/2433/Godaddy_com_LLC.html'), country=MyIpLink(text='USA', url='https://myip.ms/view/best_hosting/USA/Best_Hosting_in_USA.html'), websites=[MyIpLink(text='www.godaddy.com', url='https://myip.ms/go.php?1229687315_ITg7Im93dCkWE0kNAhQSEh0FUeHq5Q==')], usage=MyIpLink(text='512,701 sites', url='https://myip.ms/browse/sites/1/ownerID/2433/ownerIDii/2433'), usage_by_top=MyIpLink(text='951 sites', url='https://myip.ms/browse/sites/1/rankii/100000/ownerID/2433/ownerIDii/2433'), update_time=datetime.datetime(2018, 5, 2, 5, 17)),
 HostingCompany(name=MyIpLink(text='Cloudflare, Inc', url='https://myip.ms/view/web_hosting/4638/Cloudflare_Inc.html'), country=MyIpLink(text='USA', url='https://myip.ms/view/best_hosting/USA/Best_Hosting_in_USA.html'), websites=[MyIpLink(text='www.cloudflare.com', url='https://myip.ms/go.php?840626136_OiEsK2ROSxAdGl4QGhYJG+Tp6fnrv/f49w==')], usage=MyIpLink(text='488,119 sites', url='https://myip.ms/browse/sites/1/ownerID/4638/ownerIDii/4638'), usage_by_top=MyIpLink(text='16,160 sites', url='https://myip.ms/browse/sites/1/rankii/100000/ownerID/4638/ownerIDii/4638'), update_time=datetime.datetime(2018, 5, 2, 5, 10)),
 HostingCompany(name=MyIpLink(text='Amazon.com, Inc', url='https://myip.ms/view/web_hosting/615/Amazon_com_Inc.html'), country=MyIpLink(text='USA', url='https://myip.ms/view/best_hosting/USA/Best_Hosting_in_USA.html'), websites=[MyIpLink(text='www.amazonaws.com', url='https://myip.ms/go.php?990446041_JyYhKGFxThMQHUMRHhcDExHj8vul7f75')], usage=MyIpLink(text='453,230 sites', url='https://myip.ms/browse/sites/1/ownerID/615/ownerIDii/615'), usage_by_top=MyIpLink(text='9,557 sites', url='https://myip.ms/browse/sites/1/rankii/100000/ownerID/615/ownerIDii/615'), update_time=datetime.datetime(2018, 5, 2, 5, 4)),
 ...
]

Godaddy.com, LLC
https://myip.ms/view/web_hosting/2433/Godaddy_com_LLC.html
USA

晚上会用一些解释更新答案。干杯！

Answer 3

试试下面的方法。它应该为您提供来自column 2 的文本、来自column 2 的链接以及来自该表的column 3 的文本。我使用lxml 而不是BeautifulSoup 以使其更快。谢谢。

import requests
from urllib.parse import urljoin
from lxml.html import fromstring

URL = 'https://myip.ms/browse/web_hosting/World_Web_Hosting_Global_Statistics.html'

res = requests.get(URL)
root = fromstring(res.text)

for items in root.cssselect('#web_hosting_tbl tr:not(.expand-child)')[1:]:
    name = items.cssselect("td.row_name a")[0].text
    link = urljoin(URL,items.cssselect("td.row_name a")[0].attrib['href'])
    country = items.cssselect("td a[href^='/view/best_hosting/']")[0].text
    print(name, link, country)

结果：

Godaddy.com, LLC https://myip.ms/view/web_hosting/2433/Godaddy_com_LLC.html USA
Cloudflare, Inc https://myip.ms/view/web_hosting/4638/Cloudflare_Inc.html USA
Amazon.com, Inc https://myip.ms/view/web_hosting/615/Amazon_com_Inc.html USA
Ovh Sas https://myip.ms/view/web_hosting/7593/Ovh_Sas.html France
Hetzner Online Ag https://myip.ms/view/web_hosting/45081/Hetzner_Online_Ag.html Germany
Hostgator.com Llc https://myip.ms/view/web_hosting/26757/Hostgator_com_Llc.html USA
Google Inc https://myip.ms/view/web_hosting/617/Google_Inc.html USA