如何在使用 BeautifulSoup 进行抓取时修复此错误？答案

【问题标题】：How to fix this error during scraping using BeautifulSoup?如何在使用 BeautifulSoup 进行抓取时修复此错误？
【发布时间】：2023-01-05 00:52:36
【问题描述】：

我正在尝试使用 BeautifulSoup 进行网络抓取并请求 Python 库。我想从 Hacker News 网站过滤新闻标题，但在实施时显示错误。

import requests
from bs4 import BeautifulSoup

res = requests.get('https://news.ycombinator.com/news')
soup = BeautifulSoup(res.text, 'html.parser')
links = soup.select('.titleline a')
subtext = soup.select('.subtext')


def create_custom_hn(links, subtext):
    hn = []
    for index, item in enumerate(links):
        title = links[index].getText()
        href = links[index].get('href', None)
        votes = subtext[index].select('.score')
        if len(votes):
            points = int(votes[0].getText().replace(' points', ''))
            print(points)
            hn.append({'title': title, 'href': href})
    return hn


print(create_custom_hn(links, subtext))

错误说

votes = subtext[index].select('.score')
            ~~~~~~~^^^^^^^
IndexError: list index out of range

【问题讨论】：

标签： python web-scraping beautifulsoup

【解决方案1】：

尝试更具体地选择您的元素，您选择的 soup.select('.titleline a') 包含您可能想要选择的更多元素：

[<a href="https://sethmlarson.dev/urllib3-in-2022">Urllib3 in 2022</a>,
 <a href="from?site=sethmlarson.dev"><span class="sitestr">sethmlarson.dev</span></a>,...]

例子

import requests
from bs4 import BeautifulSoup

res = requests.get('https://news.ycombinator.com/news')
soup = BeautifulSoup(res.text)

data = []

for e in soup.select('tr.athing'):
    data.append({
        'title':e.select_one('.titleline a').get_text(),
        'url':e.select_one('.titleline a').get('href'),
        'votes':e.find_next(class_='subtext').text.split()[0]
    })
print(data)

输出

[{'title': 'Urllib3 in 2022', 'url': 'https://sethmlarson.dev/urllib3-in-2022', 'votes': '93'}, {'title': 'First public release of Pushup: a new compiler for making web apps in Go', 'url': 'https://github.com/adhocteam/pushup', 'votes': '16'}, {'title': 'Intelligence – A good collection of great OSINT Resources', 'url': 'https://github.com/ARPSyndicate/awesome-intelligence', 'votes': '109'}, {'title': 'Microsoft is preparing to add ChatGPT to Bing', 'url': 'https://www.bloomberg.com/news/articles/2023-01-04/microsoft-hopes-openai-s-chatbot-will-make-bing-smarter', 'votes': '755'}, {'title': 'Juan Tamariz, the godfather of close-up card magic', 'url': 'https://www.nytimes.com/2023/01/02/magazine/juan-tamariz-magic.html', 'votes': '31'}, {'title': 'The Expanding Dark Forest and Generative AI', 'url': 'https://maggieappleton.com/ai-dark-forest', 'votes': '223'}, {'title': 'Irreconcilable differences between local and distributed computing (1994)', 'url': 'https://scholar.harvard.edu/waldo/publications/note-distributed-computing', 'votes': '29'},...]

【讨论】：