如何将抓取的 URL 传递回 BeautifulSoup？答案

【问题标题】：How do I pass a scraped URL back into BeautifulSoup?如何将抓取的 URL 传递回 BeautifulSoup？
【发布时间】：2021-01-18 16:04:11
【问题描述】：

我已经抓取了一个页面并收集了网站上的 URL。然后我试图将找到的 URL 传回 BS 以打开它们并抓取它们以获取

。这可能是我想念的超级简单的事情，对此我深表歉意。我上周才开始使用python。

这是我的代码。

    from bs4 import BeautifulSoup
    import requests

    # gets website
    html_text = requests.get('https://www.marketwatch.com/latest-news?mod=top_nav').text
    # uses BeautifulSoup to read the html
    website = BeautifulSoup(html_text, 'lxml')
    # searches the html that BS read into py
    stories = website.find('div', class_='collection__elements j-scrollElement')
    # searches stories for a single story
    story = stories.find('div', class_='element element--article')
    # searches story for the headline
    article_headline = story.find('h3', class_='article__headline').text
    # get link to article from the head line to then pass through to get the articles content
    possible_links = story.find_all('a', class_='link')
    for link in possible_links:
        linkArt = (link.get('href'))
    print('Link:' + linkArt) # make sure expected link was found

    # Pass found link back into a request (part that doesn't work)
    article_html = requests.get(linkArt).text
    article = BeautifulSoup(article_html, ' lxml')

    article_wrapper = article.find('div', class_ = 'column column--full article__content')
    article_content = article_wrapper.find_all('p').text

这是我的错误：链接：https://www.marketwatch.com/articles/global-stocks-edge-lower-with-biden-adopting-traditional-dollar-stance-chinese-equities-gain-after-gdp-report-51610967473?mod=newsviewer_click 回溯（最近一次通话最后）：文件“C:\Users\MLG420\PycharmProjects\scraper\scrape.py”，第 23 行，在文章 = BeautifulSoup(article_html, 'lxml') init 中的文件“C:\Users\MLG420\PycharmProjects\scraper\venv\lib\site-packages\bs4_init_.py”，第 243 行提高 FeatureNotFound( bs4.FeatureNotFound：找不到具有您请求的功能的树生成器：lxml。需要安装解析器库吗？

【问题讨论】：

您有一个简单的错字：article = 行中 lxml 之前的一个额外空格。
facepalm....谢谢！

标签： python html beautifulsoup

【解决方案1】：

有 2 个错误：" lxml" 中有一个空格破坏了article_html 声明中的代码，并且在最后一行中，您在列表中调用了.text。这是更正后的代码：

from bs4 import BeautifulSoup
import requests
import lxml

# gets website
html_text = requests.get('https://www.marketwatch.com/latest-news?mod=top_nav').text
# uses BeautifulSoup to read the html
website = BeautifulSoup(html_text, 'lxml')
# searches the html that BS read into py
stories = website.find('div', class_='collection__elements j-scrollElement')
# searches stories for a single story
story = stories.find('div', class_='element element--article')
# searches story for the headline
article_headline = story.find('h3', class_='article__headline').text
# get link to article from the head line to then pass through to get the articles content
possible_links = story.find_all('a', class_='link')
for link in possible_links:
    linkArt = (link.get('href'))
print('Link:' + linkArt) # make sure expected link was found

# Pass found link back into a request (part that doesn't work)
article_html = requests.get(linkArt).text
article = BeautifulSoup(article_html, 'lxml')

article_wrapper = article.find('div', class_ = 'column column--full article__content')
article_content = article_wrapper.find_all('p')
for elem in article_content:
    print(elem.text)  # or do something else

【讨论】：