网页抓取：抓取页面并在 DataFrame 中存储内容

Question

以下代码可用于为三个给定的示例 url 重现网络抓取任务：

代码：

import pandas as pd
import requests
import urllib.request
from bs4 import BeautifulSoup

# Would otherwise load a csv file with 100+ urls into a DataFrame
# Example data:
links = {'url': ['https://www.apple.com/education/', 'https://www.apple.com/business/', 'https://www.apple.com/environment/']}
urls = pd.DataFrame(data=links)

def scrape_content(url):

    r = requests.get(url)
    html = r.content
    soup = BeautifulSoup(html,"lxml")

    # Get page title
    title = soup.find("meta",attrs={"property":"og:title"})["content"].strip()
    # Get content from paragraphs
    content = soup.find("div", {"class":"section-content"}).find_all('p')

    print(title)

    for p in content:
        p = p.get_text(strip=True)
        print(p)

对每个网址应用抓取：

urls['url'].apply(scrape_content)

输出：

Education
Every child is born full of creativity. Nurturing it is one of the most important things educators do. Creativity makes your students better communicators and problem solvers. It prepares them to thrive in today’s world — and to shape tomorrow’s. For 40 years, Apple has helped teachers unleash the creative potential in every student. And today, we do that in more ways than ever. Not only with powerful products, but also with tools, inspiration, and curricula to help you create magical learning experiences.
Watch the keynote
Business
Apple products have always been designed for the way we work as much as for the way we live. Today they help employees to work more simply and productively, solve problems creatively, and collaborate with a shared purpose. And they’re all designed to work together beautifully. When people have access to iPhone, iPad, and Mac, they can do their best work and reimagine the future of their business.
Environment
We strive to create products that are the best in the world and the best for the world. And we continue to make progress toward our environmental priorities. Like powering all Apple facilities worldwide with 100% renewable energy. Creating the next innovation in recycling with Daisy, our newest disassembly robot. And leading the industry in making our materials safer for people and for the earth. In every product we make, in every innovation we create, our goal is to leave the planet better than we found it. Read the 2018 Progress Report

0    None
1    None
2    None
Name: url, dtype: object

问题：

1. 代码目前只输出每页第一段的内容。我喜欢获取给定选择器中每个 p 的数据。
2. 对于最终数据，我需要一个包含 url、标题和内容的数据框。因此，我想知道如何将抓取的信息写入数据框。

感谢您的帮助。

Answer 1

你的问题出在这一行：

content = soup.find("div", {"class":"section-content"}).find_all('p')

find_all() 正在获取所有 <p> 标签，但仅在结果中 .find() - 它只返回符合条件的第一个示例。所以你在第一个div.section_content 中获得了所有<p> 标签。目前尚不清楚您的用例的正确标准是什么，但如果您只想要所有可以使用的 <p> 标签：

content = soup.find_all('p')

然后你可以让scrape_urls()合并<p>标签文本并连同标题一起返回：

content = '\r'.join([p.get_text(strip=True) for p in content])
return title, content

在函数之外，你可以构建数据框：

url_list = urls['url'].tolist()
results = [scrape_url(url) for url in url_list]
title_list = [r[0] for r in results]
content_list = [r[1] for r in results]
df = pd.DataFrame({'url': url_list, 'title': title_list, 'content': content_list})