使用 python、BeautifulSoup、pandas 将抓取结果输出到多个 .csv 文件中？

Question

我正在从多个搜索下的多个页面中抓取链接，并希望将抓取的结果输出到多个 .csv 文件中。该表显示了 .csv 文件，其中列出了我的源 URL 和所需的输出文件名：

url	outputfile
https://www.marketresearch.com/search/results.asp?categoryid=230&qtype=2&publisher=IDCs&datepub=0&submit2=Search	outputPS1xIDC.csv
https://www.marketresearch.com/search/results.asp?categoryid=90&qtype=2&publisher=IDC&datepub=0&submit2=Search	outputPS2xIDC.csv
https://www.marketresearch.com/search/results.asp?categoryid=233&qtype=2&publisher=IDC&datepub=0&submit2=Search	outputPS3xIDC.csv
https://www.marketresearch.com/search/results.asp?categoryid=169&qtype=2&publisher=IDC&datepub=0&submit2=Search	outputPS4xIDC.csv

现在，使用下面的代码，我设法按顺序读取了 url，其余代码也运行良好（当我直接指定输出文件名时）。但是，它只输出列表中 4 个页面中的最后一个，因此每次都会覆盖结果。我真正想要的是将结果从第一个 url 输出到第一个输出文件，第二个到第二个等。 （当然，我的实际源 URL 列表比这 4 个要长得多）。

请帮忙，尤其是最后一行，因为显然只是在那里写 [outputs] 是行不通的。

import requests
from bs4 import BeautifulSoup
import pandas as pd
import csv

with open('inputs.csv', newline='') as csvfile:
    reader = csv.DictReader(csvfile)
    urls = [row["url"] for row in reader]
    outputs = [row["outputfile"] for row in reader]

    data = []

    for url in urls:

        def scrape_it(url):
            page = requests.get(url, headers={'Cookie': 'ResultsPerPage=100'})
            soup = BeautifulSoup(page.text, 'html.parser')
            nexturl = soup.find_all(class_="standardLinkDkBlue")[-1]['href']
            stri = soup.find_all(class_="standardLinkDkBlue")[-1].string
            reports = soup.find_all("tr", {"class": ["SearchTableRowAlt", "SearchTableRow"]})

            for report in reports:
                data.append({
                    'title': report.find('a', class_='linkTitle').text,
                    'price': report.find('div', class_='resultPrice').text,
                    'date_author': report.find('div', class_='textGrey').text.replace(' | published by: TechNavio', ''),
                    'detail_link': report.a['href']
                })

            if 'next' not in stri:
                print("All pages completed")
            else:
                scrape_it(nexturl)


    scrape_it(url)
    myOutput = pd.DataFrame(data)
    myOutput.to_csv([outputs], header=False) #works (but only for the last url) if instead of [outputs] I have f'filename.csv'

Answer 1

我没有 Pandas，我也不想运行您的输入，但是当我查看您的代码时，有几件事让我很震惊：

1. 您似乎没有循环遍历url 和output 一起。看起来您循环遍历所有 URL，然后在所有这些循环之后编写一次。
2. 同样，data 只是附加了 HTML 表格数据，它永远不会为每个单独的 URL 重置。

无法运行它，我推荐这样的东西。抓取完全封装并与循环分离，因此您现在可以更清楚地看到输入和输出的流程：

import requests
from bs4 import BeautifulSoup
import csv
import pandas as pd


def scrape_it(url, data):
    page = requests.get(url, headers={'Cookie': 'ResultsPerPage=100'})
    soup = BeautifulSoup(page.text, 'html.parser')
    nexturl = soup.find_all(class_="standardLinkDkBlue")[-1]['href']
    stri = soup.find_all(class_="standardLinkDkBlue")[-1].string
    reports = soup.find_all("tr", {"class": ["SearchTableRowAlt", "SearchTableRow"]})

    for report in reports:
        data.append({
            'title': report.find('a', class_='linkTitle').text,
            'price': report.find('div', class_='resultPrice').text,
            'date_author': report.find('div', class_='textGrey').text.replace(' | published by: TechNavio', ''),
            'detail_link': report.a['href']
        })

    if 'next' in stri:
        data = scrape_it(nexturl, data)
    
    return data


with open('inputs.csv', newline='') as csvfile:
    reader = csv.DictReader(csvfile)
    urls = [row["url"] for row in reader]
    outputs = [row["outputfile"] for row in reader]

    for (url, output) in zip(urls, outputs):  # work on url and output together
        data = scrape_it(url, [])
        myOutput = pd.DataFrame(data)
        myOutput.to_csv(output, header=False)