网络抓取。美丽汤。多页。你到底要怎么做？

Question

您好，我是编程新手。所以我花了 4 天时间尝试学习 python。我也说出了一些新的脏话。 我特别有兴趣尝试一些网络抓取来学习新东西并获得一些曝光以了解它是如何运作的。 这就是我想出的。见最后的代码。它有效（在一定程度上）

但是缺少什么？

1. 这个网站有分页。在这种情况下，价值 11 页。您将如何添加到此脚本并让 python 也查看其他页面并执行相同的抓取。即刮第一页，刮第2、3 ... 11页并将结果发布到csv？ https://www.organicwine.com.au/vegan/?pgnum=1

   https://www.organicwine.com.au/vegan/?pgnum=2

   https://www.organicwine.com.au/vegan/?pgnum=3

   https://www.organicwine.com.au/vegan/?pgnum=4

   https://www.organicwine.com.au/vegan/?pgnum=5

   https://www.organicwine.com.au/vegan/?pgnum=6

   https://www.organicwine.com.au/vegan/?pgnum=7

8、9、10 和 11

1. 在这些页面上，图像实际上是一个缩略图，例如 251 像素 x 251 像素。 你会怎么去添加到这个脚本中说。当您访问它时，请点击详细产品页面的链接并从那里捕获图像链接，其中图像为 1600 像素 x 1600 像素，并将这些链接发布到 CSV https://www.organicwine.com.au/mercer-wines-preservative-free-shiraz-2020

2. 当我们识别出这些链接后，我们还可以将这些较大的图像下载到一个文件夹中

3. CSV 编写器。我也不明白第 58 行 对于我在范围内（23） 如果不计算它们，我怎么知道有多少产品（即第一页有 24 种产品）

所以这就是我想学习的方法。要求不高（他讽刺地说）我可以付钱请人来做这件事，但那有什么乐趣呢？这并没有教我如何“钓鱼”。

哪里有学习python的好地方？关于网络抓取的大师班。这似乎是反复试验和博客文章，您可以在任何地方获取一些信息来将它们拼凑在一起。 也许我需要一个导师。 我希望有人可以联系到我，告诉我 beautifulSoup 是什么。通过反复试验和主要猜测来解决这个问题。不了解它，但它确实有效。

无论如何，我们将不胜感激任何有助于将所有这些整合在一起以生成一个体面的脚本的帮助。 希望有人不介意帮助我。

向organicwine 道歉，因为他们将他们的网站用作学习工具。我不希望对网站造成任何伤害或滋扰

提前谢谢你 约翰

代码：

import requests
import csv
from bs4 import BeautifulSoup

URL = "https://www.organicwine.com.au/vegan/?pgnum=1"
response = requests.get(URL)
website_html = response.text

soup = BeautifulSoup(website_html, "html.parser")

product_title = soup.find_all('div', class_="caption")
# print(product_title)

winename = []
for wine in product_title:
    winetext = wine.a.text
    winename.append(winetext)
    print(f'''Wine Name: {winetext}''')
# print(f'''\nWine Name: {winename}\n''')

product_price = soup.find_all('div', class_='wrap-thumb-mob')
# print(product_price.text)

price =[]
for wine in product_price:
    wineprice = wine.span.text
    price.append(wineprice)
    print(f'''Wine Price: {wineprice}''')
# print(f'''\nWine Price: {price}\n''')


image =[]
product_image_link = (soup.find_all('div', class_='thumbnail-image'))
# print(product_image_link)
for imagelink in product_image_link:
    wineimagelink = imagelink.a['href']
    image.append(wineimagelink)
    # image.append(imagelink)
    print(f'''Wine Image Lin: {wineimagelink}''')
# print(f'''\nWine Image: {image}\n''')
#
#
# """ writing data to CSV """
# open OrganicWine2.csv file in "write" mode
# newline stops a blank line appearing in csv

with open('OrganicWine2.csv', 'w',newline='') as file:
  # create a "writer" object
  writer = csv.writer(file, delimiter=',')
  # use "writer" obj to write
  # you should give a "list"
  writer.writerow(["Wine Name", "Wine Price", "Wine Image Link"])
  for i in range(23):
    writer.writerow([
      winename[i],
      price[i],
      image[i],
    ])

Answer 1

通过将页码放入其中来创建 URL，然后将其余代码放入 for 循环中，您可以使用 len(winenames) 来计算您有多少结果。您应该在for 循环之外进行写作。这是您的代码进行了这些更改：

import requests
import csv
from bs4 import BeautifulSoup

num_pages = 11

result = []
for pgnum in range(num_pages):
    url = f"https://www.organicwine.com.au/vegan/?pgnum={pgnum+1}"
    response = requests.get(url)
    website_html = response.text

    soup = BeautifulSoup(website_html, "html.parser")

    product_title = soup.find_all("div", class_="caption")

    winename = []
    for wine in product_title:
        winetext = wine.a.text
        winename.append(winetext)

    product_price = soup.find_all("div", class_="wrap-thumb-mob")

    price = []
    for wine in product_price:
        wineprice = wine.span.text
        price.append(wineprice)

    image = []
    product_image_link = soup.find_all("div", class_="thumbnail-image")
    for imagelink in product_image_link:
        winelink = imagelink.a["href"]
        response = requests.get(winelink)
        wine_page_soup = BeautifulSoup(response.text, "html.parser")
        main_image = wine_page_soup.find("a", class_="fancybox")
        image.append(main_image['href'])

    for i in range(len(winename)):
        result.append([winename[i], price[i], image[i]])


with open("/tmp/OrganicWine2.csv", "w", newline="") as file:
    writer = csv.writer(file, delimiter=",")
    writer.writerow(["Wine Name", "Wine Price", "Wine Image Link"])
    writer.writerows(results)

以下是我将如何重写您的代码来完成此任务。它更 Pythonic（你基本上不应该写 range(len(something))，总有一种更简洁的方式）并且它不需要知道有多少页结果：

import csv
import itertools
import time

import requests
from bs4 import BeautifulSoup

data = []
# Try opening 100 pages at most, in case the scraping code is broken
# which can happen because websites change.
for pgnum in range(1, 100):
    url = f"https://www.organicwine.com.au/vegan/?pgnum={pgnum}"
    response = requests.get(url)
    website_html = response.text

    soup = BeautifulSoup(website_html, "html.parser")

    search_results = soup.find_all("div", class_="thumbnail")
    for search_result in search_results:
        name = search_result.find("div", class_="caption").a.text
        price = search_result.find("p", class_="price").span.text

        # link to the product's page
        link = search_result.find("div", class_="thumbnail-image").a["href"]

        # get the full resolution product image
        response = requests.get(link)
        time.sleep(1)  # rate limit
        wine_page_soup = BeautifulSoup(response.text, "html.parser")
        main_image = wine_page_soup.find("a", class_="fancybox")
        image_url = main_image["href"]

        # or you can just "guess" it from the thumbnail's URL
        # thumbnail = search_result.find("div", class_="thumbnail-image").a.img['src']
        # image_url = thumbnail.replace('/thumbL/', '/full/')

        data.append([name, price, link, image_url])

    # if there's no "next page" button or no search results on the current page,
    # stop scraping
    if not soup.find("i", class_="fa-chevron-right") or not search_results:
        break

    # rate limit
    time.sleep(1)


with open("/tmp/OrganicWine3.csv", "w", newline="") as file:
    writer = csv.writer(file, delimiter=",")
    writer.writerow(["Wine Name", "Wine Price", "Wine Link", "Wine Image Link"])
    writer.writerows(data)