具有页面限制的网页抓取

Question

我一直在尝试使用 BeautifulSoup 抓取此站点 (https://www.americanas.com.br/hotsite/todas-ofertas-mundo) 中的产品。我可以在一页中获取所有项目，并且由于分页在 url 中，只需使用计数器移动到下一个（例如第 2 页是 https://www.americanas.com.br/hotsite/todas-ofertas-mundo/pagina-2 等等）。所以问题是最大页面是 416，在那个数字页面之后没有显示任何产品。由于每个站点显示 24 种产品，我几乎无法达到 10k 种产品（根据页面总共有 400 万种产品）。

我试图更深入地研究类别，但我遇到了同样的问题（一些更深的类别也有超过 10k 的产品），还过滤了“marca”、“price”和“loja”以及同样的问题。因此，即使使用最好的过滤器，我也无法获得所有产品，因为我达到了其中的最大页面。

我还搜索了一个 API，因此我可以尝试绕过它，但找不到任何可以在没有产品 ID 的情况下请求目录的内容。我确实找到了一个用于获得不同品牌和卖家，但产品数量相同的问题。

这也是在其他市场上困扰我的一个问题，让我很难获得网站的所有目录，不得不过滤并尝试获得最多数量的产品，但不是全部。所以任何建议都非常受欢迎。谢谢！

这里是抓取页面的代码

import requests
import time
import re
import json
from bs4 import BeautifulSoup
import urllib3
urllib3.disable_warnings()

def html(url):
    try:
        soup = BeautifulSoup(requests.get(url,verify = False).content,'html.parser',from_encoding="utf-8")
        return soup
    except Exception as e:
        print(e)
        print("Not loading")

def product_info(prod):
    quote = {}
    if prod.find("span",{"class": re.compile(r"UnavailableTextMessage")}):
        return 
    quote['id'] = prod.find("a").get("href").replace("?","/",1).split("/")[2]
    quote['name'] = prod.find("h2").getText()
    #prod_info = prod.find("span")
    quote['price'] = prod.find("span", {"class": re.compile(r"PriceUI-bwhjk3-11")}).getText().replace(".","").split(" ")[-1]
    quote['full_price'] = quote['price']
    quote['discount'] = ''
    discount = prod.find("span",{"class": re.compile(r"TextUI-xlll2j-3")})
    if discount:
        quote["discount"] = discount.getText().replace("%","")
        disc = quote['discount'].replace("%","")
        quote['full_price'] = prod.find("span", {"class": re.compile(r"PriceUI-sc-1q8ynzz-0")}).getText().replace(".","").split(" ")[-1]
    inter = prod.find("span", {"class": re.compile(r"InternationalText")})
    quote['inter'] = 0
    if inter:
        quote['inter'] = 1
    quote['url']= "https://americanas.com.br"+ prod.find("a").get("href")
    return quote



test = "https://www.americanas.com.br/hotsite/todas-ofertas-mundo"
products = [] ##lim 10k
counter = 2
while True:
    page = "/pagina-"+str(counter)
    url = test + page
    counter += 1
    soup = html(url)
    print(url)        
    content = soup.findAll("div",{"class": "product-grid-item"}) 
    if content == []:
        print(counter)
        break
    for cont in content:
        if product_info(cont):
            products.append(product_info(cont))

Answer 1

您在寻找 API 方面的想法是正确的。如果您记录您的网络流量并访问其中一个产品页面，您会看到向多个 API 发出的请求。

第一个返回产品 ID 的集合。注意查询字符串参数offset 和limit。在此示例中，我将offset 设置为"0"（以便我们从第一个产品开始），并将limit 设置为"10"，以检索前十个产品的产品ID：

def main():

    import requests

    url = "https://mystique-v2-americanas.juno.b2w.io/search"

    params = {
        "offset": "0",
        "sortBy": "topSelling",
        "source": "omega",
        "filter": [
            '{"id":"referer","value":"/hotsite/todas-ofertas-mundo","fixed":true,"hidden":true}',
            '{"id":"currency","value":"USD","fixed":true,"name":"moeda","hidden":true}'
        ],
        "limit": "10",
        "suggestion": "true"
    }

    response = requests.get(url, params=params)
    response.raise_for_status()

    products = response.json()["products"]

    for product in products:
        print(product["id"])

    return 0


if __name__ == "__main__":
    import sys
    sys.exit(main())

输出：

158285472
107684121
88842655
88899155
84894032
94728488
107684117
84894015
80349294
84894042
>>> 

将此与另一个 API 结合使用，您可以根据产品 ID 获取每个产品的特定信息：

def get_product_info(product_id):

    import requests

    url = "https://restql-server-api-v2-americanas.b2w.io/run-query/catalogo/product-buybox/5"

    params = {
        "c_opn": "",
        "id": product_id,
        "offerLimit": "1",
        "opn": "",
        "tags": "prebf*|SUL_SUDESTE_CENTRO|livros_prevenda"
    }

    response = requests.get(url, params=params)
    response.raise_for_status()

    info = response.json()

    return info["product"]["result"]["name"], info["installment"]["result"][0][0]["total"]

def main():

    import requests

    url = "https://mystique-v2-americanas.juno.b2w.io/search"

    params = {
        "offset": "0",
        "sortBy": "topSelling",
        "source": "omega",
        "filter": [
            '{"id":"referer","value":"/hotsite/todas-ofertas-mundo","fixed":true,"hidden":true}',
            '{"id":"currency","value":"USD","fixed":true,"name":"moeda","hidden":true}'
        ],
        "limit": "10",
        "suggestion": "true"
    }

    response = requests.get(url, params=params)
    response.raise_for_status()

    products = response.json()["products"]

    for product in products:
        name, price = get_product_info(product["id"])
        print(f"The name is \"{name}\" and the price is {price}.")

    return 0


if __name__ == "__main__":
    import sys
    sys.exit(main())

输出：

The name is "Smartwatch Esportivo Blitzwolf ® BW-HL1 ip68 e Multi Idiomas" and the price is 197.17.
The name is "Bebe reborn girafinha" and the price is 466.87.
The name is "Boneca Bebe Reborn 45 Cm corpo todo de Silicone Boneca Menina Reborn Realista bebes cabelo e olhos castanhos NPKDOLL" and the price is 400.28.
The name is "Boneca Bebê Reborn 43cm Corpo Todo Silicone - Menina com Cabelo Cacheado e Ursinho de pelúcia KAYDORA" and the price is 397.48.
The name is "Boneca Bebe Reborn Menina com roupa de Pandinha 47 cm NPKDOLL" and the price is 329.28.
The name is "Fones De Ouvido Sem Fio Bluetooth Xiaomi Redmi Airdots" and the price is 256.2.
The name is "Boneca Bebe Reborn Menino Girafinha 48 Cm Menino com Pelucia Girafa Azul NPKDOLL" and the price is 464.63.
The name is "Boneca Bebê Reborn Menina Realista de Silicone e Algodão 48cm e Girafinha NPKDOLL" and the price is 289.96.
The name is "Mini Caixa de Som Portátil Speaker  a Prova D’Água - Xiaomi" and the price is 165.2.
The name is "Boneca Bebe Reborn Menina princesa com casaco de inverno de coelhinho 45 cm NPKDOLL" and the price is 372.4.
>>> 

你明白了。我实际上并没有尝试将limit 查询字符串参数设置为十以外的任何值，因此您可能想尝试一下。