使用 BeautifulSoup 抓取特定国家/地区的产品页面答案

【问题标题】：Scraping country specific product pages with BeautifulSoup使用 BeautifulSoup 抓取特定国家/地区的产品页面
【发布时间】：2021-04-16 17:28:10
【问题描述】：

我使用 BeautifulSoup 成功抓取了以下网站，但是页面上的产品列表会根据用户的位置而变化。如何包含位置标签/cookie，以便仅抓取与我的国家/地区相关的产品？

https://www.themacallan.com/en/shop

【问题讨论】：

标签： python web-scraping beautifulsoup

【解决方案1】：

数据是通过 Ajax 从外部 URL 加载的。您可以使用此示例来更改位置并加载产品 Feed：

import json
import requests

headers = {"Ocp-Apim-Subscription-Key": "f5c5fbebaa034a11975feac44b091c09"}

# los angeles
url = "https://api.edrington.com/consumer/v1/macallan/en/US/Product/location/91605"

data = requests.get(url, headers=headers).json()

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

for item in data["items"]:
    print("{:<50} {}".format(item["title"], item["availableForSale"]))

打印：

The Macallan 1824 Series Scotch No. 6              True
The Macallan Sherry Oak 30 Years Old               True
The Macallan Estate                                True
The Macallan Edition No. 5                         True
The Macallan Sherry Oak 25 Years Old               True
The Macallan Reflexion                             True
The Macallan Double Cask 18 Years Old              True

对于邮政编码 10001 - 纽约，请使用 url = "https://api.edrington.com/consumer/v1/macallan/en/US/Product/location/10001"：

The Macallan Sherry Oak 12 Years Old               True
The Macallan Rare Cask                             True
The Macallan Sherry Oak 18 Years Old               True
The Macallan Sherry Oak 30 Years Old               True
The Macallan Edition No. 5                         True
The Macallan Classic Cut 2019 Edition              True
The Macallan Edition No. 4                         True
The Macallan Sherry Oak 25 Years Old               True
The Macallan Reflexion                             True
The Macallan Double Cask 12 Years Old              True
The Macallan Double Cask 15 Years Old              True
The Macallan Double Cask 18 Years Old              True
The Macallan Edition No. 6                         True

编辑：对于英国和 URL 列表：

import json
import requests
from bs4 import BeautifulSoup


headers = {"Ocp-Apim-Subscription-Key": "f5c5fbebaa034a11975feac44b091c09"}

# great britain:
url = "https://api.edrington.com/consumer/v1/macallan/en/GB/Product"

data = requests.get(url, headers=headers).json()

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

available_ids = {i["productId"] for i in data["items"]}

soup = BeautifulSoup(
    requests.get("https://www.themacallan.com/en/shop").content, "html.parser"
)

for i in soup.select("[data-master-product-id]"):
    if i["data-master-product-id"] in available_ids:
        print("https://www.themacallan.com" + i.a["href"])

打印：

https://www.themacallan.com/en/double-cask-18-years-old
https://www.themacallan.com/en/whisky/single-malts/rare-cask/rare-cask-2020-release
https://www.themacallan.com/en/whisky/single-malts/sherry-oak-25-years-old-2019-release
https://www.themacallan.com/en/the-macallan-lumina-whisky
https://www.themacallan.com/en/the-macallan-fine-oak-10-years-old
https://www.themacallan.com/en/the-macallan-no-6
https://www.themacallan.com/en/the-macallan-quest-whisky
https://www.themacallan.com/en/the-macallan-terra-whisky
https://www.themacallan.com/en/triple-cask-matured-18-years-old
https://www.themacallan.com/en/the-macallan-lalique-tumbler
https://www.themacallan.com/en/the-macallan-rocks-glass-tumblers

注意：所有产品都加载在页面内。您只需根据available_ids 过滤它们。

编辑 2：获取“未缓存”页面：

import json
import requests
from bs4 import BeautifulSoup

# great britain:
url = "https://api.edrington.com/consumer/v1/macallan/en/GB/Product"


headers = {
    "Accept": "application/json, text/plain, */*",
    "Accept-Language": "en-US,en;q=0.5",
    "cache-control": "no-cache",
    "Connection": "keep-alive",
    "Host": "api.edrington.com",
    "Ocp-Apim-Subscription-Key": "f5c5fbebaa034a11975feac44b091c09",
    "Origin": "https://www.themacallan.com",
    "Referer": "https://www.themacallan.com/",
    "Sec-GPC": "1",
    "TE": "Trailers",
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0",
}

options_headers = {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "en-US,en;q=0.5",
    "Access-Control-Request-Headers": "cache-control,ocp-apim-subscription-key",
    "Access-Control-Request-Method": "GET",
    "Connection": "keep-alive",
    "Host": "api.edrington.com",
    "Origin": "https://www.themacallan.com",
    "Referer": "https://www.themacallan.com/",
    "Sec-GPC": "1",
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0",
}

with requests.Session() as s:
    s.options(url, headers=options_headers).text
    data = s.get(url, headers=headers).json()

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

available_ids = {i["productId"] for i in data["items"]}

soup = BeautifulSoup(
    requests.get("https://www.themacallan.com/en/shop").content, "html.parser"
)

for i in soup.select("[data-master-product-id]"):
    if i["data-master-product-id"] in available_ids:
        print("https://www.themacallan.com" + i.a["href"])

打印：

https://www.themacallan.com/en/double-cask-12-years-old
https://www.themacallan.com/en/sherry-oak-18-years-old
https://www.themacallan.com/en/whisky/single-malts/sherry-oak/sherry-oak-12-years-old
https://www.themacallan.com/en/sherry-oak-25-years-old
https://www.themacallan.com/en/whisky/single-malts/macallan-estate
https://www.themacallan.com/en/estate3
https://www.themacallan.com/en/triple-cask-matured-12-years-old
https://www.themacallan.com/en/distilling-scotland
https://www.themacallan.com/en/the-macallan-chevron-highball
https://www.themacallan.com/en/chevron-tumblers
https://www.themacallan.com/en/the-macallan-chevron-water-jug
https://www.themacallan.com/en/the-macallan-hip-flask
https://www.themacallan.com/en/the-macallan-ice-ball-maker
https://www.themacallan.com/en/the-macallan-in-lalique-glass
https://www.themacallan.com/en/the-macallan-polo-shirt

编辑 3：现在分页：

import json
import requests
from bs4 import BeautifulSoup

# great britain:
url = "https://api.edrington.com/consumer/v1/macallan/en/GB/Product"


headers = {
    "Accept": "application/json, text/plain, */*",
    "Accept-Language": "en-US,en;q=0.5",
    "cache-control": "no-cache",
    "Connection": "keep-alive",
    "Host": "api.edrington.com",
    "Ocp-Apim-Subscription-Key": "f5c5fbebaa034a11975feac44b091c09",
    "Origin": "https://www.themacallan.com",
    "Referer": "https://www.themacallan.com/",
    "Sec-GPC": "1",
    "TE": "Trailers",
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0",
}

options_headers = {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "en-US,en;q=0.5",
    "Access-Control-Request-Headers": "cache-control,ocp-apim-subscription-key",
    "Access-Control-Request-Method": "GET",
    "Connection": "keep-alive",
    "Host": "api.edrington.com",
    "Origin": "https://www.themacallan.com",
    "Referer": "https://www.themacallan.com/",
    "Sec-GPC": "1",
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0",
}

available_ids = set()
with requests.Session() as s:

    u = url
    while True:
        s.options(u, headers=options_headers).text
        data = s.get(u, headers=headers).json()

        for i in data["items"]:
            available_ids.add(i["productId"])

        if not data["hasNextPage"]:
            break

        u = url + "?after=" + data["lastCursor"]

    soup = BeautifulSoup(
        s.get("https://www.themacallan.com/en/shop").content,
        "html.parser",
    )

idx = 1
for i in soup.select("[data-master-product-id]"):
    if i["data-master-product-id"] in available_ids:
        print(idx, "https://www.themacallan.com" + i.a["href"])
        idx += 1

打印：

1 https://www.themacallan.com/en/double-cask-12-years-old
2 https://www.themacallan.com/en/double-cask-18-years-old
3 https://www.themacallan.com/en/whisky/single-malts/rare-cask/rare-cask-2020-release
4 https://www.themacallan.com/en/sherry-oak-18-years-old
5 https://www.themacallan.com/en/whisky/single-malts/sherry-oak/sherry-oak-12-years-old
6 https://www.themacallan.com/en/sherry-oak-25-years-old
7 https://www.themacallan.com/en/whisky/single-malts/sherry-oak-25-years-old-2019-release
8 https://www.themacallan.com/en/the-macallan-lumina-whisky
9 https://www.themacallan.com/en/whisky/single-malts/macallan-estate
10 https://www.themacallan.com/en/estate3
11 https://www.themacallan.com/en/the-macallan-fine-oak-10-years-old
12 https://www.themacallan.com/en/the-macallan-no-6
13 https://www.themacallan.com/en/the-macallan-quest-whisky
14 https://www.themacallan.com/en/the-macallan-terra-whisky
15 https://www.themacallan.com/en/triple-cask-matured-12-years-old
16 https://www.themacallan.com/en/triple-cask-matured-18-years-old
17 https://www.themacallan.com/en/distilling-scotland
18 https://www.themacallan.com/en/the-macallan-chevron-highball
19 https://www.themacallan.com/en/chevron-tumblers
20 https://www.themacallan.com/en/the-macallan-chevron-water-jug
21 https://www.themacallan.com/en/the-macallan-hip-flask
22 https://www.themacallan.com/en/the-macallan-ice-ball-maker
23 https://www.themacallan.com/en/the-macallan-in-lalique-glass
24 https://www.themacallan.com/en/the-macallan-lalique-tumbler
25 https://www.themacallan.com/en/the-macallan-polo-shirt
26 https://www.themacallan.com/en/the-macallan-rocks-glass-tumblers

【讨论】：

非常感谢。我如何找到欧盟国家的位置代码，例如英国。我还需要抓取原始页面上的 href 中存储的页面地址，例如：href="/en/the-macallan-quest-whisky"。我怎样才能把它结合起来？
你是该死的摇滚明星。
实际上打印出来的产品只有英国页面上可用产品的 1/3 左右。我们似乎错过了很多？
让我们continue this discussion in chat。
@theraredram 我更新了我的答案，添加了分页。