使用 BeautifulSoup 从站点检索的数据与站点上显示的数据不同

【问题标题】：Retrieved data from site using BeautifulSoup is different than data displayed on site使用 BeautifulSoup 从站点检索的数据与站点上显示的数据不同
【发布时间】：2020-08-09 07:22:22
【问题描述】：

A 使用以下代码从“https://www.adidas.com/us/men-shoes”网络抓取鞋子信息：

from bs4 import BeautifulSoup
from urllib.request import Request, urlopen
import requests

uri = "men-shoes"
hdr = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36'}
html_page = requests.get("https://www.adidas.com/us/" + str(uri), headers=hdr, timeout=15)
soup = BeautifulSoup(html_page.content, 'html.parser')
results = soup.find_all('div', attrs={'class': 'gl-product-card color-variations__fixed-size glass-product-card___17N3p'})

这是我在网络抓取特定鞋子时获得的数据示例：

<div class="gl-product-card color-variations__fixed-size glass-product-card___17N3p">
    <div class="gl-product-card__assets">
        <a class="gl-product-card__assets-link" data-auto-id="glass-hockeycard-link" href="/us/superstar-shoes/FV2820.html">
            <img
                alt="Originals Black Superstar Shoes"
                class="img_with_fallback___2aHBu gl-product-card__image"
                data-auto-id="image"
                src="https://assets.adidas.com/images/w_385,h_385,f_auto,q_auto:sensitive,fl_lossy/3c086bf61062470aa54cab8700b26add_9366/superstar-shoes.jpg"
                title="Superstar Shoes"
            />
            <img
                alt="Originals Black Superstar Shoes"
                class="img_with_fallback___2aHBu gl-product-card__image-hover"
                data-auto-id="image"
                src="https://assets.adidas.com/images/w_385,h_385,f_auto,q_auto:sensitive,fl_lossy/d04a49435d094fcfa8dfab960070f1a9_9366/superstar-shoes.jpg"
                title="Superstar Shoes"
            />
        </a>
        <div class="gl-product-card__wishlist">
            <div class="toggle_wishlist_button___1dG52" data-auto-id="wishlist-icon-container">
                <svg class="gl-icon" data-auto-id="wishlist-icon"><use xlink:href="#wishlist-inactive"></use></svg>
            </div>
        </div>
    </div>
    <div class="gl-product-card__carousel">
        <div class="product-carousel" data-auto-id="glass-mock-carousel">
            <div class="wrapper___3wqg4">
                <div class="item_wrapper___2toNm"></div>
                <div class="item_wrapper___2toNm"></div>
                <div class="item_wrapper___2toNm"></div>
                <div class="item_wrapper___2toNm"></div>
                <div class="item_wrapper___2toNm"></div>
                <div class="item_wrapper___2toNm"></div>
                <div class="item_wrapper___2toNm"></div>
                <div class="item_wrapper___2toNm"></div>
                <div class="item_wrapper___2toNm"></div>
                <div class="item_wrapper___2toNm"></div>
                <div class="item_wrapper___2toNm"></div>
                <div class="item_wrapper___2toNm"></div>
                <div class="item_wrapper___2toNm"></div>
                <div class="item_wrapper___2toNm"></div>
                <div class="item_wrapper___2toNm"></div>
                <div class="item_wrapper___2toNm"></div>
                <div class="item_wrapper___2toNm"></div>
                <div class="item_wrapper___2toNm"></div>
            </div>
        </div>
    </div>
    <div class="gl-product-card__details">
        <a class="gl-product-card__details-link" href="/us/superstar-shoes/FV2820.html">
            <div class="gl-product-card__details-top">
                <div class="gl-product-card__category" title="shoes">Originals</div>
                <div class="gl-product-card__details-icons"></div>
            </div>
            <div class="gl-product-card__details-main">
                <span class="gl-label gl-label--m gl-label--condensed gl-product-card__name" title="Superstar Shoes">Superstar Shoes</span>
                <div class="gl-price gl-price--s gl-price__inline___-VD1g notranslate"></div>
            </div>
            <div class="gl-product-card__details-bottom"><div class="gl-product-card__color">18 colors</div></div>
        </a>
    </div>
</div>

这是我直接从网站复制鞋子时得到的数据：

<div class="gl-product-card color-variations__fixed-size glass-product-card___17N3p">
    <div class="gl-product-card__assets">
        <a data-auto-id="glass-hockeycard-link" href="/us/zx-2k-4d-shoes/FW2003.html" class="gl-product-card__assets-link" data-di-id="di-id-93927325-c0dba7aa">
            <img
                data-auto-id="image"
                title="ZX 2K 4D Shoes"
                src="https://assets.adidas.com/images/w_385,h_385,f_auto,q_auto:sensitive,fl_lossy/d704fc8256204415b713ab6600f76418_9366/zx-2k-4d-shoes.jpg"
                alt="Originals White ZX 2K 4D Shoes"
                class="img_with_fallback___2aHBu gl-product-card__image performance-item"
                data-inject_ssr_performance_instrument=""
                onload="SSR_PERFORMANCE_MEASUREMENT(this)"
            />
            <img
                data-auto-id="image"
                title="ZX 2K 4D Shoes"
                src="https://assets.adidas.com/images/w_385,h_385,f_auto,q_auto:sensitive,fl_lossy/e169facc2c554c21b9d1ab880150342a_9366/zx-2k-4d-shoes.jpg"
                alt="Originals White ZX 2K 4D Shoes"
                class="img_with_fallback___2aHBu gl-product-card__image-hover"
            />
        </a>
        <div class="gl-product-card__wishlist">
            <div class="toggle_wishlist_button___1dG52" data-auto-id="wishlist-icon-container">
                <svg class="gl-icon" data-auto-id="wishlist-icon" data-di-res-id="d45e29bb-1d8adc35" data-di-rand="1596955983257"><use xlink:href="#wishlist-inactive"></use></svg>
            </div>
        </div>
        <div class="gl-badge gl-badge--small gl-badge--semi-urgent">New</div>
    </div>
    <div class="gl-product-card__details">
        <a href="/us/zx-2k-4d-shoes/FW2003.html" class="gl-product-card__details-link" data-di-id="di-id-93927325-c0dba7aa">
            <div class="gl-product-card__details-top">
                <div class="gl-product-card__category" title="shoes">Originals</div>
                <div class="gl-product-card__details-icons"></div>
            </div>
            <div class="gl-product-card__details-main">
                <span class="gl-label gl-label--m gl-label--condensed gl-product-card__name" title="ZX 2K 4D Shoes">ZX 2K 4D Shoes</span>
                <div class="gl-price gl-price--s gl-price__inline___-VD1g notranslate"><span class="gl-price__value">$200</span></div>
            </div>
        </a>
    </div>
</div>

正如您从网站和复制的数据中看到的那样，它显示了价格，在本例中为 200 美元。如何获取我的代码以显示鞋子的价格？

请附上时间优化的代码，因为我要刮一百多只鞋子。

【问题讨论】：

    标签： python python-3.x beautifulsoup python-requests


    【解决方案1】：

    对于该特定站点，您可能会更好地访问他们用于数据的 JSON API。这样你就不必抓取任何东西，只需解析 JSON 并读取它

    查看网站的网络检查器，

    【讨论】：

      【解决方案2】：

      您正在抓取的网页在使用 javascript 加载页面后动态加载其价格。 BeautifulSoup 仅处理 HTML，因此解决此问题的简单方法是加载链接到每张卡片的网页并从那里获取鞋子价格，因为每个产品的产品页面不会动态加载它们的价格。更轻量级的方式是使用网站的 JSON API。

      在这种情况下，您可以从产品列表中加载每个产品的产品 ID，然后从网站的 JSON API（https://www.adidas.com/api/search/product/FW2003）请求产品 FW2003 的信息。从该 JSON 中，您可以使用 Python 的 requests 模块构造一个字典。

      例如：

      from bs4 import BeautifulSoup
from urllib.request import Request, urlopen
import requests

uri = "men-shoes"
hdr = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36'}
html_page = requests.get("https://www.adidas.com/us/" + str(uri), headers=hdr, timeout=15)
soup = BeautifulSoup(html_page.content, 'html.parser')
results = soup.find_all('div', attrs={'class': 'grid-item___eaXVb'})
ids = []

for res in results:
    ids.append(res.get['data-grid-id'])

for id in ids:
    url = "https://www.adidas.com/api/search/product/" + id
    res = requests.get(url)
    price = res.json()['price']
    ...

      【讨论】：

      • 有没有更省时有效的方法？我需要买近 100 只鞋子
      • 这似乎不起作用。您的意思是使用“div”而不是“a”吗？
      【解决方案3】：

      这个网站的美妙之处在于一个页面中的所有产品都存储在一个名为 - DATA_STORE 的 javascript 变量中。如果您可以获得该变量值，您将获得有关产品的基本信息。但由于价格与折扣和时间段相关联，因此每次您向下滚动页面时，网站都会进行 ajax 调用，就像延迟加载一样。

      一旦您获得 productId，您必须对每个 productId 进行 ajax 调用以获取价格信息。

      以下脚本从页面获取所有产品数据，获取价格信息并将其存储到 json。一旦有了json，解析json就很容易了

      注意*：要从 javascript 变量中构造 json，这有点棘手。您需要编码和解码以删除脚本处理的反斜杠。

      from bs4 import BeautifulSoup
from urllib.request import Request, urlopen
import requests, sys, json, time

uri = "men-shoes"
hdr = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36'}
html_page = requests.get("https://www.adidas.com/us/" + str(uri), headers=hdr, timeout=15)
soup = BeautifulSoup(html_page.content, 'html.parser')

script = None
for i in soup.find_all("script"):
    if "DATA_STORE" in i.text.strip():
        script = i.text.strip()
        break

if script is None:
    print("no data found")
    sys.exit(1)

all_items = json.loads(script[script.index("{"):-3].encode().decode('unicode_escape'))

data = {}

for item in all_items['plp']['itemList']['items']:
    print(item["productId"])
    res = requests.get("https://www.adidas.com/api/search/product/{}?sitePath=us".format(item["productId"]), headers=hdr, timeout=15)
    data[item["productId"]] = res.json()
    time.sleep(1)

print(data)

with open("data.json", "w") as f:
    json.dump(data,f)

      输出：

      {'FW2003': {'price': 200, 'badgeStyle': '', 'badgeText': '', 'cached': False, 'salePrice': 200, 'image': {'src': 'https://assets.adidas.com/images/w_280,h_280,f_auto,q_auto:sensitive/d704fc8256204415b713ab6600f76418_9366/zx-2k-4d-shoes.jpg', 'cloudinary': True}, 'secondImage': {'src': 'https://assets.adidas.com/images/w_280,h_280,f_auto,q_auto:sensitive/e169facc2c554c21b9d1ab880150342a_9366/zx-2k-4d-shoes.jpg', 'cloudinary': True}, 'color': 'Cloud White / Core Black / Signal Pink', 'modelId': 'KYU21', 'orderable': True, 'validFrom': {'default@adidas-PE': '2020-07-01T03:00:00.000Z', 'default@adidas-CL': '2020-07-15T04:00:00.000Z', 'default@adidas-MX': '2020-10-01T17:00:00.000Z', 'default@adidas-CO': '2020-07-15T03:00:00.000Z', 'default@adidas-US': '2020-07-10T07:00:00.000Z', 'default@adidas-BR': '2020-07-01T03:00:00.000Z', 'default@adidas-AR': '2020-07-15T03:00:00.000Z'}, 'previewTo': '2020-07-13T07:00:00.000Z', 'isFlash': False, 'isFinalSale': False, 'isSpecialLaunch': False, 'id': 'FW2003', 'link': '/us/zx-2k-4d-shoes/FW2003.html'}, 'FX7847': {'price': 140, 'badgeStyle': '', 'badgeText': '', 'cached': False, 'salePrice': 140, 'image': {'src': 'https://assets.adidas.com/images/w_280,h_280,f_auto,q_auto:sensitive/5fba111ccaab411a9171ab57000ec9e8_9366/climacool-vento-shoes.jpg', 'cloudinary': True}, 'secondImage': {'src': 'https://assets.adidas.com/images/w_280,h_280,f_auto,q_auto:sensitive/77e6c85bc38d40e6a912ab52017d1197_9366/climacool-vento-shoes.jpg', 'cloudinary': True}, 'color': 'Signal Cyan / Orbit Grey / Signal Pink', 'modelId': 'LDT02', 'orderable': True, 'validFrom': {'default@adidas-CA': '2020-06-01T05:00:00.000Z', 'default@adidas-US': '2020-05-31T07:00:00.000Z'}, 'previewTo': '2020-06-01T07:00:00.000Z', 'isFlash': False, 'isFinalSale': False, 'isSpecialLaunch': False, 'id': 'FX7847', 'link': '/us/climacool-vento-shoes/FX7847.html'}, 'B42200': {'price': 140, 'badgeStyle': '', 'badgeText': '', 'cached': False, 'salePrice': 140, 'image': {'src': 'https://assets.adidas.com/images/w_280,h_280,f_auto,q_auto:sensitive/de7d57ddae474f139736a8ba00fcbfb8_9366/nmd_r1-shoes.jpg', 'cloudinary': True}, 'secondImage': {'src': 'https://assets.adidas.com/images/w_280,h_280,f_auto,q_auto:sensitive/ad3348fcdb5f40a0b410a8ba00fc6427_9366/nmd_r1-shoes.jpg', 'cloudinary': True}, 'color': 'Black / Black / Gum', 'modelId': 'BSZ68', 'orderable': True, 'validFrom': {'default': '2018-07-01T04:00:00.000Z', 'default@adidas-PE': '2018-07-01T03:00:00.000Z', 'default@adidas-CL': '2018-07-01T04:00:00.000Z', 'default@adidas-MX': '2018-07-01T05:00:00.000Z', 'default@adidas-CO': '2018-07-01T03:00:00.000Z', 'default@adidas-CA': '2018-06-01T05:00:00.000Z', 'default@adidas-US': '2018-06-01T07:00:00.000Z', 'default@adidas-BR': '2018-07-01T03:00:00.000Z', 'default@adidas-AR': '2019-04-01T03:00:00.000Z'}, 'previewTo': '2012-12-11T22:00:00.000Z', 'isFlash': False, 'isFinalSale': False, 'isSpecialLaunch': False, 'id': 'B42200', 'link': '/us/nmd_r1-shoes/B42200.html'}, 'EF1042': {'price': 180, 'badgeStyle': '', 'badgeText': '', 'cached': False, 'salePrice': 180, 'image': {'src': 'https://assets.adidas.com/images/w_280,h_280,f_auto,q_auto:sensitive/1e74db8746cd492b9814aafc0106ac2d_9366/ultraboost-20-shoes.jpg', 'cloudinary': True}, 'secondImage': {'src': 'https://assets.adidas.com/images/w_280,h_280,f_auto,q_auto:sensitive/c6c5c2caafc8405b8e4baaff00e21e50_9366/ultraboost-20-shoes.jpg', 'cloudinary': True}, 'color': 'Cloud White / Cloud White / Core Black', 'modelId': 'DVF21', 'orderable': True, 'validFrom': {'default@adidas-CA': '2020-01-01T06:00:00.000Z', 'default@adidas-US': '2020-01-01T08:00:00.000Z'}, 'previewTo': '2012-12-11T22:00:00.000Z', 'isFlash': False, 'isFinalSale': False, 'isSpecialLaunch': False, 'id': 'EF1042', 'link': '/us/ultraboost-20-shoes/EF1042.html'}, 'M20324': {'price': 80, 'badgeStyle': '', 'badgeText': '', 'cached': False, 'salePrice': 80, 'image': {'src': 'https://assets.adidas.com/images/w_280,h_280,f_auto,q_auto:sensitive/25c70a990dd74210aa47a59900ebfe5d_9366/stan-smith-shoes.jpg', 'cloudinary': True}, 'secondImage': {'src': 'https://assets.adidas.com/images/w_280,h_280,f_auto,q_auto:sensitive/f7f13f58f83e46698f15aacb01622c54_9366/stan-smith-shoes.jpg', 'cloudinary': True}, 'color': 'Cloud White / Core White / Green', 'modelId': 'ION05', 'orderable': True, 'validFrom': {'default': '2017-08-08T03:00:00.000Z', 'default@adidas-PE': '2017-05-01T03:00:00.000Z', 'default@adidas-CL': '2016-07-05T04:00:00.000Z', 'default@adidas-MX': '2017-01-01T06:00:00.000Z', 'default@adidas-CO': '2016-01-15T02:00:00.000Z', 'default@adidas-CA': '2015-01-27T06:00:00.000Z', 'default@adidas-US': '2014-01-15T08:00:00.000Z', 'default@adidas-BR': '2020-02-11T03:00:00.000Z', 'default@adidas-AR': '2017-08-08T03:00:00.000Z'}, 'previewTo': '2012-12-11T22:00:00.000Z', 'isFlash': False, 'isFinalSale': False, 'isSpecialLaunch': False, 'id': 'M20324', 'link': '/us/stan-smith-shoes/M20324.html'}}
...
...
...

      要获取价格：

      for k,v in a.items():
    print(f"ProductId - {k}")
    print("Price - {}".format(v["price"]))
    print("Sale Price - {}".format(v["salePrice"]))
    print("---"*20)

      输出：

      ProductId - FW2003
Price - 200
Sale Price - 200
------------------------------------------------------------
ProductId - FX7847
Price - 140
Sale Price - 140
------------------------------------------------------------
ProductId - B42200
Price - 140
Sale Price - 140
------------------------------------------------------------
ProductId - EF1042
Price - 180
Sale Price - 180
------------------------------------------------------------
ProductId - M20324
Price - 80
Sale Price - 80
------------------------------------------------------------

      【讨论】：

        猜你喜欢
        • 2012-07-16
        • 2021-02-08
        • 2012-12-07
        • 1970-01-01
        • 2015-02-17
        • 1970-01-01
        • 1970-01-01
        • 2014-07-21
        • 1970-01-01
        相关资源
        最近更新 更多
        热门标签
        Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode