如何使用python在nike页面上抓取所有鞋子

Question

我正在尝试抓取https://www.nike.com/w/mens-shoes-nik1zy7ok 上的所有鞋子。如何抓取所有鞋子，包括向下滚动页面时加载的鞋子？

我想要获取的确切信息在类“product-card__body”的 div 元素中 如下：

<div class="product-card__body " data-el-type="Card"><figure><a class="product-card__link-overlay" href="https://www.nike.com/t/air-force-1-07-mens-shoe-TjqcX1/CJ0952-001">Nike Air Force 1 '07</a><a class="product-card__img-link-overlay" href="https://www.nike.com/t/air-force-1-07-mens-shoe-TjqcX1/CJ0952-001" aria-describedby="Nike Air Force 1 '07" data-el-type="Hero"><div class="image-loader css-zrrhrw product-card__hero-image is--loaded"><picture><source srcset="https://static.nike.com/a/images/c_limit,w_592,f_auto/t_product_v1/s12ff321cn2nykxhva9j/air-force-1-07-mens-shoe-TjqcX1.jpg" media="(min-width: 1024px)"><source srcset="https://static.nike.com/a/images/c_limit,w_592,f_auto/t_product_v1/s12ff321cn2nykxhva9j/air-force-1-07-mens-shoe-TjqcX1.jpg" media="(max-width: 1023px) and (-webkit-min-device-pixel-ratio: 2), (min-resolution: 192dpi)"><source srcset="https://static.nike.com/a/images/c_limit,w_318,f_auto/t_product_v1/s12ff321cn2nykxhva9j/air-force-1-07-mens-shoe-TjqcX1.jpg" media="(max-width: 1023px)"><img src="https://static.nike.com/a/images/c_limit,w_318,f_auto/t_product_v1/s12ff321cn2nykxhva9j/air-force-1-07-mens-shoe-TjqcX1.jpg" alt="Nike Air Force 1 '07 Men's Shoe"></picture></div></a><div class="product-card__info"><div class="product_msg_info"><div class="product-card__titles"><div class="product-card__title " id="Nike Air Force 1 '07">Nike Air Force 1 '07</div><div class="product-card__subtitle ">Men's Shoe</div></div></div><div class="product-card__count-wrapper show--all"><div class="product-card__count-item"><button type="button" aria-expanded="false" class="product-card__colorway-btn"><div aria-label="Available in 3 Colors" aria-describedby="Nike Air Force 1 '07" class="product-card__product-count "><span>3 Colors</span></div></button></div></div><div class="product-card__price-wrapper "><div class="product-card__price"><div><div class="product-price css-11s12ax is--current-price" data-test="product-price">$90</div></div></div></div></div></figure></div>

这是我正在使用的代码：

    html_data = requests.get("https://www.nike.com/w/mens-shoes-nik1zy7ok").text
    shoes = json.loads(re.search(r'window.INITIAL_REDUX_STATE=(\{.*?\});', html_data).group(1))

现在它只检索最初加载到页面上的鞋子。我如何获得其余的鞋子并将其附加到 shoes 变量中？

Answer 1

通过检查网站发出的 API 调用，您可以找到一个以 https://api.nike.com/ 开头的神秘 URL。此 URL 还存储在您已用于获取前几个产品的 INITIAL_REDUX_STATE 中。所以，我只是扩展你的方法：

import requests
import json
import re

# your product page
uri = 'https://www.nike.com/w/mens-shoes-nik1zy7ok'

base_url = 'https://api.nike.com'
session = requests.Session()

def get_lazy_products(stub, products):
"""Get the lazily loaded products."""
    response = session.get(base_url + stub).json()
    next_products = response['pages']['next']
    products += response['objects']
    if next_products:
        get_lazy_products(next_products, products)
    return products

# find INITIAL_REDUX_STATE
html_data = session.get(uri).text
redux = json.loads(re.search(r'window.INITIAL_REDUX_STATE=(\{.*?\});', html_data).group(1))

# find the initial products and the api entry point for the recursive loading of additional products
wall = redux['Wall']
initial_products = re.sub('anchor=[0-9]+', 'anchor=0', wall['pageData']['next'])

# find all the products
products = get_lazy_products(initial_products, [])

# Optional: filter by id to get a list with unique products
cloudProductIds = set()
unique_products = []
for product in products:
    try:
        if not product['id'] in cloudProductIds:
            cloudProductIds.add(product['id'])
            unique_products.append(product)
    except KeyError:
        print(product)

api 还返回产品的总数，尽管这个数字似乎有所不同，并且取决于 api 的 URL 中的 count 参数。

您需要帮助解析或汇总结果吗？