使用 Python 3 进行沃尔玛价格抓取

Question

我对这个概念很陌生，但我正在尝试学习如何使用 python 来操作 HTML 数据。

我编写了一个 python (ver. 3.4.1) 脚本，它获取 URL 并返回一些信息，我使用 BeautifulSoup (ver. 4) 对其进行解析。

import urllib.request
from bs4 import BeautifulSoup

response = urllib.request.urlopen('http://www.walmart.ca/en/ip/xbox-one/6000187109065')
html = response.read()

soup = BeautifulSoup(html)

print(soup.find_all('div', {"class" : "price-current"}))

在本例中，我试图获取 Xbox One 的价格。我选择这个 div 是因为它是在网页上向用户显示价格的那个。我知道有一个<span itemprop="price">$399.99</span> 可以抓取。从中提取是相当直截了当的，我更好奇为什么价格不会显示在它应该显示的 div 中。

我假设它与 HTML 标头有关，或者可能与使用标准 Web 浏览器浏览时自动发送的某种 POST/GET 数据有关。谁能解释为什么价格没有显示以及我必须做些什么才能让它们按预期显示？

Answer 1

价格是使用 javascript 加载的。这是请求：

Request URL:http://www.walmart.ca/ws/online/products
Request Method:POST

Form Data:
products:[{"productid":"6000187109066","skus":[[{"skuid":"6000187109066","status":"10"}]]}]
csrfToken:b08bfe580f3d9a0d893435fb

由于它包含一个 csrfToken，因此您需要在发出发布请求之前弄清楚它是如何生成或提供的。他们也可能依赖会话 cookie。

沃尔玛似乎有一个 API：https://developer.walmartlabs.com/

Answer 2

免责声明：我在 SerpApi 工作。

您可以通过 SerpApi 的 google-search-results 包获取沃尔玛产品的价格。这是一个免费试用的付费 API。

Full example

import os
from serpapi import GoogleSearch

params = dict(
    engine="walmart",
    query="xbox one",
    api_key=os.getenv("API_KEY"),
)

search = GoogleSearch(params)
results = search.get_dict()

for organic_result in results["organic_results"]:
    product_id = organic_result["us_item_id"] or organic_result["product_id"]

    search.params_dict = dict(
        engine="walmart_product",
        product_id=product_id,
        api_key=os.getenv("API_KEY"),
    )

    product_result = search.get_dict()

    print(
        f"Price of '{product_result['product_result']['title']}': '{product_result['product_result']['price_map']['price']} {product_result['product_result']['price_map']['currency']}'"
    )

输出

Price of 'Microsoft Xbox One S 1TB Console (Certified Refurbished)': '429 $'
Price of 'Star Wars Jedi: Fallen Order, Electronic Arts, Xbox One': '39.82 $'
Price of 'Microsoft - Xbox One S 1TB All-Digital Edition Console with Xbox One Wireless Controller (Renewed)': '499.99 $'
Price of 'Need for Speed: Heat, Electronic Arts, Xbox One, 014633373233': '25 $'
Price of 'Call of Duty: Black Ops Cold War, Activision, Xbox One': '49.94 $'
Price of 'NBA 2K21, 2K, Xbox One, 710425596858': '29.96 $'
Price of 'Refurbished Microsoft Factory Xbox One X 1TB, 4K Ultra HD Gaming Console in Black, FMQ-00042, 889842246971': '499.99 $'
Price of 'Madden NFL 21, Electronic Arts, Xbox One': '29.66 $'
Price of 'Need for Speed Payback, Electronic Arts, Xbox One, 014633370058': '14.96 $'
Price of 'Nickelodeon Kart Racers, Gamemill, Xbox One, 856131008060': '14.96 $'
Price of 'Sonic Mania (Sega), Xbox One, 010086640939': '14.96 $'
Price of 'Minecraft Master Collection, Microsoft, Xbox One, 889842394979': '42.88 $'
Price of 'Mortal Kombat 11, Warner Bros., Xbox One, 883929668977': '15 $'
Price of 'UFC 4, Electronic Arts, Xbox One': '29.96 $'
Price of 'Grand Theft Auto V, Rockstar Games, Xbox One': '22.8 $'
Price of 'Man Eater, Deep Silver, Xbox One, 816819017517': '24.96 $'
Price of 'Crash N. Sane Trilogy, Activision, Xbox One, 047875881969': '39.99 $'
Price of 'Spyro Reignited Trilogy, Activision, Xbox One, 047875882423': '29.83 $'
Price of 'Call of Duty: Modern Warfare, Activision, Xbox One, 0047875884366': '47.9 $'
Price of 'Cyberpunk 2077, Warner Bros, Xbox One': '39.74 $'
Price of 'Mass Effect Andromeda, Electronic Arts, Xbox One, 014633734096': '6.91 $'
Price of 'THQ theHunter: Call Of the Wild (Xbox One)': '19.93 $'
Price of 'Wreckfest, THQ-Nordic, Xbox One, 0811994021649': '29.83 $'
Price of 'Assassin’s Creed Valhalla Xbox Series X|S, Xbox One Standard Edition': '36.51 $'
Price of 'NBA 2K20, 2K, Xbox One, 710425595264': '21.99 $'
Price of 'LEGO Worlds, Warner Bros, Xbox One': '15 $'
Price of 'Call of Duty: Black Ops 3 Zombie Chronicles Edition, Activision, Xbox One, 047875881228': '37.88 $'
Price of 'Jumanji The Video Game, Bandai Namco, Xbox One, 819338020792': '19.93 $'
Price of 'Red Dead Redemption 2, Rockstar Games, Xbox One': '34.99 $'
Price of 'HALO 5, Microsoft, Xbox One, 885370928518': '19 $'
Price of 'LEGO Jurassic World, Warner, Xbox One, 883929472727': '19.8 $'
Price of 'Plants vs. Zombies: Battle for Neighborville, Electronic Arts, Xbox One, 014633736007': '14.96 $'
Price of 'The LEGO Marvel Collection, Warner Bros., Xbox One, 00883929670499': '24.98 $'
Price of 'LEGO The Incredibles, Warner Bros, Xbox One, 883929633005': '18 $'
Price of 'Assassin's Creed: Origins, Ubisoft, Xbox One, 887256028459': '19.93 $'
Price of 'Spongebob Squarepants: Battle for Bikini Bottom Rehydrated, THQ-Nordic, Xbox One, 811994022165': '29.77 $'
Price of 'The SIMS 4, Electronic Arts, Xbox One': '33.84 $'
Price of 'Star Wars Battlefront 2, Electronic Arts, Xbox One, 014633735321': '14.96 $'
Price of 'Pac-Man Championship Edition 2 + Arcade Game Series, Bandai/Namco, Xbox One, 722674220705': '14.49 $'
Price of 'Microsoft Xbox One S 1TB All Digital Edition 3 Game Bundle (Disc-free Gaming), White, NJP-00050': '198.95 $'