如何使用 Python BS4 抓取 Shopify 产品信息答案

【问题标题】：How to Scrape Shopify Product Information Using Python BS4如何使用 Python BS4 抓取 Shopify 产品信息
【发布时间】：2020-06-23 03:04:23
【问题描述】：

<html>
 <body>
  <p>
   {"products":[{"id":2069443051648,"title":"BornxRaised Indian Summer Print Button-Up Shirt - Multi","handle":"bxrb4010smpt-mlt"}]
  </p>
 </body>
</html>

嗨，我对 bs4 和网络抓取非常陌生。我正在尝试制作一个返回产品信息的应用程序，例如项目的 ID、标题和句柄。上面的 html 代码来自一个 shopify 网站，是我目前正在使用的。下面我发布了到目前为止我在提取内容方面的成果

import requests
from bs4 import BeautifulSoup

source = requests.get('https://kith.com/products.json').text
soup = BeautifulSoup(source, 'lxml')
p = soup.find('p')
text = list(p.children)[0]


print('')
print(text.strip())


{"products":[{"id":2069443051648,"title":"BornxRaised Indian Summer Print Button-Up Shirt - Multi","handle":"bxrb4010smpt-mlt"}]

我遇到了一些问题，将其进一步简化为仅显示 ID、标题和句柄。有没有人有什么建议？我什至可能把这一切都搞错了，我在这里有点被挖掘出来了..

【问题讨论】：

标签： python beautifulsoup python-requests shopify extract

【解决方案1】：

在这种情况下，您无需使用 BeautifulSoup，因为您使用的是 json 提要。 requests 提供了一种方便的方式来访问作为 json 对象的响应；因此，您需要做的就是：

import requests

plist = requests.get('https://kith.com/products.json').json()
for product in plist:
    print(product['id'], product['title'], product['handle'])

【讨论】：

【解决方案2】：

由于 URL 是 json，所以可以直接使用 json 库，而不是使用 Beautifulsoup。你需要做的是：

import requests
import json

source = requests.get('https://kith.com/products.json').text
js = json.loads(source)

id = js["products"][0]["id"]
title = js["products"][0]["title"]
handle = js["products"][0]["handle"]

print(id, title, handle)

结果会是这样的：

2069443051648 BornxRaised Indian Summer Print Button-Up Shirt - Multi bxrb4010smpt-mlt

【讨论】：

【解决方案3】：

获取id、title和handle，可以试试：

from bs4 import BeautifulSoup
import requests
import json

source = requests.get('https://kith.com/products.json').text

soup = BeautifulSoup(source, 'lxml')
p = soup.find('p')
product = json.loads(p.text)

for i in range(len(product["products"])):
    id = product["products"][i]["id"]
    title = product["products"][i]["title"]
    handle = product["products"][i]["handle"] 
    
    print("id : " + str(id) + " title : " + str(title) + " handle: "+ str(handle))

输出将是：

id : 2069443051648 title : BornxRaised Indian Summer Print Button-Up Shirt - Multi handle: bxrb4010smpt-mlt
id : 2069447803008 title : Kith Patchwork Seersucker Bucket Hat - Multi handle: kh5850-115
id : 2069446688896 title : Kith Patchwork Howard Tee - Red / Navy / Multi handle: kh3789-115
id : 2069445705856 title : Kith Patchwork Camp Shirt - Red / Navy / Multi handle: kh3790-115
id : 2069445738624 title : Kith Patchwork Bandana Hardaway - Red / Navy / Multi handle: kh6327-115
id : 2069447508096 title : Kith Patchwork Bandana Gi - Red / Navy / Multi handle: kh1242-115
id : 2069445509248 title : Nike Toddler Air Jordan 5 Retro Top 3 - Black / Emerald / Fire / Grape handle: jbcz2991-001
id : 2069442494592 title : Nike Pre-School Air Jordan 5 Retro Top 3 - Black / Emerald / Fire / Grape handle: jbcz2990-001
id : 2069442920576 title : Nike Grade School Air Jordan 5 Retro Top 3 - Black / Emerald / Fire / Grape handle: jbcz2989-001
id : 2069442723968 title : Nike Air Jordan 5 Retro Top 3 - Black / Emerald / Fire / Grape handle: jbcz1786-001
id : 2069448622208 title : A Cold Wall Block Logo Socks - Charcoal handle: acwmsk001whl-chr
id : 2069441478784 title : 1017 ALYX 9SM Soft Classic Hat Curved Zip - Beige handle: aauha0026fa01beg0004
id : 2069394227328 title : Vans x Kenzo UA OG Sk8-Hi LX - Multi handle: vans-x-kenzo-ua-og-sk8-hi-lx-multi
id : 2069394260096 title : Vans x Kenzo UA OG Old Skool LX - Multi handle: vn0a4p3x01h
id : 2069441642624 title : Nike Toddler Air Jordan 6 Retro - Neutral Grey / Black / White / True Red handle: jb384667-062
id : 2069441380480 title : Nike Pre-School Air Jordan 6 Retro - Neutral Grey / Black / White / True Red handle: jb384666-062
id : 2069441314944 title : Nike Grade School Air Jordan 6 Retro - Neutral Grey / Black / White / True Red handle: jb384665-062
id : 2069441249408 title : Nike Air Jordan 6 Retro - Neutral Grey / Black / White / True Red handle: jbct8529-062
id : 2069438070912 title : Nike WMNS Daybreak - Pale Ivory / Pollen Rise / Shimmer / Track Red handle: nkck2351-102
id : 2069445443712 title : Kith Fix The System Tee - Black handle: kh3fix-100
id : 2069445476480 title : Kith Fix The System Tee - White handle: kith-fix-the-system-tee-white
id : 2069445312640 title : adidas Consortium Hyke AOH 001 - White handle: aafv3915
id : 2069445345408 title : adidas Consortium Hyke AOH 001 - Python / Cloud White handle: aafv4254
id : 2069422964864 title : Proenza Schouler Jersey Bodega Print Tee - Pistachio handle: pswl2024146-grn
id : 2069344223360 title : No Ka`oi Kindly 7/8 Leggings - Sky handle: nkp3cssnokw72584a0bl
id : 2069344125056 title : No Ka`oi Sweetie On Zip-Up Jacket - Multicolor 50 handle: nkp3cfenokw72540a0
id : 2069359263872 title : Area Crystal Sweetheart Mini Dress - Rainbow handle: arre20d20068-mlt
id : 2069387608192 title : Nike Air Tailwind 79 SE - Midnight Navy / Black handle: nkck4712-400
id : 2069404352640 title : Nike Pre-School Sunray Protect 2 - Oracle Aqua / Ghost Green / Blue handle: nk943826-303
id : 2069352382592 title : Nike Air Zoom Spiridon Cage 2 - Light Smoke Grey / Metallic Silver handle: nkcj1288-001

【讨论】：