使用 BeautifulSoup python 从 HTML 中提取 JSON答案

【问题标题】：Extracting JSON from HTML using BeautifulSoup python使用 BeautifulSoup python 从 HTML 中提取 JSON
【发布时间】：2019-06-16 14:40:00
【问题描述】：

当我在网页上练习一些网页抓取（需要参数 cookie）时，我发现自己在抓取 HTML 中嵌入的 JSON 数据时遇到了问题。以下是我所做的：

import requests from bs4
import BeautifulSoup as soup
import json
   
my_url = 'https://www.lazada.sg/catalog/?spm=a2o42.home.search.1.488d46b5mJGzEu&q=switch%20games&_keyori=ss&from=search_history&sugg=switch%20games_0_1'  

cookies = {
    "Hm_lpvt_7cd4710f721b473263eed1f0840391b4": "1548175412",
    "Hm_lvt_7cd4710f721b473263eed1f0840391b4": "1548140525",
    "x5sec":"7b22617365727665722d6c617a6164613b32223a223832333339343739626466613939303562613535386138333266383365326132434c4b516e65494645495474764a322b706f6d6f6941453d227d", }

ret = requests.get(my_url, cookies=cookies)
print("New Super Mario Bros" in ret.text) # True

page_soup = soup(ret.text, 'html.parser')
data = page_soup.findAll('script', {'type':'application/ld+json'})

输出如下：

[
  <script type="application/ld+json">{
    "@context": "https://schema.org",
    "@type": "BreadcrumbList",
    "itemListElement": [
      {
        "item": {
          "name": "Home",
          "@id": "https://www.lazada.sg/"
        },
        "@type": "ListItem",
        "position": "1"
      }
    ]
  }</script>,
  <script type="application/ld+json">{
    "@context": "https://schema.org",
    "@type": "ItemList",
    "itemListElement": [
      {
        "offers": {
          "priceCurrency": "SGD",
          "@type": "Offer",
          "price": "71.00",
          "availability": "https://schema.org/InStock"
        },
        "image": "https://sg-test-11.slatic.net/p/670a73a9613c36b2bb01555ab4092ba2.jpg",
        "@type": "Product",
        "name": "Switch: Super Mario Party [Available in Stock! Immediate Shipping]",
        "url": "https://www.lazada.sg/products/switch-super-mario-party-available-in-stock-immediate-shipping-i278269540-s429667097.html?search=1"
      },
      ...

我尝试关注现有线程Extract json from html in python beautifulsoup，但发现自己卡住了，可能是由于 HTML 汤中的 JSON 格式不同。我刮出的部分包含该页面中的所有不同产品，有没有办法进一步刮出每个产品的详细信息（例如标题、价格、评级等）并计算存在的产品数量？谢谢！

【问题讨论】：

您必须发布（完整但缩写的）HTML，其中包含 JSON 以供任何人帮助...
你需要像 [tag.text for tag in data] 这样的东西来获取实际的 JSON blob，然后每个 json.loads()。
这个类似问题的答案可能会有所帮助Parsing html for specific script type

标签： python html json web-scraping beautifulsoup

【解决方案1】：

您可以在使用json.loads 加载后从 json 循环解析出来。这些容器的所有产品信息都列在一个脚本标签中，因此您可以抓住它。

import requests 
from bs4 import BeautifulSoup as soup
import json
import pandas as pd

my_url = 'https://www.lazada.sg/catalog/?spm=a2o42.home.search.1.488d46b5mJGzEu&q=switch%20games&_keyori=ss&from=search_history&sugg=switch%20games_0_1'  

cookies = {
    "Hm_lpvt_7cd4710f721b473263eed1f0840391b4": "1548175412",
    "Hm_lvt_7cd4710f721b473263eed1f0840391b4": "1548140525",
    "x5sec":"7b22617365727665722d6c617a6164613b32223a223832333339343739626466613939303562613535386138333266383365326132434c4b516e65494645495474764a322b706f6d6f6941453d227d", }

ret = requests.get(my_url, cookies=cookies)
print("New Super Mario Bros" in ret.text) # True

page_soup = soup(ret.text, 'lxml')
data = page_soup.select("[type='application/ld+json']")[1]
oJson = json.loads(data.text)["itemListElement"]
numProducts = len(oJson)
results = []

for product in oJson:
    results.append([product['name'], product['offers']['price'], product['offers']['availability'].replace('https://schema.org/', '')])  # etc......

df =  pd.DataFrame(results)
print(df)

【讨论】：