Python Web Scraping Beautiful Soup Parse Javascript Element 和 process 变量答案

【问题标题】：Python Web Scraping Beautiful Soup Parse Javascript Element and process variablePython Web Scraping Beautiful Soup Parse Javascript Element 和 process 变量
【发布时间】：2020-11-02 08:03:02
【问题描述】：

我正在尝试将 json 格式的变量检索到 python。

url = "https://limitededt.com/collections/footwear/products/adidas-originals-jonah-hill-superstar-fw7577"
source = requests.get(url).text

soup = BeautifulSoup(source, 'lxml')
print(soup.findAll('script')[10])

上面的代码产生：

<script>window.ShopifyAnalytics = window.ShopifyAnalytics || {};
window.ShopifyAnalytics.meta = window.ShopifyAnalytics.meta || {};
window.ShopifyAnalytics.meta.currency = 'SGD';
var meta = {"product":{"id":4738795503687,"gid":"gid:\/\/shopify\/Product\/4738795503687","vendor":"adidas Originals","type":"footwear","variants":[{"id":32432939204679,"price":18000,"name":"+ Jonah Hill Superstar - 7","public_title":"7","sku":"FW7577"},{"id":32432939237447,"price":18000,"name":"+ Jonah Hill Superstar - 7.5","public_title":"7.5","sku":"FW7577"},{"id":32432939270215,"price":18000,"name":"+ Jonah Hill Superstar - 8","public_title":"8","sku":"FW7577"},{"id":32432939302983,"price":18000,"name":"+ Jonah Hill Superstar - 8.5","public_title":"8.5","sku":"FW7577"},{"id":32432939335751,"price":18000,"name":"+ Jonah Hill Superstar - 9","public_title":"9","sku":"FW7577"},{"id":32432939368519,"price":18000,"name":"+ Jonah Hill Superstar - 9.5","public_title":"9.5","sku":"FW7577"},{"id":32432939401287,"price":18000,"name":"+ Jonah Hill Superstar - 10","public_title":"10","sku":"FW7577"},{"id":32432939434055,"price":18000,"name":"+ Jonah Hill Superstar - 10.5","public_title":"10.5","sku":"FW7577"},{"id":32432939466823,"price":18000,"name":"+ Jonah Hill Superstar - 11","public_title":"11","sku":"FW7577"},{"id":32432939499591,"price":18000,"name":"+ Jonah Hill Superstar - 11.5","public_title":"11.5","sku":"FW7577"}]},"page":{"pageType":"product","resourceType":"product","resourceId":4738795503687}};
for (var attr in meta) {
  window.ShopifyAnalytics.meta[attr] = meta[attr];
}</script>

基本上我想检索元变量的内容并访问python中的键值对。

我该怎么做？

【问题讨论】：

这能回答你的问题吗？ Parsing variable data out of a javascript tag using python
@metatoaster 谢谢，但它没有
使用这个正则表达式 - meta = ({(.*)}) 来获取 json。 regex101.com/r/CTaxPX/1
@bawagoc25 链接的线程为您提供了有关如何处理解决方案的总体思路，但有时我猜好人会免费分发专门针对您的情况的完整解决方案。

标签： python web-scraping beautifulsoup

【解决方案1】：

这对我有用

import json
import requests

url = "https://limitededt.com/collections/footwear/products/adidas-originals-jonah-hill-superstar-fw7577"
json_data = session.get(url + '.json').json()

【讨论】：

【解决方案2】：

import re, json

url = "https://limitededt.com/collections/footwear/products/adidas-originals-jonah-hill-superstar-fw7577"
source = requests.get(url).text

soup = BeautifulSoup(source, 'lxml')
script = soup.findAll('script')[10].text.strip()

p = re.compile("meta = ({(.*)})")
m = p.search(script)
json_str = m[1]

json_data = json.loads(json_str)
print(json_data)

输出：

{'product': {'id': 4738795503687,
  'gid': 'gid://shopify/Product/4738795503687',
  'vendor': 'adidas Originals',
  'type': 'footwear',
  'variants': [{'id': 32432939204679,
    'price': 18000,
    'name': '+ Jonah Hill Superstar - 7',
    'public_title': '7',
    'sku': 'FW7577'},
   {'id': 32432939237447,
    'price': 18000,
    'name': '+ Jonah Hill Superstar - 7.5',
    'public_title': '7.5',
    'sku': 'FW7577'},
   {'id': 32432939270215,
    'price': 18000,
    'name': '+ Jonah Hill Superstar - 8',
    'public_title': '8',
    'sku': 'FW7577'},
   {'id': 32432939302983,
    'price': 18000,
    'name': '+ Jonah Hill Superstar - 8.5',
    'public_title': '8.5',
    'sku': 'FW7577'},
   {'id': 32432939335751,
    'price': 18000,
    'name': '+ Jonah Hill Superstar - 9',
    'public_title': '9',
    'sku': 'FW7577'},
   {'id': 32432939368519,
    'price': 18000,
    'name': '+ Jonah Hill Superstar - 9.5',
    'public_title': '9.5',
    'sku': 'FW7577'},
   {'id': 32432939401287,
    'price': 18000,
    'name': '+ Jonah Hill Superstar - 10',
    'public_title': '10',
    'sku': 'FW7577'},
   {'id': 32432939434055,
    'price': 18000,
    'name': '+ Jonah Hill Superstar - 10.5',
    'public_title': '10.5',
    'sku': 'FW7577'},
   {'id': 32432939466823,
    'price': 18000,
    'name': '+ Jonah Hill Superstar - 11',
    'public_title': '11',
    'sku': 'FW7577'},
   {'id': 32432939499591,
    'price': 18000,
    'name': '+ Jonah Hill Superstar - 11.5',
    'public_title': '11.5',
    'sku': 'FW7577'}]},
 'page': {'pageType': 'product',
  'resourceType': 'product',
  'resourceId': 4738795503687}}

【讨论】：