如何用beautifulsoup从scripts标签中抓取数据？答案

【问题标题】：How to scrape data from inside the scripts tag with beautifulsoup?如何用beautifulsoup从scripts标签中抓取数据？
【发布时间】：2021-11-05 15:06:23
【问题描述】：

我正在尝试从网站的脚本中提取一些数据。我能够从整个页面获取数据，但是我被困在如何从脚本标签内部获取数据。这是我喝汤的方法。

import requests
from bs4 import BeautifulSoup

searchf = input("Enter the product you want: ")

url =f"https://www.daraz.com.np/catalog/?q={searchf}&_keyori=ss&from=input&spm=a2a0e.searchlist.search.go.12ec45adZnn1MP"

r = requests.get(url)
contents = r.content

soup = BeautifulSoup(contents,'html.parser')
print(soup.prettify())

这是我要抓取的网站，我只想获取那里列出的产品名称。 https://www.daraz.com.np/catalog/?q=cars&_keyori=ss&from=input&spm=a2a0e.searchlist.search.go.3b834ce4JPBUHm

【问题讨论】：

标签： python beautifulsoup python-requests

【解决方案1】：

数据也由 javacript 动态加载。转到网络选项卡> xhr/fetch> headers，然后您将看到 api url，如果您单击预览选项卡，您将看到数据，项目总数为 4080，每页包含 40 个项目，我已经使用查询字符串参数作为参数与 api url 一起发送。

代码：

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36'}

api_url = 'https://www.daraz.com.np/catalog/'

params = {
    'keyori': 'ss',
    'ajax': 'true',
    'from': 'input',
    #'page': '2',
    'q': 'cars',
    'spm': 'a2a0e.searchlist.search.go.3b834ce4JPBUHm'
    
    }

for num in range(0, 4080, 40):
    params['totalResults'] = num
    print(num)
    resp = requests.get(api_url, params=params, headers=headers).json()

    items = resp['mods']['listItems']
    for item in items:
        name = item['name']
        print(name)

输出：

Hot wheels 3 Pack Cars
XimiVogue  Pull-Back-And-Go Alloy Suv Car Toy With Sound (Bmw Mini)
Kids Spider Stunt Flip Car
Funny Diy Electric Thomas Train Truck Set Toy Set Orange
Monster Trucks Inertia Car Toys For Kids
Remote Control Transformer Car
Remote Control Emulation Car
Spiderman  Friction Stunt Car
Remote Control Car
360 Degree Spinning Rolling stunt Farm Truck Super Tipping Car for Kids
Monster Trucks Friction Powered Inertia Cars Toys For Kids (Color May Vary)
Gallant Toy Bus-Set Of 2
Remote Control Transformer Car
19026B Express Train Set - Battery Operated Black Train Toys for Kids Aged 3+ Years
Friction Powered Monster Truck Toy (Multicolor) Pack Of 1
Toy Train
Model Concept R/C Car
Friction Powered 3-Pack Mini Push and Go Car Truck Jam Playset
Makes Your Kids Happy -  Remote Control Vehicle Car For Kids
2.4Ghz 1/18 Rc Rock Crawler Vehicle Buggy Car 4 Wd Shaft Drive High Speed Remote Control 4X4 Monster Off Road Truck
XimiVogue  Pull-Back-And-Go Alloy Suv Car Toy With Sound
Kids Spider Stunt Flip Car Small Size
Black ForMula Remote Control Car For Kids

...依此类推（共 4080 条）

【讨论】：

非常感谢先生?
也谢谢你。请点击向上箭头进行投票