无法使用 BeautifulSoup 提取产品“href”答案

【问题标题】：Can't Extract Product "href" using BeautifulSoup无法使用 BeautifulSoup 提取产品“href”
【发布时间】：2021-12-26 03:05:43
【问题描述】：

我想从这个pages 中提取产品href 属性。是否有任何错误或阻塞？

我的代码：

import datetime, re, time, requests
import pandas as pd 
import numpy as np 
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter
from bs4 import BeautifulSoup as bs4
    

url2 = "https://www.vidri.com.sv/catalogo/07040101/Taladros-y-atornilladores-inalambricos.html"

headers = {"User-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36 Edg/95.0.1020.40"}

r2 = requests.get(url2, headers)

soup2 = bs4(r2.content, "lxml", parse_only=None)
products_container = soup2.find("div", class_= "products__container")

for item in products_container.find_all("div", class_= "productCard"):
    for sku in item.find_all("a", href = True):
    link_list.append(sku["href"])
        link_list

【问题讨论】：

标签： python web-scraping beautifulsoup python-requests

【解决方案1】：

您可以通过以下方式使其工作：Dev tools -> Network -> Fetch/XHR -> find an appropriate name (left tab) and click on it -> Preview (opened tab on the right)，如果您要提取数据，请从Headers 选项卡向此 URL 发出请求。

您可能会通过调用 requests .json() method 找到所有需要的数据，这会将 JSON 字符串解码为字典。

代码和example in the online IDE：

import requests
    
# Dev tools -> Network -> Fetch/XHR -> Headers -> Request Headers (list)
headers = {
    "x-requested-with": "XMLHttpRequest",  # only this request header matter, otherwise it will throw an error
}

# https://docs.python-requests.org/en/master/user/quickstart/#json-response-content
response = requests.get("https://www.vidri.com.sv/catalogo/07040101/Taladros-y-atornilladores-inalambricos.html", headers=headers).json()

for result in response["articles"]["data"]:
    sku = result["sku"]
    title = result["title"]
    price = int(float(result["price"]))  # gets rid of floating number: "300.00000000"

    print(f"{sku}\n{title}\n{price}\n")


# part of the output
'''
107477
Taladro inalámbrico 1/2 pulg 20v brushless cd791d2-b3
300

127397
Taladro atornillador 3/8" inalambrico bosch gsr 12v-15 fc1
225

132423
Taladro inalámbrico 3/8 pulg 12v dcd700c2-b3
175
'''

【讨论】：