如何在美丽的汤和python中指定css选择器？答案

【问题标题】：How to specify css selector in beautiful soup and python?如何在美丽的汤和python中指定css选择器？
【发布时间】：2021-02-16 14:18:28
【问题描述】：

我正在尝试从此链接中刮取卡片的标题：https://www.axisbank.com/retail/cards/credit-card

使用下面的代码

from urllib.request import urlopen
from bs4 import BeautifulSoup
import json, requests, re

axis_url = ["https://www.axisbank.com/retail/cards/credit-card"]

html = requests.get(axis_url[0])
soup = BeautifulSoup(html.content, 'lxml')

print(soup.select("#ulCreditCard h3"))

输出如下：

[]

我主要关心的是，当我在 soup.select() 函数中以 #ulCreditCard h3 的这种形式使用选择器小工具时，我应该如何指定我使用选择器小工具获得的 css 选择器标签。

【问题讨论】：

您所做的对 BeautifulSoup 来说很好，问题实际上是因为该站点使用 JavaScript 动态加载卡片信息。 BeautifulSoup 不执行 JavaScript 并动态更新 HTML。如果您打印出通过请求获得的 HTML，我敢打赌您找不到卡片，这就是 CSS 选择器找不到它的原因。您可能必须使用 Selenium 之类的可以在 HTML 上执行 JS 的东西，然后您应该能够找到内容。
@facelessuser 你能写出相应的硒代码吗！！！

标签： python web-scraping beautifulsoup python-requests scrapinghub

【解决方案1】：

如前所述，您要查找的信息是通过 Javascript 获得的。它使用稍微不同的 URL 来获取包含所有卡片详细信息的 JSON 数据。如果您改用它，您可以轻松列出所有卡片名称，而无需使用 BeautifulSoup。例如：

import requests
import json

axis_url = "https://www.axisbank.com/AjaxService/GetCreditCardsProducts"
data = {"strcategory" : "[]", "strrewardtypes" :"[]"}
r = requests.post(axis_url, data=data)

for entry in json.loads(r.json()[0]):
    print(entry['Name'])

会给你以下卡片：

Axis Bank ACE Credit Card
Axis Bank AURA Credit Card
Privilege Easy Credit Card
Axis Bank Reserve Credit Card
Axis Bank Freecharge Plus Credit Card
IndianOil Axis Bank Credit Card
Axis Bank Magnus Credit Card
Flipkart Axis Bank Credit Card
Axis Bank Freecharge Credit Card
Axis Bank MY Zone Credit Card
Axis Bank Neo Credit Card
Axis Bank Vistara Credit Card
Axis Bank Vistara Signature Credit Card
Axis Bank Vistara Infinite Credit Card
Axis Bank Privilege Credit Card
Miles and More Axis Bank Credit Card
Axis Bank Select Credit Card
Axis Bank Pride Platinum Credit Card
Axis Bank Pride Signature Credit Card
Axis Bank MY Zone Easy Credit Card
Axis Bank Insta Easy Credit Card
Axis Bank Signature Credit Card with Lifestyle Benefits
Platinum Credit Card
Titanium Smart Traveler Credit Card
Axis Bank My Wings Credit Card

【讨论】：