美丽的汤。使用 Python3 抓取多个 url

Question

当我运行以下代码时...

import requests
from bs4 import BeautifulSoup

counter = []
url = 'https://www.somemuseum.org/exhibitions/current-exhibitions'
req = requests.get(url)
soup = BeautifulSoup(req.text, 'html.parser')
links = soup.find_all(href="{{ card.url }}")
counter.append(links)
print(counter)

它返回...

<a class="card card--exhibit {{ card.type }}" href="{{ card.url }}">

检查网站上的相同元素显示它存储为...

<a href="/exhibitions/listings/2018/current-listing" class="card card--exhibit is-tier1">

我想做的是类似于以下的for loop...

for link in links:
    if card.type=="is-tier1":
        exhibit = soup.get('card.url')
        counter.append(exhibit)

我是 Beautiful Soup 的新手，因此不胜感激。谢谢。

Answer 1

很遗憾，BeautifulSoup 无法获取 href 数据，因为它们是由 js 渲染的。但是，您有几个选择。

第一个选项是selenium。
Selenium 运行 js 并且有选择 html 元素的方法，但是它非常缓慢和沉重。

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


url = "https://www.metmuseum.org/exhibitions/current-exhibitions"
driver = webdriver.Firefox()
driver.get(url)
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, 'a.card')))
elements = driver.find_elements_by_css_selector('a.is-tier1')
links = [e.get_attribute("href") for e in elements]
driver.quit()

---

第二种选择是使用 api。
数据通过 xhr 请求加载到/api/Exhibitions/CurrentExhibitionsListing。您可以直接从 api 请求数据，并以 json 格式获取结果。

import requests

url = 'https://www.metmuseum.org/api/Exhibitions/CurrentExhibitionsListing?location=main|breuer|cloisters&page=1'
req = requests.get(url)
results = req.json()['results']
links = [
    'https://www.metmuseum.org' + i['url'] 
    for i in results if i['type'] == 'is-tier1'
    ]

两种方法产生相同的结果，但我会使用第二种方法，因为它更快。