Selenium webdriver 从 find_elements_by_X 返回空列表

Question

我的目标是获取在一天的 24 小时内发布在 https://www.prusaprinters.org/prints 上的所有新项目的名称列表。

通过一些阅读，我了解到我应该使用 Selenium，因为我正在抓取的网站是动态的（在用户滚动时加载更多对象）。

问题是，除了webdriver.find_elements_by_ 中列出的任何后缀为https://selenium-python.readthedocs.io/locating-elements.html 的空列表外，我似乎什么也得不到。

在网站上，当我检查想要获取标题的元素时，我看到了 "class = name" 和 "class = clamp-two-lines"（见截图），但我似乎无法返回页面上所有元素的列表使用 name 类或 clamp-two-lines 类。

这是我到目前为止的代码（注释掉的行是失败的尝试）：

from timeit import default_timer as timer
start_time = timer()
print("Script Started")

import bs4, selenium, smtplib, time
from bs4 import BeautifulSoup 
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

driver = webdriver.Chrome(r'D:\PortableApps\Python Peripherals\chromedriver.exe')

url = 'https://www.prusaprinters.org/prints'
driver.get(url)
# foo = driver.find_elements_by_name('name')
# foo = driver.find_elements_by_xpath('name')
# foo = driver.find_elements_by_class_name('name')
# foo = driver.find_elements_by_tag_name('name')
# foo = [i.get_attribute('href') for i in driver.find_elements_by_css_selector('[id*=name]')]
# foo = [i.get_attribute('href') for i in driver.find_elements_by_css_selector('[class*=name]')]
# foo = [i.get_attribute('href') for i in driver.find_elements_by_css_selector('[id*=clamp-two-lines]')]
# foo = WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.XPATH, '//*[@id="printListOuter"]//ul[@class="clamp-two-lines"]/li')))
print(foo)
driver.quit()

print("Time to run: " + str(round(timer() - start_time,4)) + "s")

我的研究：

1. Selenium only returns an empty list
2. Selenium find_elements_by_css_selector returns an empty list
3. Web Scraping Python (BeautifulSoup,Requests)
4. Get HTML Source of WebElement in Selenium WebDriver using Python
5. How to get Inspect Element code in Selenium WebDriver
6. Web Scraping Python (BeautifulSoup,Requests)
7. https://chrisalbon.com/python/web_scraping/monitor_a_website/
8. https://www.codementor.io/@gergelykovcs/how-and-why-i-built-a-simple-web-scrapig-script-to-notify-us-about-our-favourite-food-fcrhuhn45
9. https://www.tutorialspoint.com/python_web_scraping/python_web_scraping_dynamic_websites.htm

Answer 1

要获取文本，请等待元素的可见性。标题的 CSS 选择器是 #printListOuter h3:

titles = WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, '#printListOuter h3')))

for title in titles:
    print(title.text)

短版：

wait = WebDriverWait(driver, 10)
titles = [title.text for title in wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, '#printListOuter h3')))]

Answer 2

这是项目名称的 xpath：

.//div[@class='print-list-item']/div/a/h3/span