使用 selenium 的网页抓取表仅获取 html 元素但没有内容

Question

我正在尝试使用 selenium 和 beautifulsoup 从这 3 个网站抓取表：

https://www.erstebank.hr/hr/tecajna-lista

https://www.otpbanka.hr/tecajna-lista

https://www.sberbank.hr/tecajna-lista/

对于所有 3 个网站，结果都是表格的 HTML 代码，但没有文本。

我的代码如下：

import requests
from bs4 import BeautifulSoup
import pyodbc
import datetime

from selenium import webdriver

PATH = r'C:\Users\xxxxxx\AppData\Local\chromedriver.exe'

driver = webdriver.Chrome(PATH)

driver.get('https://www.erstebank.hr/hr/tecajna-lista')

driver.implicitly_wait(10)

soup = BeautifulSoup(driver.page_source, 'lxml')

table = soup.find_all('table')

print(table)

driver.close()

请帮助我错过了什么？

谢谢

Answer 1

您可以将此作为进一步工作的基础：-

from bs4 import BeautifulSoup as BS
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

TDCLASS = 'ng-binding'

options = webdriver.ChromeOptions()
options.add_argument('--headless')
with webdriver.Chrome(options=options) as driver:
    driver.get('https://www.erstebank.hr/hr/tecajna-lista')
    try:
        # There may be a cookie request dialogue which we need to click through
        WebDriverWait(driver, 5).until(EC.presence_of_element_located(
            (By.ID, 'popin_tc_privacy_button_2'))).click()
    except Exception:
        pass  # Probably timed out so ignore on the basis that the dialogue wasn't presented
    # The relevant <td> elements all seem to be of class 'ng-binding' so look for those
    WebDriverWait(driver, 5).until(
        EC.presence_of_element_located((By.CLASS_NAME, TDCLASS)))
    soup = BS(driver.page_source, 'lxml')
    for td in soup.find_all('td', class_=TDCLASS):
        print(td)

Answer 2

网站正在花时间加载table 中的数据。

要么申请time.sleep

import time

driver.get('https://www.erstebank.hr/hr/tecajna-lista')
time.sleep(10)...

或应用Explicit wait，以便rows 加载到tabel。

import requests
from bs4 import BeautifulSoup

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait

driver = webdriver.Chrome(executable_path="path to chromedriver.exe")
driver.maximize_window()

driver.get('https://www.erstebank.hr/hr/tecajna-lista')

wait = WebDriverWait(driver,30)
wait.until(EC.presence_of_all_elements_located((By.XPATH,"//table/tbody/tr[@class='ng-scope']")))

# driver.find_element_by_id("popin_tc_privacy_button_2").click() # Cookie setting pop-up. Works fine even without dealing with this pop-up. 
soup = BeautifulSoup(driver.page_source, 'html5lib')

table = soup.find_all('table')

print(table)

Answer 3

BeautifulSoup 不会找到该表，因为它在参考点上并不存在。在这里，如果 Selenium 注意到某个元素尚不存在，则告诉 Selenium 暂停 Selenium 驱动程序匹配器：

# This only works for the Selenium element matcher
driver.implicitly_wait(10)

然后，紧接着，你得到当前的 HTML 状态（表格仍然不存在）并将其放入 BeautifulSoup 的解析器。 BS4 将无法看到表格，即使它稍后加载，因为它将使用您刚刚提供的当前 HTML 代码：

# You now move the CURRENT STATE OF THE HTML PAGE to BeautifulSoup's parser
soup = BeautifulSoup(driver.page_source, 'lxml')

# As this is now in BS4's hands, it will parse it immediately (won't wait 10 seconds)
table = soup.find_all('table')

# BS4 finds no tables as, when the page first loads, there are none.

要解决此问题，您可以要求 Selenium 尝试获取 HTML 表本身。由于 Selenium 将使用您之前指定的 implicitly_wait，它会等到它存在，然后才允许其余的代码执行持续存在。此时，当 BS4 接收到 HTML 代码时，表格就会在那里。

driver.implicitly_wait(10)

# Selenium will wait until the element is found
# I used XPath, but you can use any other matching sequence to get the table
driver.find_element_by_xpath("/html/body/div[2]/main/div/section/div[2]/div[1]/div/div/div/div/div/div/div[2]/div[6]/div/div[2]/table/tbody/tr[1]")

soup = BeautifulSoup(driver.page_source, 'lxml')

table = soup.find_all('table')

---

但是，这有点矫枉过正。是的，您可以使用 Selenium 来解析 HTML，但您也可以只使用 requests 模块（从您的代码中，我看到您已经导入了该模块）直接获取表格数据。

数据从this端点异步加载（您可以使用Chrome DevTools自行查找）。您可以将其与json 模块配对，将其转换为格式良好的字典。这种方法不仅速度更快，而且占用的资源也少得多（Selenium 必须打开整个浏览器窗口）。

from requests import get
from json import loads

# Get data from URL
data_as_text = get("https://local.erstebank.hr/rproxy/webdocapi/fx/current").text

# Turn to dictionary
data_dictionary = loads(data_as_text)