python 使用 selenium 从 JS 获取 HTML

Question

我正在尝试从https://www.workday.com/en-us/company/careers/open-positions.html#?q= 获取 div HTML。

但列出工作职位的 div 是基于网络 XHR 从granite.min.js 加载的。

from selenium import webdriver
from bs4 import BeautifulSoup
from pprint import pprint


path_to_chromedriver = "/Users/RichWin/Documents/chromedriver.exe"
browser = webdriver.Chrome(executable_path=path_to_chromedriver)

driver = browser.get('https://www.workday.com/en-us/company/careers/open-positions.html#?q=')

elem = driver.find_element_by_id('template-content')

soup = BeautifulSoup(elem.get_text, "html.parser")

for tag in soup.find_all('div'):
    pprint(tag)

谁能帮帮我？

Answer 1

好的，所以你的代码有几个问题。

a) 您需要等待 template-content div 加载其内容。在下面的代码中，我使用 implicitly_wait 等待 30 秒。
b) find_element_by_id 不返回 HTML 而是一个 Selenium 对象。因此不能将其传递给BeautifulSoup 进行解析。

from pprint import pprint
from bs4 import BeautifulSoup
from selenium import webdriver


url = 'https://www.workday.com/en-us/company/careers/open-positions.html#?q='
path_to_chromedriver = "/Users/RichWin/Documents/chromedriver.exe"

browser = webdriver.Chrome(executable_path=path_to_chromedriver)
browser.implicitly_wait(30)
browser.get(url)

elem = browser.find_element_by_id('template-content')
elem_html = elem.get_attribute('innerHTML')

soup = BeautifulSoup(elem_html, "html.parser")
for tag in soup.find_all('div'):
    pprint(tag)

browser.quit()