有没有办法使用 selenium webdriver 从 shadowroot 读取数据

Question

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

options = Options()
options.binary_location = r'C:\Program Files\Mozilla Firefox\firefox.exe'
driver = webdriver.Firefox(executable_path=r'C:\Users\******\Downloads\geckodriver.exe', options=options)
#driver.get('http://google.com/')
url =  "https://www.foxnews.com/sports/nfl-great-frank-gore-says-hall-of-famer-numbers-speak-themselves"
driver.get(url)
item = driver.execute_script("return document.querySelector('div.article-comments').shadowRoot.querySelector('ul.spcv_messages-list')")
print(item.text)
driver.maximize_window()
driver.get(url)

注意：我正在尝试抓取福克斯新闻 cmets：https://www.foxnews.com/sports/nfl-great-frank-gore-says-hall-of-famer-numbers-speak-themselves。请帮我解决这个问题

Answer 1

你可以试试这个脚本。

import time

....

driver.maximize_window()
driver.get(url)
time.sleep(5)
item = driver.execute_script('''
    let divs =  document.querySelector('div.article-comments').querySelectorAll('div');
    let shadowroot = null;
    for (el of divs) {
        if (el.hasAttribute('data-spot-im-shadow-host')) {
            shadowroot = el.shadowRoot;
        }
    }

    return shadowroot.querySelector('ul.spcv_messages-list');
''')

在您的情况下，div.article-comments 没有shadowRoot 属性，您需要选择包含它的特定元素。在这种情况下，div 将具有属性 data-spot-im-shadow-host。