如何使用 selenium 获取 9gag 中的所有评论？

Question

我正在努力从 9gag 中抓取模因及其所有 cmets。 我在下面使用了这段代码，但我只得到了几个额外的 cmets。

actions = ActionChains(driver)
link = driver.find_element(By.XPATH, "//button[@class='comment-list__load-more']")
actions.move_to_element(link).click(on_element=link).perform()

我也想通过模拟点击查看更多回复来访问评论下的子集。

从 html 中我发现这个 XPATH element = driver.find_element(By.XPATH, "//div[@class='vue-recycle-scroller ready page-mode direction-vertical']")holds cmets 部分，但我不确定如何遍历该元素中的每个评论并模拟这些点击。

如果您想测试它，如果存在必要的库，此代码应该可以直接运行。

请帮助我完成以下任务：

1. 从查看所有 cmets 中获取所有 cmets
2. 遍历每个评论部分并单击查看更多回复以获取所有子集

我的代码

import time
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
import undetected_chromedriver as uc

if __name__ == '__main__':

    options = Options()
    # options.headless = True
    options.add_argument("start-maximized")  # ensure window is full-screen
    driver = uc.Chrome(service=Service(ChromeDriverManager().install()), options=options)
    driver.get("https://9gag.com/gag/a5EAv9O")
    prev_h = 0
    for i in range(10):
        height = driver.execute_script("""
                   function getActualHeight() {
                       return Math.max(
                           Math.max(document.body.scrollHeight, document.documentElement.scrollHeight),
                           Math.max(document.body.offsetHeight, document.documentElement.offsetHeight),
                           Math.max(document.body.clientHeight, document.documentElement.clientHeight)
                       );
                   }
                   return getActualHeight();
               """)
        driver.execute_script(f"window.scrollTo({prev_h},{prev_h + 200})")
        time.sleep(1)
        prev_h += 200
        if prev_h >= height:
            break
    time.sleep(5)
    title = driver.title[:-7]
    try:
        upvotes_count = \
        driver.find_element(By.XPATH, "//meta[@property='og:description']").get_attribute("content").split(' ')[0]
        comments_count = \
        driver.find_element(By.XPATH, "//meta[@property='og:description']").get_attribute("content").split(' ')[3]
        upvotes_count = int(upvotes_count) if len(upvotes_count) <= 3 else int("".join(upvotes_count.split(',')))
        comments_count = int(comments_count) if len(comments_count) <= 3 else int("".join(comments_count.split(',')))
        date_posted = driver.find_element(By.XPATH, "//p[@class='message']")
        date_posted = date_posted.text.split("·")[1].strip()
        # actions = ActionChains(driver)
        # link = driver.find_element(By.XPATH, "//button[@class='comment-list__load-more']")
        # actions.move_to_element(link).click(on_element=link).perform()
        element = driver.find_element(By.XPATH,
                                      "//div[@class='vue-recycle-scroller ready page-mode direction-vertical']")
        print(element.text)
        driver.quit()
    except NoSuchElementException or Exception as err:
        print(err)

输出

编辑：

我设法使代码更好地工作。它滚动页面，直到看到所有的 cmets。如果有子集，它也会点击查看更多回复。

但它只能从中间到结尾读取 cmets。也许随着页面向下滚动，初始的 cmets 会被动态隐藏。我不知道如何克服这一点。并且点击查看更多回复在一些点击后停止并抛出错误

selenium.common.exceptions.MoveTargetOutOfBoundsException: Message: move target out of bounds

这是更新后的代码

import driver as driver
from selenium.webdriver.remote.webelement import WebElement
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
import time
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException, ElementClickInterceptedException
from selenium.webdriver.support.wait import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
import undetected_chromedriver as uc

def scroll_page(scrl_hgt):
    prev_h = 0
    for i in range(10):
        height = driver.execute_script("""
                       function getActualHeight() {
                           return Math.max(
                               Math.max(document.body.scrollHeight, document.documentElement.scrollHeight),
                               Math.max(document.body.offsetHeight, document.documentElement.offsetHeight),
                               Math.max(document.body.clientHeight, document.documentElement.clientHeight)
                           );
                       }
                       return getActualHeight();
                   """)
        driver.execute_script(f"window.scrollTo({prev_h},{prev_h + scrl_hgt})")
        time.sleep(1)
        prev_h += scrl_hgt
        if prev_h >= height:
            break

if __name__ == '__main__':
    options = Options()
    # options.headless = True
    driver = uc.Chrome(service=Service(ChromeDriverManager().install()), options=options)
    driver.maximize_window()
    driver.get("https://9gag.com/gag/a5EAv9O")
    time.sleep(5)

    # click on I accept cookies
    actions = ActionChains(driver)
    consent_button = driver.find_element(By.XPATH, '//*[@id="qc-cmp2-ui"]/div[2]/div/button[2]')
    actions.move_to_element(consent_button).click().perform()

    scroll_page(150)
    time.sleep(2)

    # click on fresh comments sectin
    fresh_comments = driver.find_element(By.XPATH, '//*[@id="page"]/div[1]/section[2]/section/header/div/button[2]')
    actions.move_to_element(fresh_comments).click(on_element=fresh_comments).perform()

    time.sleep(5)

    # getting meta data
    title = driver.title[:-7]
    upvotes_count = driver.find_element(By.XPATH, "//meta[@property='og:description']").get_attribute("content").split(' ')[0]
    comments_count = driver.find_element(By.XPATH, "//meta[@property='og:description']").get_attribute("content").split(' ')[3]
    upvotes_count = int(upvotes_count) if len(upvotes_count) <= 3 else int("".join(upvotes_count.split(',')))
    comments_count = int(comments_count) if len(comments_count) <= 3 else int("".join(comments_count.split(',')))
    date_posted = driver.find_element(By.XPATH, "//p[@class='message']")
    date_posted = date_posted.text.split("·")[1].strip()

    time.sleep(3)

    # click on lood more comments button to load all the comments
    load_more_comments = driver.find_element(By.XPATH, "//button[@class='comment-list__load-more']")
    actions.move_to_element(load_more_comments).click(on_element=load_more_comments).perform()

    scroll_page(500)

    print([my_elem.text for my_elem in driver.find_elements(By.CSS_SELECTOR, "div.comment-list-item__text")])

    comments = driver.find_elements(By.CSS_SELECTOR, "div.vue-recycle-scroller__item-view")
    for item in comments:
        html = item.get_attribute("innerHTML")
        if "comment-list-item__text" in html:
            print(item.find_element(By.CSS_SELECTOR, "div.comment-list-item__text").text)
        elif "comment-list-item__deleted-text" in html:
            print(item.find_element(By.CSS_SELECTOR, "div.comment-list-item__deleted-text").text)

        # get sub comments
        if "comment-list-item__replies" in html:
            #item.find_element(By.CSS_SELECTOR, "div.comment-list-item__replies").click()
            sub_comments = item.find_element(By.CSS_SELECTOR, "div.comment-list-item__replies")
            actions.move_to_element(sub_comments).click(on_element=sub_comments).perform()
        time.sleep(2)
    driver.quit()

PS：我的目标是按出现的顺序获取每个 cmets 及其所有子 cmets（无论是文本、图像、gif 等）并将它们保存在某个地方，以便我应该能够重新创建 cmets 部分再次。

Answer 1

要提取和打印你需要诱导WebDriverWait为visibility_of_all_elements_located()的评论文本，你可以使用以下Locator Strategies：

driver.get("https://9gag.com/gag/a5EAv9O")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.comment-list__load-more"))).click()
print([my_elem.text for my_elem in driver.find_elements(By.CSS_SELECTOR, "div.comment-list-item__text")])

控制台输出：

['Man, the battle of the cults is getting interesting now.', 'rent free in your head', 'Sorry saving all my money up for the Joe Biden Depends Multipack and the Karmella knee pads.', "It's basically a cult now.", "I'll take one. I'm not even American", '', 'that eagle looks familiar.', "Who doesn't want a trump card?"]

注意：您必须添加以下导入：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC