网络抓取：最后一篇文章需要帮助并找到链接

Question

首先，对不起我的英语不好。
实际上，我有一个脚本scrapes 一个网站以在网页中的 python 中查找 cmets。
它用于 scrape 页面中的所有消息，但我希望 scrape 只是最后一个帖子。 请问这个怎么做？
同样，我想找到可能在上一条消息中发布的网络链接，但是一个完整的链接。
有可能吗？
这是网页链接和脚本：

https://www.dealabs.com/discussions/suivi-erreurs-de-prix-1063390?page=9999

#!/usr/bin/env python3
# https://www.jeuxvideo.com/forums/42-47-66784467-1-0-1-0-aide-scraping-python-forum-dealabs.htm
# scraping_dealabs.py

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By 

url = "https://www.dealabs.com/discussions/suivi-erreurs-de-prix-1063390?page=9999"

options = Options()
options.headless = True

driver = webdriver.Chrome(options=options)
driver.get(url)

# Accepter les cookies
button = WebDriverWait(driver, 2).until(
    EC.element_to_be_clickable((By.XPATH, "/html/body/main/div[4]/div[1]/div/div[1]/div[2]/button[2]/span"))
)
button.click()

# On recherche les commentaires et on affiche le texte
comments = driver.find_elements_by_class_name("commentList-item")

for comment in comments:
    _id = comment.get_attribute("id")
    author = comment.find_element_by_class_name('userInfo-username').text
    content = comment.find_element_by_class_name('userHtml-content').text
    timestamp = comment.find_element_by_class_name('text--color-greyShade').text
    comment_url = f"{url}#{_id}"

    print("Posté par", author)
    print(content)
    print("Publication:", timestamp)
    print("Lien du commentaire:")
    print(comment_url)
    print('-' * 30)

driver.close()

感谢您的时间和回复！

Answer 1

首先，我希望您使用正确的定位器，因此请尝试使用此 CSS 选择器 .btn--mode-primary.overflow--wrap-on，而不是 /html/body/main/div[4]/div[1]/div/div[1]/div[2]/button[2]/span。
为了获得最后一条评论，您可以使用这个 XPath：(//div[@class='commentList-item'])[last()] 所以为了得到最后的评论细节，只有你的代码可以修改成这样：

#!/usr/bin/env python3
# https://www.jeuxvideo.com/forums/42-47-66784467-1-0-1-0-aide-scraping-python-forum-dealabs.htm
# scraping_dealabs.py

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By 
from selenium.webdriver.common.action_chains import ActionChains

url = "https://www.dealabs.com/discussions/suivi-erreurs-de-prix-1063390?page=9999"

options = Options()
options.headless = True

driver = webdriver.Chrome(options=options)
driver.get(url)

actions = ActionChains(driver)

# Accepter les cookies
WebDriverWait(driver, 2).until(
EC.element_to_be_clickable((By.CSS_SELECTOR, ".btn--mode-primary.overflow--wrap-on"))).click()

last_comment = driver.find_element_by_xpath("(//div[@class='commentList-item'])[last()]")

actions.move_to_element(last_comment).perform()
time.sleep(0.5)
last_comment = driver.find_element_by_xpath("(//div[@class='commentList-item'])[last()]")

_id = last_comment.get_attribute("id")
author = last_comment.find_element_by_xpath(".//span[contains(@class,'userInfo-username')]").text
content = last_comment.find_element_by_xpath(".//*[contains(@class,'userHtml-content')]").text
timestamp = last_comment.find_element_by_xpath(".//*[contains(@class,'text--color-greyShade')]").text
comment_url = f"{url}#{_id}"

print("Posté par", author)
print(content)
print("Publication:", timestamp)
print("Lien du commentaire:")
print(comment_url)
print('-' * 30)

driver.close()

UPD
如您在 cmets 中所述，要获取页面上的最后一个元素，您必须将定位器从

last_comment = driver.find_element_by_xpath("(//div[@class='commentList-item'])[last()]")

到

last_comment = driver.find_element_by_xpath("(//div[@class='commentList-comment'])[last()]")

所以上面的整个代码将是：

#!/usr/bin/env python3
# https://www.jeuxvideo.com/forums/42-47-66784467-1-0-1-0-aide-scraping-python-forum-dealabs.htm
# scraping_dealabs.py

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By 
from selenium.webdriver.common.action_chains import ActionChains

url = "https://www.dealabs.com/discussions/suivi-erreurs-de-prix-1063390?page=9999"

options = Options()
options.headless = True

driver = webdriver.Chrome(options=options)
driver.get(url)

actions = ActionChains(driver)

# Accepter les cookies
WebDriverWait(driver, 2).until(
EC.element_to_be_clickable((By.CSS_SELECTOR, ".btn--mode-primary.overflow--wrap-on"))).click()

last_comment = driver.find_element_by_xpath("(//div[@class='commentList-comment'])[last()]")

actions.move_to_element(last_comment).perform()
time.sleep(0.5)
last_comment = driver.find_element_by_xpath("(//div[@class='commentList-comment'])[last()]")

_id = last_comment.get_attribute("id")
author = last_comment.find_element_by_xpath(".//span[contains(@class,'userInfo-username')]").text
content = last_comment.find_element_by_xpath(".//*[contains(@class,'userHtml-content')]").text
timestamp = last_comment.find_element_by_xpath(".//*[contains(@class,'text--color-greyShade')]").text
comment_url = f"{url}#{_id}"

print("Posté par", author)
print(content)
print("Publication:", timestamp)
print("Lien du commentaire:")
print(comment_url)
print('-' * 30)

driver.close()