带有 Selenium 的 Scrapy 不会检测到动态加载的 HTML 元素

Question

我正在使用带有 Selenium 的 Scrapy 来从该页面抓取内容：https://nikmikk.itch.io/door-knocker

在其中，div下有一个类.game_info_panel_widget的表，其中第一行Published 62 days ago似乎是动态加载的。

我已尝试获取 Scrapy 看到的页面，但在 html 中找不到该行。

scrapy fetch --nolog https://nikmikk.itch.io/door-knocker > test.html

这是我在test.html 中看到的，第一行是状态，而不是像我直接在 Chrome 中查看页面源时那样的已发布行。

<div class="game_info_panel_widget">                                                                                                                                         
    <table>                                                                                                                                              
        <tbody>                                                                                                                                                  
           <tr>                                                                                                                                                      
               <td>Status</td>                                                                                                                                                       
               <td>Prototype</td>                                                                                                                                                            
               ...                                                                                                                                               

           </tr>
            ...

在我的课堂SpiderDownloaderMiddleware 中，我加入了Selenium：

options = webdriver.ChromeOptions()
options.add_argument('headless')
options.add_argument('window-size=1200x600')

driver = webdriver.Chrome(chrome_options=options)

class SpiderDownloaderMiddleware(object):
# Omitted other codes
    def process_request(self, request, spider):
        driver.get(request.url)

        WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.CSS_SELECTOR, ".game_info_panel_widget"))
        )

        body = driver.page_source
        return HtmlResponse(driver.current_url, body=body, encoding='utf-8-sig', request=request)

如何检查该行的加载方式以及如何抓取这些信息？

更新： 我按照下面@Yosuva A 的回答得到了这样的结果：

 9 days ago

In development
Platforms
Windows
Rating
(9)
Author
David Clark
Genre
Survival, Puzzle
Tags
3D, Creepy, First-Person, Horror, Psychological Horror, Short, Singleplayer, Spooky, Unity
Average session
A few seconds
Languages
English

但是输出不一致，有时它给出了想要的，有时它没有。我猜是因为 Selenium 等待一般的 td 元素，这很常见：

"//div[@class='game_info_panel_widget']//table//tr//td"

我尝试修改为使用td[@text='Published']，但 Selenium 会超时。

我的代码：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome('chromedriver')  # Optional argument, if not specified will search path.
driver.implicitly_wait(15)

driver.get("https://thehive.itch.io/promnesia");
driver.find_element(By.XPATH,"//a[@class='toggle_info_btn']").click()

WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//div[@class='game_info_panel_widget']//table//tr//td"))) #Wait for specific element 

table_rows= driver.find_elements(By.XPATH,"//div[@class='game_info_panel_widget']//table//tr//td")

for rows in table_rows:
    print(rows.text)

driver.quit()

还有其他方法吗？

结论： 如果我们按照 Yosuva A 的建议在 click() 之后 time.sleep(2) 就可以了。

Answer 1

请让我知道这是否有帮助

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome('/usr/local/bin/chromedriver')  # Optional argument, if not specified will search path.
driver.implicitly_wait(15)

driver.get("https://thehive.itch.io/promnesia");
driver.find_element(By.XPATH,"//a[@class='toggle_info_btn']").click()
time.sleep(2)
WebDriverWait(driver, 3).until(EC.presence_of_element_located((By.XPATH, "//div[@class='game_info_panel_widget']/table//tr//td"))) #Wait for specific element 

table_rows= driver.find_elements(By.XPATH,"//div[@class='game_info_panel_widget']/table//tr//td")

for rows in table_rows:
    print rows.text

driver.quit()

输出

Updated
1 day ago
Published
9 days ago
Status
In development
Platforms
Windows
Rating
(9)
Author
David Clark
Genre
Survival, Puzzle
Tags
3D, Creepy, First-Person, Horror, Psychological Horror, Short, Singleplayer, Spooky, Unity
Average session
A few seconds
Languages
English