Selenium 滚动到页面底部偶尔会停止

Question

我有一个可以“无限”向下滚动的页面（为了说明，我将使用页面https://www.reuters.com/news/us，每次滚动到页面底部时都会显示更多旧新闻）。我想向下滚动到页面底部以使用 selenium 多次加载旧项目。以下代码原则上有效：

driver = webdriver.Chrome(executable_path=r"/Users/bernhardclemm/Desktop/chromedriver")
driver.get('https://www.reuters.com/news/us')

# Accept Cookies
wait = WebDriverWait(driver, 5)
cookies = wait.until(EC.element_to_be_clickable((By.XPATH, "//button[@id='_evidon-banner-acceptbutton']")))
cookies.click()

# Switch of "Photos" to speed up loading
wait = WebDriverWait(driver, 5)
photos_off = wait.until(EC.element_to_be_clickable((By.XPATH, "//label[contains(text(),'Photos')]")))
photos_off.click()

# Scroll down 100 times
i  = 1
while i < 500:
    print(i)
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    i += 1
    time.sleep(1)

正如您在下面的 GIF 中看到的那样，对于循环中的每一轮（i 打印在右侧的 Python 控制台中），Selenium 驱动程序向下滚动一次（左侧的 Google Chrome）。

但是，有时 Selenium 驱动程序已到达页面底部但没有加载新内容，如 GIF 所示。注意循环是如何在 Python 中运行的：

当我在 Chrome 中手动向上滚动一点时，自动滚动再次开始。因此，这不是，因为该网站已到达其内容的末尾。

知道这是由于网站还是我的代码中的缺陷造成的吗？如果是关于网站的问题，有什么方法可以避免 Selenium 的问题？

Answer 1

我建议检查Loading more div 元素是否消失。 使用try..except块检查如果它没有找到元素Loading more做一点scroll up页面。执行时间会多一点。但是你会达到你想要的。

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome(executable_path=r"/Users/bernhardclemm/Desktop/chromedriver")
driver.get('https://www.reuters.com/news/us')

# Accept Cookies - you will have to adapt the XPATH to your system language
wait = WebDriverWait(driver, 5)
cookies = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#_evidon-banner-acceptbutton")))
cookies.click()

# Scroll down 100 times
i  = 1

while i < 500:
    print(i)
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    i += 1
    try:
       WebDriverWait(driver,3).until(EC.invisibility_of_element_located((By.XPATH,"//div[contains(.,'Loading more')]")))
    except:
        print('Re-try.........')
        #scroll page up to certain limit
        driver.find_element_by_tag_name("body").send_keys(Keys.UP)
        time.sleep(1)

输出：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
Re-try.........
30
31
32
33
Re-try.........
34
35
36
37
38
Re-try.........
39
40
41
Re-try.........
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
Re-try.........
81
82
83
84
85
Re-try.........
86
87
88
Re-try.........
89
90
91
Re-try.........
92
Re-try.........
93
94
95
96
97
98
99
100
101
102
103
Re-try.........
104
105
106
107
Re-try.........
108
Re-try.........
109
110
Re-try.........
111
112
113
114
115
116
117
Re-try.........
118
119
Re-try.........
120
121
122
123
Re-try.........
124
Re-try.........
125
126
127
128
Re-try.........
129
130
Re-try.........
131
132
133
134
Re-try.........
135
136
137
Re-try.........
138
Re-try.........
139
140
141
Re-try.........
142
143
Re-try.........
144
Re-try.........
145
Re-try.........
146
147
148
Re-try.........
149
Re-try.........
150
Re-try.........
151
152
Re-try.........
153
154
Re-try.........
155
Re-try.........
156
Re-try.........
157
158
Re-try.........
159
Re-try.........
160 and son on...