为什么 selenium 在执行这段代码之前要等待很长时间？

Question

我正在尝试在此页面上进行无限滚动，这是我的代码：

from selenium import webdriver
import time

profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override","Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:28.0) Gecko/20100101 Firefox/28.0")
driver = webdriver.Firefox(profile)

driver.get("http://www.quora.com/Programming-Languages/followers")
for n in range(0,5): # For testing I have capped this at 5, will handle this properly once things start to work.
    driver.execute_script("window.scrollTo(0,1000000);")
    time.sleep(2)

因此，当我运行此程序时，它会在进行任何滚动之前等待很多秒（有时超过 1 分钟），然后在下一次滚动之前再次等待相同的时间。 该代码似乎在其他页面上运行良好。 有关如何解决此问题的任何想法？

当我尝试使用 Chrome 而不是 firefox 时，我收到以下错误： driver = webdriver.Chrome('/home/asdf/apps/chromedrive/chromedriver') 添加到 .py 文件中。

Traceback (most recent call last):
  File "ok.py", line 8, in <module>
    driver = webdriver.Chrome('/home/asdf/apps/chromedrive/chromedriver')
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/chrome/webdriver.py", line 65, in __init__
    keep_alive=True)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 73, in __init__
    self.start_session(desired_capabilities, browser_profile)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 121, in start_session
    'desiredCapabilities': desired_capabilities,
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 171, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 349, in execute
    return self._request(command_info[0], url, body=data)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 379, in _request
    self._conn.request(method, parsed_url.path, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 791, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 772, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 553, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno -2] Name or service not known

Answer 1

切换到Chrome() 帮助我解决了问题：

import time
from selenium import webdriver

followers_per_page = 18

driver = webdriver.Chrome()
driver.get("http://www.quora.com/Programming-Languages/followers")

# get the followers count
element = driver.find_element_by_class_name('count')
followers_count = int(element.text.replace('k', '000').replace('.', ''))
print followers_count

# scroll down the page iteratively with a delay
for _ in xrange(0, followers_count/followers_per_page + 1):
    driver.execute_script("window.scrollTo(0, 0,1000000);")
    time.sleep(2)

仅供参考，我正在使用一种不同的方法：解析关注者的数量并计算每页的关注者，同时考虑到它一次加载 18 个关注者这一事实。

我之前实际上已经研究过类似的 quora 问题，请参阅：

• Scrolling web page using selenium python webdriver

---

嗯，这不是我想到的第一件事。这是故事。

问题是对http://tch840195.tch.quora.com/up/chan5-8886/updates URL 的待处理请求需要几分钟才能完成。这就是让 selenium 认为页面没有完全加载的原因。而且，事情变得更糟了——这是每隔 X 秒就会发生的周期性事情。将其视为长池。

我使用Firefox webdriver 尝试了多种方法来解决这个问题：

• 将webdriver.load.strategy 偏好设置为unstable
• 设置network.http.response.timeout、network.http.connection-timeout和network.http.keep-alive.timeout和network.http.request.max-start-delay首选项

• 设置页面加载超时：

  driver.set_page_load_timeout(3)
  

• 设置脚本超时：

  driver.set_script_timeout(3)
  

• 致电window.stop();，希望它会停止活动请求：

  driver.execute_script('window.stop();')
  

• 更新到最新的 Firefox 和 selenium 软件包版本

另一个可能可行的选项是使用代理服务器以某种方式阻止对 "slow url" 的请求并将 Firefox 指向它，或者，如果可能的话，让 Firefox 知道将 URL 列入黑名单（可能通过扩展)。

另请参阅内部多种解决方法的相关问题：

• Support a timeout argument on page load operations

另见：

• How to Stop the page loading in firefox programmatically?
• Firefox DelayedCommand waits for pending XMLHttpRequests when it doesn't need to.
• FirefoxDriver webdriver.load.strategy unstable findelements getting elements from the wrong page
• Set up a real timeout for loading page in Selenium WebDriver?
• Selenium Firefox Open timeout