我需要浏览网页中的每个链接及其子页面链接

Question

我正在使用 selenium 驱动程序并使用 python 脚本来执行此操作。这是我的代码。

d = webdriver.Chrome()
d.get("http://localhost:8080")
list_links = d.find_elements_by_tag_name('a')

for i in list_links:
    print url

上面的程序正确地给出了输出

https://www.w3schools.com/
https://www.ubuntu.com/
None

但是当我编译下面的代码时：

d = webdriver.Chrome()
d.get("http://localhost:8080")
list_links = d.find_elements_by_tag_name('a')

for i in list_links:
    url=i.get_attribute('href')
    print url
    d.get(url)

它成功导航到第一个链接https://www.w3schools.com/。然后它说：

Traceback (most recent call last):
File "web_nav.py", line 20, in <module>
url=i.get_attribute('href')
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 141, in get_attribute
resp = self._execute(Command.GET_ELEMENT_ATTRIBUTE, {'name': name})
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 493, in _execute
return self._parent.execute(command, params)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 256, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
(Session info: chrome=59.0.3071.115)
(Driver info: chromedriver=2.30.477691 
(6ee44a7247c639c0703f291d320bdf05c1531b57),platform=Linux 4.4.0-31-
generic x86_64)

这里我使用的是 ubuntu 14.04，语言 python，我使用的是 selenium 网络驱动程序

Answer 1

首先获取所有网址，然后导航到它们

d = webdriver.Chrome()
d.get("http://localhost:8080")
list_links =  d.find_elements_by_tag_name('a')
urls = []    
for i in list_links:
    urls.append(i.get_attribute('href'))
for url in urls:
    d.get(url)

你可以用一个函数来简化它

def get_link_urls(url,driver):
    driver.get(url)
    urls = []
    for link in d.find_elements_by_tag_name('a'):
        urls.append(link.get_attribute('href'))
    return urls

urls = get_link_urls("http://localhost:8080")
sub_urls = []
for url in urls:
    sub_urls.extend(get_link_urls(url))