如何导航到 Selenium 中的新网页？答案

【问题标题】：How to Navigate to a New Webpage In Selenium?如何导航到 Selenium 中的新网页？
【发布时间】：2014-09-06 16:45:47
【问题描述】：

我有以下代码：

driver.get(<some url>)
for element in driver.find_elements_by_class_name('thumbnail'):
    element.find_element_by_xpath(".//a").click() #this works and navigates to new page
    element.find_element_by_link_text('Click here').click() #this doesn't

需要通过单击指向新页面的缩略图来导航以下HTML（当然是简化的），然后需要单击该新页面中的Click here链接：

<!DOCTYPE html>
<html lang="en-US" prefix="og: http://iuytp.me/ns# fb: http://iuytp.me/ns/fb#">
<head>
<meta charset="UTF-8" />
<title>Releases</title>
</head>

<body class="archive category category-releases category-4 custom-background">
    <div id="main">
        <div id="container" class="one-column">
            <div id="content" role="main">

                <h1 class="page-title">Releases</h1>

            <div id="thumbnail-post-display">
        <div id="thumbnail-post" class="post-7158 post type-post status-publish format-standard has-post-thumbnail hentry category-blog category-designer category-releases category-uncategorized">
            <div class="thumbnail"><a href="http://records.net/uncategorized/designer-7-inch-bufu-records-co-release/" title="Permanent link to Designer &#8211; 7 inch" rel="bookmark"><img width="300" height="300" src="http://records.net/dev/wp-content/uploads/2014/05/dboypledge32-300x300.png" class="attachment-thumbnail wp-post-image" alt="dboypledge3" /></a></div>
            <h2><a href="http://records.net/uncategorized/designer-7-inch-bufu-records-co-release/" title="Permanent link to Designer &#8211; 7 inch" rel="bookmark">Designer &#8211; 7 inch</a></h2>
        </div>
    </div><!--end thumbnail post display-->

            <div id="thumbnail-post-display">
        <div id="thumbnail-post" class="post-7107 post type-post status-publish format-standard has-post-thumbnail hentry category-blog category-releases">
            <div class="thumbnail"><a href="http://records.net/releases/people-2014-tour-demos/" title="Permanent link to All My People &#8211; 2014 Tour Demos" rel="bookmark"><img width="300" height="300" src="http://records.net/dev/wp-content/uploads/2014/04/01_Doubt-mp3-image-300x300.png" class="attachment-thumbnail wp-post-image" alt="" /></a></div>
            <h2><a href="http://records.net/releases/people-2014-tour-demos/" title="Permanent link to All My People &#8211; 2014 Tour Demos" rel="bookmark">All My People &#8211; 2014 Tour Demos</a></h2>
        </div>
    </div><!--end thumbnail post display-->

            <div id="thumbnail-post-display">
        <div id="thumbnail-post" class="post-7089 post type-post status-publish format-standard has-post-thumbnail hentry category-blog category-releases">
            <div class="thumbnail"><a href="http://records.net/releases/sirens-blossom-talk/" title="Permanent link to Syrins &#8211; Boss Talk" rel="bookmark"><img width="300" height="300" src="http://records.net/dev/wp-content/uploads/2014/04/sirens_final_smaller-300x300.jpg" class="attachment-thumbnail wp-post-image" alt="sirens_final_smaller" /></a></div>
            <h2><a href="http://records.net/releases/sirens-blossom-talk/" title="Permanent link to Syrins &#8211; Boss Talk" rel="bookmark">Syrins &#8211; Boss Talk</a></h2>
        </div>
    </div><!--end thumbnail post display-->

            <div id="thumbnail-post-display">
        <div id="thumbnail-post" class="post-7073 post type-post status-publish format-standard has-post-thumbnail hentry category-blog category-releases">
            <div class="thumbnail"><a href="http://records.net/releases/worlds-strongest-man-scares/" title="Permanent link to World&#8217;s Tough Man &#8211; Sorry Scares You" rel="bookmark"><img width="300" height="300" src="http://records.net/dev/wp-content/uploads/2014/03/a2312749950_10-300x300.jpg" class="attachment-thumbnail wp-post-image" alt="a2312749950_10" /></a></div>
            <h2><a href="http://records.net/releases/worlds-strongest-man-scares/" title="Permanent link to World&#8217;s Tough Man &#8211; Sorry Scares You" rel="bookmark">World&#8217;s Tough Man &#8211; Sorry Scares You</a></h2>
        </div>
    </div><!--end thumbnail post display-->

            <div id="thumbnail-post-display">
        <div id="thumbnail-post" class="post-7046 post type-post status-publish format-standard has-post-thumbnail hentry category-blog category-releases">
            <div class="thumbnail"><a href="http://records.net/releases/sundog-space-criminal/" title="Permanent link to Dog &#8211; Space Criminal" rel="bookmark"><img width="300" height="300" src="http://records.net/dev/wp-content/uploads/2014/03/Sundog_cover_high_res-300x300.jpg" class="attachment-thumbnail wp-post-image" alt="dog_cover_high_res" /></a></div>
            <h2><a href="http://records.net/releases/sundog-space-criminal/" title="Permanent link to Dog &#8211; Space Criminal" rel="bookmark">Dog &#8211; Space Criminal</a></h2>
        </div>
    </div><!--end thumbnail post display-->

<div style="clear:both"></div>


        </div><!-- #container -->

    </div><!-- #main -->
</div><!-- #wrapper -->

</div><!--#bg-wrapper-->

</body>
</html>

然而，我的代码吐出以下错误：

Traceback (most recent call last):
  ...
  File "crawler.py", line 17, in main
    driver.find_element_by_link_text('Click here').click()
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 254, in find_element_by_link_text
    return self.find_element(by=By.LINK_TEXT, value=link_text)
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 662, in find_element
    {'using': by, 'value': value})['value']
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 173, in execute
    self.error_handler.check_response(response)
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: u'no such element\n  (Session info: chrome=35.0.1916.153)\n  (Driver info: chromedriver=2.10.267517,platform=Mac OS X 10.9.3 x86_64)'

问题似乎是元素没有被新页面的内容更新。用driver 替换有问题的行的element 也不起作用。我做错了什么？

请注意，我必须能够为所有缩略图执行此操作（因此是 for 循环）。

【问题讨论】：

请将相关的 HTML 代码贴在您尝试此操作的位置。

标签： python-2.7 selenium

【解决方案1】：

你的第一行：

for element in driver.find_elements_by_class_name('thumbnail'):

抓取第一个页面上的所有元素。你的下一行：

element.find_element_by_xpath(".//a").click() #this works and navigates to new page

正如您在评论中指出的那样，

过渡到一个全新的新页面。此时element已经没有了，所以下一行：

element.find_element_by_link_text('Click here').click() #this doesn't

没有机会做任何事情，因为它指的是不存在的东西。这正是NoSuchElementException 告诉您的内容。

您需要从头开始，例如：

driver.find_element_by_link_text('Click here').click()

补充答案：

要解决您的迭代困境，您可以采取以下方法 - 请注意，我不熟悉 python 语法！下面是 Groovy 语法，你必须将其调整为 Python！

// first count the number links you are going to hit; no point in storing this WebElement,
// since it will be gone after we navigate to the first page
def linkCount = driver.findElements(By.className("thumbnail")).size()
// Start a loop based on the count. Inside the loop we are going to have to find each of
// the links again, based on this count. I am going to use XPath; this can probably be done
// on CSS as well. Remember that XPath is 1-based!
(1..linkCount).each {

    // find the element again
    driver.findElement(By.xpath("//div[@class='thumbnail'][$it]/a")).click()

    // do something on the new page ...

    // and go back
    driver.navigate().back()
}

【讨论】：

对，但是我如何在不指定新的element 变量的情况下迭代“缩略图”元素？理想情况下，我希望能够重新分配/更新要使用新页面内容更新的驱动程序对象。
这听起来像是一个完全不同的问题。如果搜索web+scrape+python，应该可以得到很多很多的选择。
所以这对于 Selenium 是不可能的？我必须使用另一个图书馆，比如 Beautiful Soup 来做这种事情？ Selenium 有许多 HTML 解析功能，所以我希望我只需将它用于我的 HTML 解析/导航需求。
一切皆有可能！但是，使用 3.5k+ rep，您应该知道您的 OP 已被回答，并且您现在要求解决一个新问题。
感谢您尝试回答这个问题，但是我需要知道这是否可能仅与 Python 中的 Selenium 相关，并且您的伪代码在这种情况下并没有真正帮助，考虑使用 Selenium 依赖严重依赖它公开的 API，而您的伪代码没有解决这方面的问题。

【解决方案2】：

事实证明，您需要提前存储要导航到的链接。这就是最终为我工作的原因（发现 this thread 很有帮助）：

driver.get(<some url>)
elements = driver.find_elements_by_xpath("//h2/a")

links = []
for i in range(len(elements)):
    links.append(elements[i].get_attribute('href'))

for link in links:
    print 'navigating to: ' + link
    driver.get(link)

    # do stuff within that page here...

    driver.back()

【讨论】：

谢谢！你拯救了我的一天！
有没有办法做到这一点并等待为每个页面加载 javascript？
好问题，不幸的是我已经好几年没接触过硒了，所以我帮不上什么忙。
我收到错误 selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document 我该如何解决这个问题？
我认为你应该就你的新问题提出一个单独的问题，如果这是你尝试的依据，请链接到这个答案。