尽管检测到所有元素，但未选择所有元素

Question

我有一个脚本，可以在this page 底部单击“显示更多”四次以显示更多评论线程。

即使我的XPATH 将选择所有“查看更多回复.../查看更多回复...”元素，脚本也不会最终单击所有这些元素。 （在撰写本文时，它只点击了 13 个元素中的 7 个。）

XPath 选择器

//ui-view//a[contains(@class, "commentAction")]

脚本部分（很长，如果您想/需要看更多内容，请告诉我。）：

tab_comments = browser.find_elements_by_xpath('//a[@gogo-test="comments_tab"]')

if len(tab_comments) > 0:

    browser.implicitly_wait(5)

    try:
        comments_count = int(''.join(filter(str.isdigit, str(tab_comments[0].text))))
    except ValueError:
        comments_count = 0

    if comments_count > 0:
        # 1. Switch to Comments Tab
        tab_comments[0].click()

        # 2. Expose Additional Threads
        show_more_comments = WebDriverWait(browser, 10).until(
            EC.element_to_be_clickable((By.XPATH, '//ui-view//a[text()="show more"]'))
        )

        clicks = 0
        while clicks <= 3:
            try:
                clicks += 1
                show_more_comments.click()
            except Exception:
                break

        # 3. Expand All Threads
        see_n_more_replies = browser.find_elements_by_xpath('//ui-view//a[contains(@class, "commentAction")]')
        for idx, see_replies in enumerate(see_n_more_replies):
            print('\n\n\n\nidx: ' + str(idx) + '\n\n\n\n')
            see_replies.click()

按钮是否需要在视图中才能单击它们？（其他按钮似乎不是这种情况，但此时我正在抓住稻草。 )

问题是我在步骤# 4. ... 中解析了 cmets，并且由于它不会以多个响应扩展所有线程，这是它应该做的，这些字段在日志中变为空.

不会引发错误或异常。

我正在使用 Firefox/geckodriver。

Answer 1

执行以下代码sn-p，通过点击“显示更多”，加载页面上的所有cmets，直到显示更多链接消失

comment_pages = 0
no_of_comments = len(driver.find_elements_by_tag_name('desktop-comment'))
while True:
    show_more_link = driver.find_elements_by_partial_link_text('show more')
    if len(show_more_link) == 0:  # if the 'show more' link does not exist on the page
        break
    # before clicking on the link, it is important to bring the link inside the viewport. Otherwise `ElementNotVisible` exception is encountered
    driver.execute_script('arguments[0].scrollIntoView(true);', show_more_link[0])
    show_more_link[0].click()
    try:
        # wait for more comments to load by waiting till the comment count after clicking the button is greater than before the click
        WebDriverWait(driver, 10, poll_frequency=2).until(lambda x: len(driver.find_elements_by_tag_name('desktop-comment')) > no_of_comments)
    except:
        break
    no_of_comments = len(driver.find_elements_by_tag_name('desktop-comment'))
    comment_pages += 1

执行此代码后，您的 dom 包含所有 cmets 的内容。发布您开始实际抓取页面的帖子。

comments = driver.find_elements_by_tag_name('desktop-comment')
for comment in comments:
    author = comment.find_element_by_xpath(".//div[@class='commentLayout-header']/a[contains(@href, 'individuals')]").text
    print 'Comment by person : ' + author

    has_more_replies = len(comment.find_elements_by_partial_link_text("more replies...")) > 0
    if has_more_replies:
        more_replies = comment.find_element_by_partial_link_text("more replies...")
        driver.execute_script('arguments[0].scrollIntoView()', more_replies)
        more_replies.click()
    reply_count = len(comment.find_elements_by_xpath(".//div[contains(@class, 'commentLayout-reply')]"))
    print 'Number of replies to the comment : ' + str(reply_count)
    print '-------------------------------------------------------------------'

其输出如下：

Comment by person : Jeff Rudd
Number of replies to the comment : 1
-------------------------------------------------------------------
Comment by person : Martin Boyle
Number of replies to the comment : 1
-------------------------------------------------------------------
Comment by person : John Bickerton
Number of replies to the comment : 1
-------------------------------------------------------------------
Comment by person : Mikkel Taanning
Number of replies to the comment : 2
-------------------------------------------------------------------
Comment by person : Christopher Sams
Number of replies to the comment : 2
-------------------------------------------------------------------
Comment by person : Marc Vieux
Number of replies to the comment : 2
-------------------------------------------------------------------

.......................

您可以修改 for 循环以获取有关 cmets 的更多详细信息