使用 selenium 和 webdriver 抓取 twitter 时不断收到“该页面不存在”错误答案

【问题标题】：Keep getting "that page doesn't exist" error when scraping twitter with selenium and webdriver使用 selenium 和 webdriver 抓取 twitter 时不断收到“该页面不存在”错误
【发布时间】：2020-06-05 09:20:45
【问题描述】：

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get("https://twitter.com/WholesomeMeme")
html = driver.page_source
driver.close()
open("twitter.html", "w", encoding='utf8').write(html)

通过执行上面的代码成功打开了一个 Google chrome 实例，我能够看到 WholesomeMeme 个人资料页面，但是当我打开 twitter.html 时，只显示“抱歉，该页面不存在”。

【问题讨论】：

标签： selenium selenium-webdriver twitter webdriver

【解决方案1】：

我已将您的代码与几个附加选项一起使用，并且能够在将部分页面源写入twitter.html之前部分检索页面源，如下所示：

代码块：

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://twitter.com/WholesomeMeme')
html = driver.page_source
print(driver.page_source)
driver.quit()
open("twitter.html", "w", encoding='utf8').write(html)

控制台输出：

<html dir="ltr" lang="en" style="font-size: 15px;"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=0,viewport-fit=cover">
.
<html dir="ltr" lang="en" style="font-size: 15px;"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=0,viewport-fit=cover">
.
.
.
  <noscript>
    <form action="https://mobile.twitter.com/i/nojs_router?path=%2FWholesomeMeme" method="POST" style="background-color: #fff; position: fixed; top: 0; left: 0; right: 0; bottom: 0; z-index: 9999;">
      <div style="font-size: 18px; font-family: Helvetica,sans-serif; line-height: 24px; margin: 10%; width: 80%;">
    <p>We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter?</p>
    <p style="margin: 20px 0;">
      <button type="submit" style="background-color: #1da1f2; border-radius: 100px; border: none; box-shadow: none; color: #fff; cursor: pointer; font-size: 14px; font-weight: bold; line-height: 20px; padding: 6px 16px;">Yes</button>
    </p>
      </div>
    </form>
  </noscript>
  .
  <div id="react-root" style="height:100%;display:flex;"><div class="css-1dbjc4n r-13awgt0 r-12vffkv">
  .
//# sourceMappingURL=runtime.19384dc4.js.map
</script>
<script type="text/javascript" charset="utf-8" nonce="" crossorigin="anonymous" src="https://abs.twimg.com/responsive-web/web/polyfills.604422d4.js"></script>
<script type="text/javascript" charset="utf-8" nonce="" crossorigin="anonymous" src="https://abs.twimg.com/responsive-web/web/vendors~main.31775294.js"></script>
<script type="text/javascript" charset="utf-8" nonce="" crossorigin="anonymous" src="https://abs.twimg.com/responsive-web/web/i18n-rweb/en.cb24d894.js"></script>
<script type="text/javascript" charset="utf-8" nonce="" crossorigin="anonymous" src="https://abs.twimg.com/responsive-web/web/i18n-horizon/en.d212af84.js"></script>
<script type="text/javascript" charset="utf-8" nonce="" crossorigin="anonymous" src="https://abs.twimg.com/responsive-web/web/main.a4500454.js"></script>
<script nonce="">
  if (!window.__SCRIPTS_LOADED__['main']) {
    document.getElementById('ScriptLoadFailure').style.display = 'block';
  }
</script>
<script async="" src="https://twitter.com/i/js_inst?c_name=ui_metrics"></script></body></html>
[11684:11992:0605/150054.450:ERROR:broker_win.cc(55)] Error reading broker pipe: The pipe has been ended. (0x6D)

观察：观察到以下错误：

Error reading broker pipe: The pipe has been ended. (0x6D)

读取代理管道时出错：管道已结束。 (0x6D)

此错误消息...

ERROR:broker_win.cc(55)] Error reading broker pipe: The pipe has been ended. (0x6D)

...表示管道已损坏，就像浏览器端已关闭一样。

此错误在 Chromium 代码存储库中的 broker_win.cc 中定义如下：

Channel::MessagePtr WaitForBrokerMessage(PlatformHandle platform_handle,
                     BrokerMessageType expected_type) {
  char buffer[kMaxBrokerMessageSize];
  DWORD bytes_read = 0;
  BOOL result = ::ReadFile(platform_handle.handle, buffer,
               kMaxBrokerMessageSize, &bytes_read, nullptr);
  if (!result) {
    // The pipe may be broken if the browser side has been closed, e.g. during
    // browser shutdown. In that case the ReadFile call will fail and we
    // shouldn't continue waiting.
    PLOG(ERROR) << "Error reading broker pipe";
    return nullptr;
  }

原因

您看到此错误的主要原因是因为ChromeDriver 控制的 Chrome 浏览器被检测到并且导航被阻止。

解决方案

作为一种解决方案，您可能需要使用某些配置来配置 ChromeDriver / Chrome，因此 Selenium 驱动的 Chrome 浏览上下文 不会被检测到。

参考文献

您可以在以下位置找到一些相关的详细讨论：

tl;博士

Broken pipe error selenium webdriver, when there is a gap between commands?

【讨论】：