为什么 Mechanize HTML 内容打印 <!-- Session expired -->？

【问题标题】：Why does Mechanize HTML content print <!-- Session expired -->?为什么 Mechanize HTML 内容打印 <!-- Session expired -->？
【发布时间】：2014-01-13 10:07:47
【问题描述】：

我想使用 Mechanize (2.7.3) 登录我的银行帐户并打印当前余额。

但在成功登录后，有些地方不太对劲，因为 Mechanize 没有显示页面上应该显示的正确信息。

输出很有趣。尤其是&lt;!-- Session expired --&gt; 部分。

这是我正在使用的代码：

require 'mechanize'
require 'logger'

# Instantiate a new Mechanize object
a = Mechanize.new do |agent|
    agent.user_agent_alias = 'Mac Safari'
    agent.follow_meta_refresh = true
    agent.log = Logger.new "mech.log"
end

# Fetch URL with Mechanize
a.get('https://mijn.ing.nl/internetbankieren/SesamLoginServlet') do |page|
    login_form = page.forms.first

    # Store the randomly generated input names into variables
    username_input_name = login_form.fields[0].name
    password_input_name = login_form.fields[1].name

    # Fill in the username and password form
    login_form.field_with(:name => username_input_name).value = 'username'
    login_form.field_with(:name => password_input_name).value = 'password'

    # Login
    dashboard_page = login_form.submit(nil, {'Cookie' => HTTP::Cookie.cookie_value(a.cookie_jar.cookies)})

    puts dashboard_page.content

    # Check if the login was successfull
    puts    "=================="
    puts check_1 = dashboard_page.title == 'Mijn ING Overzicht  - Mijn ING' ?  "CHECK 1 LOGIN SUCCESS" : "CHECK 1 LOGIN FAIL"
    puts    "=================="
end

这是a.follow_meta_refresh = true 的控制台输出：

<html>
<body>
<!-- Session expired -->
<script>
<!-- Hide script from old browsers
function urlencode(str) {
return escape(str).replace('%3A', ':').replace('+', '%2B').replace('%20', '+').replace('*', '%2A').replace('/', '%2F').replace('@', '%40').replace('%2F', '/');
}
var url_encoded_referrer = urlencode(document.location);
var url = 'https://' + document.location.hostname + '/ssm/sso/login?Target=' + url_encoded_referrer;
window.location = url;
//-- Stop hiding script -->
</script>
</body>
</html>
==================
CHECK 1 LOGIN FAIL
==================

这是日志文件：

# Logfile created on 2013-12-27 09:16:17 +0100 by logger.rb/41954
I, [2013-12-27T09:16:17.112862 #978]  INFO -- : Net::HTTP::Get: /internetbankieren/SesamLoginServlet
D, [2013-12-27T09:16:17.112910 #978] DEBUG -- : request-header: accept-encoding => gzip,deflate,identity
D, [2013-12-27T09:16:17.112935 #978] DEBUG -- : request-header: accept => */*
D, [2013-12-27T09:16:17.112957 #978] DEBUG -- : request-header: user-agent => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/534.51.22 (KHTML, like Gecko) Version/5.1.1 Safari/534.51.22
D, [2013-12-27T09:16:17.112978 #978] DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2013-12-27T09:16:17.112999 #978] DEBUG -- : request-header: accept-language => en-us,en;q=0.5
D, [2013-12-27T09:16:17.113019 #978] DEBUG -- : request-header: host => mijn.ing.nl
I, [2013-12-27T09:16:17.678728 #978]  INFO -- : status: Net::HTTPOK 1.1 200 OK
D, [2013-12-27T09:16:17.678787 #978] DEBUG -- : response-header: date => Fri, 27 Dec 2013 08:16:17 GMT
D, [2013-12-27T09:16:17.678813 #978] DEBUG -- : response-header: cache-control => no-cache, no-store
D, [2013-12-27T09:16:17.678834 #978] DEBUG -- : response-header: pragma => no-cache
D, [2013-12-27T09:16:17.678855 #978] DEBUG -- : response-header: expires => Thu, 01 Jan 1970 00:00:00 GMT
D, [2013-12-27T09:16:17.678877 #978] DEBUG -- : response-header: set-cookie => sessiontype=mpb; Secure, aac=332016DB0A85C1245596211F7D403A78; Expires=Sat, 27 Dec 2014 08:16:17 GMT; Domain=.ing.nl; Secure, internetbankierenmi=1575004352.20480.0000; path=/, TS765584=312694932a75764058a09928f7990ca23a092c849a2d804452bd3751d4efbe89959deba9debace3fc06c3994e27e0b6b0f2fcdc7; Path=/
D, [2013-12-27T09:16:17.678900 #978] DEBUG -- : response-header: vary => Accept-Encoding,User-Agent
D, [2013-12-27T09:16:17.678921 #978] DEBUG -- : response-header: content-encoding => gzip
D, [2013-12-27T09:16:17.678942 #978] DEBUG -- : response-header: keep-alive => timeout=90, max=1000
D, [2013-12-27T09:16:17.678962 #978] DEBUG -- : response-header: connection => Keep-Alive
D, [2013-12-27T09:16:17.678983 #978] DEBUG -- : response-header: content-type => text/html;charset=ISO-8859-1
D, [2013-12-27T09:16:17.679003 #978] DEBUG -- : response-header: content-language => en
D, [2013-12-27T09:16:17.679023 #978] DEBUG -- : response-header: transfer-encoding => chunked
D, [2013-12-27T09:16:17.679118 #978] DEBUG -- : Read 10 bytes (10 total)
D, [2013-12-27T09:16:17.679974 #978] DEBUG -- : Read 1443 bytes (1453 total)
D, [2013-12-27T09:16:17.680403 #978] DEBUG -- : Read 1448 bytes (2901 total)
D, [2013-12-27T09:16:17.680469 #978] DEBUG -- : Read 1112 bytes (4013 total)
D, [2013-12-27T09:16:17.680566 #978] DEBUG -- : gzip response
D, [2013-12-27T09:16:17.683042 #978] DEBUG -- : saved cookie: sessiontype=mpb
D, [2013-12-27T09:16:17.683568 #978] DEBUG -- : saved cookie: aac=332016DB0A85C1245596211F7D403A78
D, [2013-12-27T09:16:17.683684 #978] DEBUG -- : saved cookie: internetbankierenmi=1575004352.20480.0000
D, [2013-12-27T09:16:17.683778 #978] DEBUG -- : saved cookie: TS765584=312694932a75764058a09928f7990ca23a092c849a2d804452bd3751d4efbe89959deba9debace3fc06c3994e27e0b6b0f2fcdc7
I, [2013-12-27T09:16:17.685622 #978]  INFO -- : form encoding: ISO-8859-1
D, [2013-12-27T09:16:17.685989 #978] DEBUG -- : query: "a8yzXvCb7f5ZKsFaT=n.aramjan&a9jSD_u8G4nO23oXI=EYoss2303"
I, [2013-12-27T09:16:17.686528 #978]  INFO -- : Net::HTTP::Post: /internetbankieren/SesamLoginServlet
D, [2013-12-27T09:16:17.686562 #978] DEBUG -- : request-header: accept-encoding => gzip,deflate,identity
D, [2013-12-27T09:16:17.686587 #978] DEBUG -- : request-header: accept => */*
D, [2013-12-27T09:16:17.686609 #978] DEBUG -- : request-header: user-agent => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/534.51.22 (KHTML, like Gecko) Version/5.1.1 Safari/534.51.22
D, [2013-12-27T09:16:17.686631 #978] DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2013-12-27T09:16:17.686653 #978] DEBUG -- : request-header: accept-language => en-us,en;q=0.5
D, [2013-12-27T09:16:17.686677 #978] DEBUG -- : request-header: cookie => TS765584=312694932a75764058a09928f7990ca23a092c849a2d804452bd3751d4efbe89959deba9debace3fc06c3994e27e0b6b0f2fcdc7; aac=332016DB0A85C1245596211F7D403A78; internetbankierenmi=1575004352.20480.0000; sessiontype=mpb
D, [2013-12-27T09:16:17.686711 #978] DEBUG -- : request-header: host => mijn.ing.nl
D, [2013-12-27T09:16:17.686739 #978] DEBUG -- : request-header: referer => https://mijn.ing.nl/internetbankieren/SesamLoginServlet
D, [2013-12-27T09:16:17.686760 #978] DEBUG -- : request-header: content-type => application/x-www-form-urlencoded
D, [2013-12-27T09:16:17.686781 #978] DEBUG -- : request-header: content-length => 55
I, [2013-12-27T09:16:18.225851 #978]  INFO -- : status: Net::HTTPOK 1.1 200 OK
D, [2013-12-27T09:16:18.225913 #978] DEBUG -- : response-header: date => Fri, 27 Dec 2013 08:16:17 GMT
D, [2013-12-27T09:16:18.225938 #978] DEBUG -- : response-header: x-ci => r=HPHBPNBMP;c=;a=VRRCSL;u=JXWMBKM
D, [2013-12-27T09:16:18.225962 #978] DEBUG -- : response-header: cache-control => no-cache, no-store, no-cache, no-store
D, [2013-12-27T09:16:18.225983 #978] DEBUG -- : response-header: pragma => no-cache, no-cache
D, [2013-12-27T09:16:18.226005 #978] DEBUG -- : response-header: expires => Thu, 01 Jan 1970 00:00:00 GMT
D, [2013-12-27T09:16:18.226030 #978] DEBUG -- : response-header: set-cookie => SESSESSIONID=0000e3Sar84Bz7XlOymbQkge4gX:17s9ivth6; Path=/; Domain=.ing.nl; Secure, iid=VQGIG0Xq3%2BzTjLSPo5YDiw%3D%3D%3Bmss1; Expires=Wed, 26 Feb 2014 08:16:17 GMT; Path=/; Domain=.ing.nl, gsc=m=s; HttpOnly; Path=/; Domain=.mijn.ing.nl; Secure, Session201=HKVCKUYMLURFZQIENENOANWSSNRUERNF051e7e43; HttpOnly; Path=/; Domain=.mijn.ing.nl; Secure, cookiepref=3; Expires=Sat, 27 Dec 2014 08:16:17 GMT; Path=/; Domain=.ing.nl, TS765584=9e3c30b00a5f1162344f71403a543e573a092c849a2d804452bd3751d4efbe89959deba9debace3fc06c3994e27e0b6b0f2fcdc7eccc06cabcea90646ac25a59abcf00af2532f3ca7d510a274552b96c4b433c8b5f6a631185c8cdc8; Path=/
D, [2013-12-27T09:16:18.226053 #978] DEBUG -- : response-header: vary => Accept-Encoding,User-Agent
D, [2013-12-27T09:16:18.226073 #978] DEBUG -- : response-header: content-encoding => gzip
D, [2013-12-27T09:16:18.226094 #978] DEBUG -- : response-header: keep-alive => timeout=90, max=1000
D, [2013-12-27T09:16:18.226115 #978] DEBUG -- : response-header: connection => Keep-Alive
D, [2013-12-27T09:16:18.226139 #978] DEBUG -- : response-header: content-type => text/html;charset=ISO-8859-1
D, [2013-12-27T09:16:18.226159 #978] DEBUG -- : response-header: content-language => en
D, [2013-12-27T09:16:18.226180 #978] DEBUG -- : response-header: transfer-encoding => chunked
D, [2013-12-27T09:16:18.226252 #978] DEBUG -- : Read 326 bytes (326 total)
D, [2013-12-27T09:16:18.226381 #978] DEBUG -- : gzip response
D, [2013-12-27T09:16:18.226730 #978] DEBUG -- : saved cookie: SESSESSIONID=0000e3Sar84Bz7XlOymbQkge4gX:17s9ivth6
D, [2013-12-27T09:16:18.226933 #978] DEBUG -- : saved cookie: iid=VQGIG0Xq3%2BzTjLSPo5YDiw%3D%3D%3Bmss1
D, [2013-12-27T09:16:18.227051 #978] DEBUG -- : saved cookie: gsc=m=s
D, [2013-12-27T09:16:18.227152 #978] DEBUG -- : saved cookie: Session201=HKVCKUYMLURFZQIENENOANWSSNRUERNF051e7e43
D, [2013-12-27T09:16:18.227311 #978] DEBUG -- : saved cookie: cookiepref=3
D, [2013-12-27T09:16:18.227423 #978] DEBUG -- : saved cookie: TS765584=9e3c30b00a5f1162344f71403a543e573a092c849a2d804452bd3751d4efbe89959deba9debace3fc06c3994e27e0b6b0f2fcdc7eccc06cabcea90646ac25a59abcf00af2532f3ca7d510a274552b96c4b433c8b5f6a631185c8cdc8
I, [2013-12-27T09:16:18.230318 #978]  INFO -- : Net::HTTP::Get: /particulier/betalen/index
D, [2013-12-27T09:16:18.230355 #978] DEBUG -- : request-header: accept-encoding => gzip,deflate,identity
D, [2013-12-27T09:16:18.230379 #978] DEBUG -- : request-header: accept => */*
D, [2013-12-27T09:16:18.230401 #978] DEBUG -- : request-header: user-agent => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/534.51.22 (KHTML, like Gecko) Version/5.1.1 Safari/534.51.22
D, [2013-12-27T09:16:18.230421 #978] DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2013-12-27T09:16:18.230456 #978] DEBUG -- : request-header: accept-language => en-us,en;q=0.5
D, [2013-12-27T09:16:18.230478 #978] DEBUG -- : request-header: cookie => SESSESSIONID=0000e3Sar84Bz7XlOymbQkge4gX:17s9ivth6; Session201=HKVCKUYMLURFZQIENENOANWSSNRUERNF051e7e43; cookiepref=3; gsc=m=s; iid=VQGIG0Xq3%2BzTjLSPo5YDiw%3D%3D%3Bmss1
D, [2013-12-27T09:16:18.230501 #978] DEBUG -- : request-header: host => bankieren.mijn.ing.nl
I, [2013-12-27T09:16:18.714637 #978]  INFO -- : status: Net::HTTPOK 1.1 200 OK
D, [2013-12-27T09:16:18.714692 #978] DEBUG -- : response-header: content-length => 506
D, [2013-12-27T09:16:18.714716 #978] DEBUG -- : response-header: content-type => text/html
D, [2013-12-27T09:16:18.714738 #978] DEBUG -- : response-header: date => Fri, 27 Dec 2013 08:16:18 GMT
D, [2013-12-27T09:16:18.714760 #978] DEBUG -- : response-header: p3p => CP="NON CUR OTPi OUR NOR UNI"
D, [2013-12-27T09:16:18.714781 #978] DEBUG -- : response-header: cache-control => no-cache
D, [2013-12-27T09:16:18.714801 #978] DEBUG -- : response-header: pragma => no-cache
D, [2013-12-27T09:16:18.714825 #978] DEBUG -- : response-header: set-cookie => S-SESSION-ID=2_0_nG1XBMZWH2kJdOBjCloZA651nXttTm5wiYowRBYDQY7gxH1L; Path=/; Secure; HttpOnly, lb-4-25a-1=208176394.20480.0000; path=/, TS31e294=06caec21f242b201e4b52847e45a8a475bd58411f71b544852bd3752022ecc54fb717d9b1b4d09a87ea0bb21; Path=/
D, [2013-12-27T09:16:18.714889 #978] DEBUG -- : Read 506 bytes (506 total)
D, [2013-12-27T09:16:18.715236 #978] DEBUG -- : saved cookie: S-SESSION-ID=2_0_nG1XBMZWH2kJdOBjCloZA651nXttTm5wiYowRBYDQY7gxH1L
D, [2013-12-27T09:16:18.715340 #978] DEBUG -- : saved cookie: lb-4-25a-1=208176394.20480.0000
D, [2013-12-27T09:16:18.715437 #978] DEBUG -- : saved cookie: TS31e294=06caec21f242b201e4b52847e45a8a475bd58411f71b544852bd3752022ecc54fb717d9b1b4d09a87ea0bb21

有什么想法吗？要是能在圣诞节解决这个问题就好了！

【问题讨论】：

  • 也许您需要将 cookie 与提交请求一起发送，如下所示：login_form.submit(nil, {'Cookie' =&gt; HTTP::Cookie.cookie_value(a.cookie_jar.cookies)})
  • 不幸的是还是一样的结果
  • 你可能被屏蔽了，尝试设置 Mozila 或 Sfari 等代理，在 Google 中查看。
  • @PepperoniPizza 结果还是一样。这次我使用logger 记录了一次尝试。查看上面日志文件的内容，可能有用吗？谢谢

标签： ruby mechanize


【解决方案1】：

仔细检查 cookie 是否在点击后保持不变，并尝试 selenium - 可能该页面上出现了一些有趣的 JS 内容，而 selenium 会为您处理它 (http://rubygems.org/gems/selenium-webdriver)

更新

有关 Mechanize 中 cookie 处理的更多信息，请参阅此答案：Maintaining cookies between Mechanize requests

另外，请尝试在浏览器中禁用 JS，然后登录您的银行，确保在没有 JS 的情况下一切仍然正常（如果可以正常工作，则很好，如果不能，则需要使用支持 JS 的驱动程序）

结论

因为在这种情况下是银行的网站，它依赖于客户端 JavaScript 代码才能正常运行，因此像 mechanize 这样不支持 JS 的驱动程序将无法运行。您将需要使用支持 JavaScript 的驱动程序之一 - 其中最流行的是 selenium、webkit 和 poltergeist - 都不错，但都真正令人满意。

考虑到这三个都需要一些主机端依赖项（尽管至少是 poltergeist），“什么安装最好”可能是在它们之间进行选择的主要因素。

对于 Heroku 部署....

  1. poltergeist 似乎确实存在一些问题（与 websocket 相关，似乎 heroku 阻止了它们）但这是部署 heroku https://github.com/jonleighton/poltergeist/issues/194 的最佳选择
  2. 对于 selenium 和 webkit，快速 google 显示了许多问题 - 它们似乎需要 X 和/或 xvfb，请参阅 Is it possible to run capybara-webkit (i.e. forked webkit_server) on Heroku Cedar? 和 Is there a working nodejs/phantomjs Heroku buildpack? 和 Running selenium browser on server (Flask/Python/Heroku) 以进一步阅读

【讨论】：

  • 如何仔细检查 cookie 是否在点击后保留？我不喜欢硒。我已经使用名为 Watir (watir.com) 的 Selenium 替代品成功完成了这项工作。但我想解决机械化问题并让它发挥作用。
  • @narzero 我用关于使用 Mechanize 处理 cookie 的 SO 链接更新了答案
  • 感谢您提供更新的答案。我禁用了 JS 并登录了我的银行帐户。它不起作用，我看到一个空白页面，当我查看页面的源代码时，它与我的 Mechanize 代码的控制台输出完全相同。可以肯定地说，我需要一个支持 JS 的驱动程序。你碰巧认识一个好人吗？
  • @narzero，不，它们都很糟糕 :)（请参阅我在 stackoverflow.com/a/20888890/1983072 上的咆哮）我在 webkit、selenium 和 poltergeist 方面有足够的经验。我主要使用 poltergeist 进行无头执行，但它有其怪癖。当您尝试在远程主机（依赖项）上部署这些测试时会出现更多问题，因此请考虑“在我的部署主机上安装最好的东西”。但是...如果你有 watir 工作，为什么不去 watir？
  • 我不会选择 Watir，因为我无法将它部署到 Heroku。是的，答案涵盖了我最初的问题。请用结论/原因更新您的原始答案，我会接受。干杯
猜你喜欢
  • 2014-12-15
  • 1970-01-01
  • 2011-12-12
  • 1970-01-01
  • 1970-01-01
  • 2016-01-17
  • 1970-01-01
  • 2014-05-17
  • 2010-11-01
相关资源
最近更新 更多
热门标签
Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode