无法使用 Java 中的 WebClient 连接到特定的 https 网站

【问题标题】：Can't connect to specific https website using WebClient in Java无法使用 Java 中的 WebClient 连接到特定的 https 网站
【发布时间】：2020-10-24 15:51:06
【问题描述】：

总结：我创建了一个简单的代码块，过去我曾多次成功使用过，但它不适用于本网站（“https://mid-usa.com/login”）因为某些原因。我可以从我的电脑远程登录到 443 上的 mid-usa.com，我可以在浏览器中打开这个页面。

    WebClient client = new WebClient(BrowserVersion.BEST_SUPPORTED);
    client.getOptions().setSSLClientProtocols(new String[] { "TLSv1.2","TLSv1.1"});
    client.getOptions().setCssEnabled(false);
    client.getOptions().setJavaScriptEnabled(false);
    client.getOptions().setUseInsecureSSL(true);
    client.getOptions().setTimeout(10000);
    HtmlPage page = (HtmlPage) client.getPage("https://mid-usa.com/login");

我将 setSSLClientProtocols 设置为 TLSv1.2 的原因是，在查看运行扫描仪的网站协议时，我认为它会有所帮助（通过阅读其他帖子），但它没有。我也只留下了 TLSv1.2，仍然得到以下打印输出：

Jul 03, 2020 3:23:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {s}->https://mid-usa.com:443: Connection reset
Jul 03, 2020 3:23:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://mid-usa.com:443
Jul 03, 2020 3:23:29 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {s}->https://mid-usa.com:443: Connection reset
Jul 03, 2020 3:23:29 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://mid-usa.com:443
Jul 03, 2020 3:23:29 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {s}->https://mid-usa.com:443: Connection reset
Jul 03, 2020 3:23:29 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://mid-usa.com:443
Connection resetjava.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(Unknown Source)
    at java.net.SocketInputStream.read(Unknown Source)
    at sun.security.ssl.InputRecord.readFully(Unknown Source)
    at sun.security.ssl.InputRecord.read(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
    at com.gargoylesoftware.htmlunit.httpclient.HtmlUnitSSLConnectionSocketFactory.connectSocket(HtmlUnitSSLConnectionSocketFactory.java:184)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
    at com.gargoylesoftware.htmlunit.HttpWebConnection.getResponse(HttpWebConnection.java:193)
    at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1537)
    at com.gargoylesoftware.htmlunit.WebClient.loadWebResponse(WebClient.java:1456)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:447)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:368)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:520)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:501)
    at ScraperBase.autoLogin(ScraperBase.java:25)
    at MidwayDataCollection$4.actionPerformed(MidwayDataCollection.java:222)
    at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
    at javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
    at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
    at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
    at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
    at java.awt.Component.processMouseEvent(Unknown Source)
    at javax.swing.JComponent.processMouseEvent(Unknown Source)
    at java.awt.Component.processEvent(Unknown Source)
    at java.awt.Container.processEvent(Unknown Source)
    at java.awt.Component.dispatchEventImpl(Unknown Source)
    at java.awt.Container.dispatchEventImpl(Unknown Source)
    at java.awt.Component.dispatchEvent(Unknown Source)
    at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
    at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
    at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
    at java.awt.Container.dispatchEventImpl(Unknown Source)
    at java.awt.Window.dispatchEventImpl(Unknown Source)
    at java.awt.Component.dispatchEvent(Unknown Source)
    at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
    at java.awt.EventQueue.access$500(Unknown Source)
    at java.awt.EventQueue$3.run(Unknown Source)
    at java.awt.EventQueue$3.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
    at java.awt.EventQueue$4.run(Unknown Source)
    at java.awt.EventQueue$4.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
    at java.awt.EventQueue.dispatchEvent(Unknown Source)
    at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
    at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
    at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
    at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
    at java.awt.EventDispatchThread.run(Unknown Source)

有什么想法吗？

【问题讨论】：

  • 看看这是否有帮助stackoverflow.com/questions/5507878/ssl-connection-reset
  • 感谢您提供链接 tgdavies。在搜索我的问题时，我注意到了这一点并进行了扫描，发现它支持 TLSv1.2，这就是我在代码中设置它的原因。我还尝试使用较低级别的库，如帖子中所述，手动创建套接字。我仍然会重置连接。我不知道为什么，而且我没有想法。浏览器工作正常。
  • 在重定向的情况下，我确实添加了重定向“client.getOptions().setRedirectEnabled(true);”我仍然遇到与连接重置相同的问题。我只是不确定为什么它只适用于这个特定的网站。我对其进行了扫描并启用了协议。我增加了超时。我可以将 URL 更改为其他任何内容，没有问题，但我可以使用实际浏览器轻松访问此页面。关于我可以在哪里准确检测到该网站的特别之处以及我需要克服的任何想法？

标签： java web-scraping webclient htmlunit


【解决方案1】：

我认为问题在于服务器要使用的密码套件在以后的 JDK 中被禁用。

当我通过curl -v https://mid-usa.com >/dev/null 连接时，我得到了：

* SSL connection using TLSv1.2 / DES-CBC3-SHA

这映射到 JSSE 中的 SSL_RSA_WITH_3DES_EDE_CBC_SHA (https://docs.oracle.com/cd/E19728-01/820-2550/cipher_suites.html)

我重写了您的示例以仅使用 URLConnection：

        URL url = new URL("https://mid-usa.com");
        URLConnection urlConnection = url.openConnection();
        InputStream is = urlConnection.getInputStream();

然后用-Djavax.net.debug=all -Dhttps.cipherSuites="SSL_RSA_WITH_3DES_EDE_CBC_SHA" 运行它

Ignoring disabled cipher suite: SSL_RSA_WITH_3DES_EDE_CBC_SHA for TLSv1.2
No available cipher suite for TLSv1.2
main, handling exception: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)

所以我认为服务器想要使用 JDK 不允许的密码。

这个 Jira 问题似乎与 https://bugs.openjdk.java.net/browse/JDK-8213846 类似——发送握手后服务器断开连接。

所以现在的问题变成了：如何启用该密码套件，我还没有弄清楚该怎么做！

【讨论】：

  • 是的，8u171 和 j10 up 默认禁用 3DES 密码套件，在 jdk.tls.disabledAlgorithms 安全属性中；见sweet32.info。这可以通过 (1) 编辑 j8/lib/security/java.security 或 j9+/conf/security/java.security (2) 在别处创建已编辑的 java.security 并使用 sysprop java.security.properties 指向它来更改，请参阅stackoverflow.com/questions/4521119 和 stackoverflow.com/questions/35140079 (3) 在代码的早期（在加载 TLS 类之前）调用 Security.setProperty。 SSLabs 为此给该服务器打了 C 级！
  • 你们太棒了。太感谢了。我从您的链接中学到了很多东西，并且能够解决问题。 java.security 文件默认启用了对其进行编辑的能力 (security.overridePropertiesFile=true)。我能够在我的代码中设置以下内容，而无需在环境中编辑任何内容：Security.setProperty("jdk.tls.disabledAlgorithms", "RC4, MD5withRSA"); Security.setProperty("jdk.tls.legacyAlgorithms", "TLS_RSA_WITH_3DES_EDE_CBC_SHA"); System.setProperty("https.protocols", "TLSv1,TLSv1.1");
猜你喜欢
  • 2017-07-13
  • 2019-12-29
  • 1970-01-01
  • 2013-05-19
  • 1970-01-01
  • 1970-01-01
  • 2020-12-17
  • 1970-01-01
  • 1970-01-01
相关资源
最近更新 更多
热门标签
Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode