Jsoup 无法从网页中获取完整内容？答案

【问题标题】：Jsoup is unable to fetch complete content from a webpage?Jsoup 无法从网页中获取完整内容？
【发布时间】：2015-09-14 03:15:50
【问题描述】：

我正在尝试使用 JSOUP 从以下页面获取内容：

http://www.peshawarairport.com.pk/Schedule.aspx?Type=Arrival

但它不会获取整个页面内容，而是只返回内容，直到标签关闭。它返回的内容如下：

<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"  lang="en" dir="ltr"> <![endif]-->
<!--[if IE 7]>    <html class="no-js lt-ie9 lt-ie8"  lang="en" dir="ltr"> <![endif]-->
<!--[if IE 8]>    <html class="no-js lt-ie9"  lang="en" dir="ltr"> <![endif]-->
<!--[if IE 9]>    <html class="no-js ie9"  lang="en" dir="ltr"> <![endif]-->
<!--[if gt IE 9]><html class="no-js"  lang="en" dir="ltr"> <![endif]-->
<html>
<head>
    <meta charset="utf-8" />
    <script type="text/javascript">        var _prum = { id: "5227f1fbabe53ddc1f000000" }; var PRUM_EPISODES = PRUM_EPISODES || {}; PRUM_EPISODES.q = []; PRUM_EPISODES.mark = function (b, a) { PRUM_EPISODES.q.push(["mark", b, a || new Date().getTime()]) }; PRUM_EPISODES.measure = function (b, a, b) { PRUM_EPISODES.q.push(["measure", b, a, b || new Date().getTime()]) }; PRUM_EPISODES.done = function (a) { PRUM_EPISODES.q.push(["done", a]) }; PRUM_EPISODES.mark("firstbyte"); (function () { var b = document.getElementsByTagName("script")[0]; var a = document.createElement("script"); a.type = "text/javascript"; a.async = true; a.charset = "UTF-8"; a.src = "//rum-static.pingdom.net/prum.min.js"; b.parentNode.insertBefore(a, b) })();</script>
    <link href="~/images/favicon.ico" rel="CAA Shortcut Icon"></link>
    <title>Bacha Khan International Airport, Peshawar | www.peshawarairport.com</title>
    <meta name="description" content="">
    <meta name="apple-mobile-web-app-capable" content="yes" />
    <!-- <meta name="p:domain_verify" content="297cb2c48faff5539c27d75f076408b8"/> -->

    <style type="text/css">
        @import url("http://www.caapakistan.com.pk/css/jiap-website/system.base.css?nkrgyj");
        @import url("http://www.caapakistan.com.pk/css/jiap-website/system.messages.css?nkrgyj");
        @import url("http://www.caapakistan.com.pk/css/jiap-website/system.theme.css?nkrgyj");
    </style>
    <style type="text/css">
        @import url("http://www.caapakistan.com.pk/css/jiap-website/comment.css?nkrgyj");
        @import url("http://www.caapakistan.com.pk/sites/all/modules/contrib/date/date_api/date9687.css?nkrgyj");
        @import url("http://www.caapakistan.com.pk/css/jiap-website/field.css?nkrgyj");
        @import url("http://www.caapakistan.com.pk/css/jiap-website/node.css?nkrgyj");
        @import url("http://www.caapakistan.com.pk/css/jiap-website/search.css?nkrgyj");
        @import url("http://www.caapakistan.com.pk/css/jiap-website/user.css?nkrgyj");
        @import url("http://www.caapakistan.com.pk/sites/all/modules/contrib/workflow/workflow_admin_ui/workflow_admin_ui9687.css?nkrgyj");
        @import url("http://www.caapakistan.com.pk/sites/all/modules/contrib/views/css/views9687.css?nkrgyj");
    </style>
    <style type="text/css">
        @import url("http://www.caapakistan.com.pk/sites/all/modules/contrib/ctools/css/ctools9687.css?nkrgyj");
        @import url("http://www.caapakistan.com.pk/sites/all/modules/contrib/panels/css/panels9687.css?nkrgyj");
    </style>
    <style type="text/css">
        @import url("http://www.caapakistan.com.pk/sites/all/themes/sfo/css/bootstrap-n-responsive.min9687.css?nkrgyj");
        @import url("http://www.caapakistan.com.pk/sites/all/themes/sfo/css/base9687.css?nkrgyj");
        @import url("http://www.caapakistan.com.pk/sites/all/themes/sfo/css/theme_flysfo9687.css?nkrgyj");
        @import url("http://www.caapakistan.com.pk/sites/all/themes/sfo/css/flysfo_cn9687.css?nkrgyj");
        @import url("http://www.caapakistan.com.pk/sites/all/themes/sfo/css/mobilestyle9687.css?nkrgyj");
        @import url("http://www.caapakistan.com.pk/sites/all/themes/sfo/css/jplayer.sfo/jplayer.blue.monday9687.css?nkrgyj");
    </style>
    <script type="text/javascript" src="http://www.caapakistan.com.pk/sites/all/themes/sfo/js/libs/modernizr-2.5.3.min.js"></script>
    <script type="text/javascript">        var switchTo5x = false;</script>
    <script type="text/javascript">        stLight.options({ publisher: "a574d78b-ed29-4436-b50d-0213b9613fe7", doNotHash: true, doNotCopy: true, hashAddressBar: true, offsetTop:

虽然如果我使用浏览器或任何 Rest 客户端点击相同的 url，我会得到整个页面的 html。

Java 代码：

Document doc = Jsoup.connect("http://www.peshawarairport.com.pk/Schedule.aspx?Type=Arrival").userAgent("Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36").maxBodySize(0).timeout(maxTimeout)
                    .get();

【问题讨论】：

对我有用，问题一定出在你没有引用的代码中。
但是你的标题是：是的，JSoup 可以从网页中获取完整的内容。
您的代码运行良好。我读到</html> 和timeout=1000。
@T.J.Crowder 你是对的，问题出在其他地方，jsoup 工作正常。我想我应该删除这个问题:(
@user818455：没必要皱眉。 :-) 很高兴您发现了问题！但是，是的，这个问题将来可能对其他人没有帮助，除非您更新它以描述问题所在，然后发布一个答案，说明您是如何修复它的。如果您认为其他人会犯同样的错误，那将是合适的。但是删除它也很好。最好的，

标签： java web-scraping jsoup

【解决方案1】：

URL url = new URL("https://www.google.com/");
    URLConnection con = url.openConnection();
    InputStream is = con.getInputStream();
    BufferedReader br = new BufferedReader(new InputStreamReader(is));
    String line="";
    while (br.readLine() != null) {
        line = line + br.readLine();
    }
    //System.out.println(line);

Document doc = Jsoup.parse(line.replace("<!--","").replace("-->",""));

试试这个来获取内容

【讨论】：