【发布时间】：2020-10-15 10:31:21
【问题描述】：
数据来自数据库，那么如何用这些数据解析 HTML，在我的代码中，我使用 cookie 直接连接到目标页面，然后连接到设置搜索表单，那么如何用结果解析搜索。它来自后端。下面的代码首先包含我的服务连接代码，然后是搜索表单响应给出的HTML，最后是在提交API post 方法的搜索表单后返回的浏览器网络中出现的html。
Connection.Response redirectResponse = Jsoup.connect(internalConstant.getSearchResultURL())
.userAgent(USER_AGENT)
.sslSocketFactory(utilService.socketFactory())
.cookies(coky)
.method(Method.GET)
.execute();
//search file
String yestarday = utilService.getYesterdayDateString();
logger.info("yestarday date: " + yestarday);
Document responseDocument = redirectResponse.parse();
FormElement searchForm = (FormElement) responseDocument.select("form[id=searchDisputesForm]").first();
checkElement("form element", searchForm);
FormElement form = (FormElement) searchForm;
Element searchField = form.select("input[name=DateFrom]").first();
checkElement("date from: ", searchField);
searchField.val(yestarday);
Element searchField1 = form.select("input[name=DateTo]").first();
checkElement("Date to: ", searchField1);
searchField1.val(yestarday);
Connection.Response searchResponse = form.submit()
.cookies(coky)
.sslSocketFactory(utilService.socketFactory())
.userAgent(USER_AGENT)
.method(Method.POST)
.ignoreHttpErrors(true)
.execute();
//and here is what gives me:
2020-10-15 11:37:38.193 INFO 3360 --- [http-nio-8095-exec-1] c.d.e.serviceImpl.UploaderTemplateImpl : searchResponse :<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<title>404 - File or directory not found.</title>
<style type="text/css">
<!--
body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}
fieldset{padding:0 15px 10px 15px;}
h1{font-size:2.4em;margin:0;color:#FFF;}
h2{font-size:1.7em;margin:0;color:#CC0000;}
h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;}
#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:"trebuchet MS", Verdana, sans-serif;color:#FFF;
background-color:#555555;}
#content{margin:0 0 0 2%;position:relative;}
.content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}
-->
</style>
</head>
<body>
<div id="header">
<h1>Server Error</h1>
</div>
<div id="content">
<div class="content-container">
<fieldset>
<h2>404 - File or directory not found.</h2>
<h3>The resource you are looking for might have been removed, had its name changed, or is temporarily unavailable.</h3>
</fieldset>
</div>
</div>
</body>
</html>
//but the response in the network shows what I want:
<form action="/123Mobile/Portal/Dispute/SearchDisputes" class="form-horizontal" enctype="multipart/form-data" id="searchDisputesForm" method="post"><input name="__RequestVerificationToken" type="hidden" value="ClhG-V26p7Aj3vi6W8tarCwHk_enagGT4mDVv2AZsU4MHATOBTbEQHuFmylooB5qvxF29aF1-wSvZZ26ijlcZJn5p_OSshL3KeqcEbAaYg01" />
<div class='col-md-12'>
<div class='panel panel-grey equal-height-column' id=''>
<div class='panel-heading'>
<h3 class='panel-title'>
<i class='fa fa-tasks'></i>
Search criteria
</h3>
</div>
<div class='panel-body'>
<div class='form-group'> <label class="col-md-2 control-label" for="DateFrom">Date From</label>
<div class='col-md-4 col-md-offset-0' id=''>
<input class="form-control datepicker" data-val="true" data-val-date="The field Date From must be a date." id="DateFrom" name="DateFrom" placeholder="" type="text" value="2020-10-14" />
<span class="field-validation-valid text-danger" data-valmsg-for="DateFrom" data-valmsg-replace="true"></span>
</div><label class="col-md-2 control-label" for="DateTo">Date To</label>
<div class='col-md-4 col-md-offset-0' id=''>
<input class="form-control datepicker" data-val="true" data-val-date="The field Date To must be a date." id="DateTo" name="DateTo" placeholder="" type="text" value="2020-10-14" />
<span class="field-validation-valid text-danger" data-valmsg-for="DateTo" data-valmsg-replace="true"></span>
</div></div>
<div class='col-md-12'>
<div class=' col-md-6'>
</div>
<button type='button' class='btn btn-success col-md-2 col-md-offset-1 ' id='ResetBtn' name='ResetBtn' onclick=''>
<span class='glyphicon glyphicon-refresh' aria-hidden='true'></span> Reset
</button>
<button type='submit' class='btn btn-primary col-md-2 col-md-offset-1 ' id='SearchBtn' name='SearchBtn' formaction='/123Mobile/Portal/Reports/SearchTransactionReport' onclick=''>
<span class='glyphicon glyphicon-search' aria-hidden='true'></span> Search
</button></div>
</div>
</div>
</div>
<div class='col-md-12'>
<div class='panel panel-grey equal-height-column' id=''>
<div class='panel-heading'>
<h3 class='panel-title'>
<i class='fa fa-tasks'></i>
Search results
</h3>
</div>
<div class='panel-body'>
<div style='' class='table-responsive'><table class='table table-hover table-striped'><thead><tr><th style='width:15%;'>File Name</th><th style='width:10%;'>Scheme</th><th style='width:15%;'>Proccessing Date</th><th style='width:8%;'></th><th style='width:8%;'></th></tr> </thead><tbody><tr><td style='-moz-word-break: break-all; -o-word-break: break-all; word-break: break-word;-ms-word-break: break-all;overflow-wrap: break-word;word-wrap: break-word; width: 15%;' class=''>Transaction-FIBWallet-20201014</td><td style='-moz-word-break: break-all; -o-word-break: break-all; word-break: break-word;-ms-word-break: break-all;overflow-wrap: break-word;word-wrap: break-word; width: 10%;' class=''>FIBWallet</td><td style='-moz-word-break: break-all; -o-word-break: break-all; word-break: break-word;-ms-word-break: break-all;overflow-wrap: break-word;word-wrap: break-word; width: 15%;' class=''>2020-10-14</td><td style='width: 8%;' class=''><a href="/123Mobile/Portal/Reports/DownloadTransactionReport/40072">Download as Text </a></td><td style='width: 8%;' class=''><a href="/123Mobile/Portal/Reports/DownloadTransactionReport/40073">Download as Excel </a></td></tr></tbody></table></div>
</div>
</div>
</div>
<div class='col-md-12'> <div class="pagination-container"><ul class="pagination"><li class="disabled PagedList-skipToFirst"><a>««</a></li><li class="disabled PagedList-skipToPrevious"><a rel="prev">«</a></li><li class="active"><a>1</a></li><li class="disabled PagedList-skipToNext"><a rel="next">»</a></li><li class="disabled PagedList-skipToLast"><a>»»</a></li></ul></div></div>
</form>
【问题讨论】：
-
那么，问题出在哪里？您正在使用 Jsoup 进行解析，发生了什么？您能否详细说明您的问题？
-
我添加了它们，再次查看代码和上面的描述
-
您的响应是 404。我不确定解析它是否有用。
-
"form[id='searchDisputesForm']"带撇号？
标签： java spring-boot web-scraping web-crawler jsoup