如何从 HTMLUnit 中的 Javascript 链接下载文件

Question

正如标题所说，我正在尝试从 javascript 链接下载带有 HTMLUnit 的文件。

我开始的页面是https://ppair.uspto.gov/TruePassWebStart/AuthenticationChooser.html。当我在浏览器中单击“使用 Java Web Start（新方法）进行身份验证”链接时，会下载一个 .jnlp 文件，然后运行该文件以打开一个要求提供身份验证信息的 Java 程序窗口。身份验证成功后，原始浏览器窗口会加载包含我将要抓取的信息的页面。

起始页的链接源代码sn-p为：

<tr>
<!-- onClick="return launchWebStart('authenticate');" -->
    <td><a href="javascript:void(0)" id="webstart-authenticate" ><font size="5">Authenticate with Java Web Start (new method)</font></a>
</tr>

用于此过程的 javascript 文件位于 https://ppair.uspto.gov/TruePassWebStart/js/WebStart.js。基本上，javascript 获取一个 cookie，对其进行编码，并将其附加到 URL 以请求 jnlp 文件。我想过只是模拟这个过程，但HTMLUnit 文件不鼓励这样做。 （他们说“通过单击元素并转移焦点来像用户一样操作页面要好得多”，所以我正在尝试这样做。）

我在 HTMLUnit 中遇到的问题是，在单击（）适当的锚点后，我无法接收预期的 jnlp 文件。我尝试了从本网站的其他问题中发现的几种不同的方法，包括：

HtmlUnit and JavaScript in links 和 HtmlUnit to invoke javascript from href to download a file

这是我使用的代码：

import java.io.IOException;
import java.io.InputStream;
import java.net.MalformedURLException;

import com.gargoylesoftware.htmlunit.*;
import com.gargoylesoftware.htmlunit.html.HtmlAnchor;
import com.gargoylesoftware.htmlunit.html.HtmlPage;

public class Test {

    public static void main(String[] args) throws FailingHttpStatusCodeException, MalformedURLException, IOException {
        WebClient webClient = new WebClient(BrowserVersion.FIREFOX_45);

        // open starting webpage
        HtmlPage page = webClient.getPage("https://ppair.uspto.gov/TruePassWebStart/AuthenticationChooser.html");

        // id of the element where the link is
        String linkID = "webstart-authenticate";

        // identify the appropriate anchor
        HtmlAnchor anchor = (HtmlAnchor) page.getElementById(linkID);

        // click the anchor
        Page p = anchor.click();

        // get the InputStream for the response; print it out
        InputStream is = p.getWebResponse().getContentAsStream();
        int b = 0;
        while ((b = is.read()) != -1) {
            System.out.print((char)b);
        }
        webClient.close();
    }
}

从上面的代码中打印出来的是起始网页的 html，而不是预期的 jnlp 文件。控制台还每 3 秒打印一次来自 javascript WebConsole 的状态更新（至少如果我让代码等待足够长的时间），所以我知道 javascript 发生了一些事情（函数 launchWebStart 和 followMediator 位于单独的 javascript 文件 WebStart .js):

Nov 21, 2016 2:53:25 PM com.gargoylesoftware.htmlunit.WebConsole info
INFO: launchWebStart

Nov 21, 2016 2:53:25 PM com.gargoylesoftware.htmlunit.WebConsole info
INFO: followMediator

Nov 21, 2016 2:53:25 PM com.gargoylesoftware.htmlunit.WebConsole info
INFO: responseReceived:200
WAIT

Nov 21, 2016 2:53:25 PM com.gargoylesoftware.htmlunit.WebConsole info
INFO: mediatorCallback: next wait

我还尝试使用 downloading files behind javascript button with htmlunit 中描述的 CollectingAttachmentHandler 对象：

import java.io.IOException;
import java.net.MalformedURLException;
import java.util.List;

import com.gargoylesoftware.htmlunit.*;
import com.gargoylesoftware.htmlunit.attachment.Attachment;
import com.gargoylesoftware.htmlunit.attachment.CollectingAttachmentHandler;
import com.gargoylesoftware.htmlunit.html.HtmlAnchor;
import com.gargoylesoftware.htmlunit.html.HtmlPage;

public class Test2 {

    public static void main(String[] args) throws FailingHttpStatusCodeException, MalformedURLException, IOException {
        WebClient webClient = new WebClient(BrowserVersion.FIREFOX_45);

        // open starting webpage
        HtmlPage page = webClient.getPage("https://ppair.uspto.gov/TruePassWebStart/AuthenticationChooser.html");

        // id of the element where the link is
        String linkID = "webstart-authenticate";

        // identify the appropriate anchor
        HtmlAnchor anchor = (HtmlAnchor) page.getElementById(linkID);

        CollectingAttachmentHandler attachmentHandler = new CollectingAttachmentHandler();
        webClient.setAttachmentHandler(attachmentHandler);
        attachmentHandler.handleAttachment(anchor.click());
        List<Attachment> attachments = attachmentHandler.getCollectedAttachments();

        int i = 0;
        while (i < attachments.size()) {
            Attachment attachment = attachments.get(i);
            Page attachedPage = attachment.getPage();
            WebResponse attachmentResponse = attachedPage.getWebResponse();
            String content = attachmentResponse.getContentAsString();
            System.out.println(content);
            i++;
        }
        webClient.close();
    }
}

此代码还打印出起始网页的内容。所以其他解决方案似乎都不适合我。我无法弄清楚我做错了什么。我已经没有关于如何让它工作的想法了（我认为这很容易！）非常感谢任何建议！

Answer 1

这是基于您的 Test2 的工作版本

    WebClient webClient = new WebClient(BrowserVersion.FIREFOX_45);

    // open starting webpage
    HtmlPage page = webClient.getPage("https://ppair.uspto.gov/TruePassWebStart/AuthenticationChooser.html");

    // id of the element where the link is
    String linkID = "webstart-authenticate";

    // identify the appropriate anchor
    HtmlAnchor anchor = (HtmlAnchor) page.getElementById(linkID);

    CountDownLatch latch = new CountDownLatch(1);
    webClient.setWebStartHandler(new WebStartHandler(){

        @Override
        public void handleJnlpResponse(WebResponse webResponse)
        {
            System.out.println("downloading...");
            try (FileOutputStream fos = new FileOutputStream("/Users/Franklyn/Downloads/uspto-auth.authenticate2.jnlp"))
            {
                IOUtils.copy(webResponse.getContentAsStream(),fos);
            } catch (IOException e)
            {
                throw new RuntimeException(e);
            }
            System.out.println("downloaded");
            latch.countDown();
        }
    });

    anchor.click();
    latch.await();//wait downloading to finish

    webClient.close();

那么为什么您的 Test2 不工作？因为响应的Content-Type对应的下载文件是application/x-java-jnlp-file，所以需要使用WebStartHandler。如果响应标头包含名为“Content-Disposition”的标头，并且其值以“attachment”开头，那么您的 Test2 可能会正常工作。