从使用 doPost 请求 Java 的网站下载文件

Question

我是 Java 新手。我正在尝试编写一个从这些网站https://servizissiir.regione.emilia-romagna.it/FlussiMTS/ 和https://www.ilmeteo.it/portale/archivio-meteo 下载文件的代码。 在这两个网站中，在开始下载之前，用户必须设置参数。

对于第二个，我成功地使用了这段代码，因为我可以在 url 中插入参数。

public class Downloader {

    public static void main(String[] args) {
        String remoteFile = "https://www.ilmeteo.it/portale/archivio-meteo/Bologna/2021/Gennaio?format=csv";
        String localFile = "src/Bologna.csv";

        try {
            FileUtils.copyURLToFile(new URL(remoteFile), new File(localFile), 20000, 20000);
        } catch (IOException ex) {
            System.out.println(ex.getMessage());
        }

    }
}

但是第一个我不能使用网址。 我开始下载时看到的网址是https://servizissiir.regione.emilia-romagna.it/FlussiMTS/MobiliterServlet 所以我认为参数被发送到 doPost servlet。

如果我把这个网址放在前面的代码中，我会在我的本地文件夹中获得一个损坏的 rar 存档。

有人知道如何解决这个问题吗？

Answer 1

我找到了这个解决方案

`

 public static void PostRequest(String url, int i) throws IOException, InterruptedException {
    RequestConfig rc = RequestConfig.custom().setCookieSpec(CookieSpecs.STANDARD).build();
    CloseableHttpClient client = HttpClients.custom().setDefaultRequestConfig(rc).build();
    HttpPost httpPost;
    ArrayList<NameValuePair> postParameters;
    httpPost = new HttpPost(url);
    postParameters = new ArrayList<NameValuePair>();
    postParameters.add(new BasicNameValuePair("forward", "esportaReportCSV"));
    postParameters.add(new BasicNameValuePair("yearPromptName", "AnnoMese:"));
    postParameters.add(new BasicNameValuePair("esportaReportCSV", ""));
    postParameters.add(new BasicNameValuePair("tipoIntervallo", "M"));
    postParameters.add(new BasicNameValuePair("AnnoMese:", "202108"));
    postParameters.add(new BasicNameValuePair("prompt", "Nessuno"));
    httpPost.setEntity(new UrlEncodedFormEntity(postParameters, "UTF-8"));
    CloseableHttpResponse response = client.execute(httpPost);    
    HttpEntity entity = response.getEntity();
    if (entity != null) {
        if (response.getEntity().getContentType().getValue().trim().equals("application/zip")) {
            String name = response.getFirstHeader("Content-Disposition").getValue();
            String fileName = name.replaceFirst("(?i)^.*filename=\"([^\"]+)\".*$", "$1");
            FileOutputStream fos = new FileOutputStream("path/" + fileName);
            entity.writeTo(fos);
            fos.close();
            System.out.println("Download success!");
        } else if (!response.getEntity().getContentType().getValue().trim().equals("application/zip") && i < 10) {
            i++;
            PostRequest(url, i);
        } else if (!response.getEntity().getContentType().getValue().trim().equals("application/zip") && i == 10) {
            System.out.println("Download failed!");
        }
    }
    client.close();
}

public static void GetRequest(String url, int i) throws IOException, InterruptedException {
    RequestConfig rc = RequestConfig.custom().setCookieSpec(CookieSpecs.STANDARD).build();
    CloseableHttpClient client = HttpClients.custom().setDefaultRequestConfig(rc).build();
    HttpGet httpGet = new HttpGet(url);
    httpGet.setHeader("User-Agent", "User-Agent");
    CloseableHttpResponse response = client.execute(httpGet);
    HttpEntity entity = response.getEntity();
    if (entity != null) {
        if (response.getEntity().getContentType().getValue().trim().equals("text/csv")) {
            String name = response.getFirstHeader("Content-Disposition").getValue();
            String fileName = name.replaceFirst("(?i)^.*filename=\"([^\"]+)\".*$", "$1");
            FileOutputStream fos = new FileOutputStream("path/" + fileName);
            entity.writeTo(fos);
            fos.close();
            System.out.println("Download success!");
        } else if (response.getEntity().getContentType().getValue().trim().equals("text/html") && i < 10) {
            i++;
            GetRequest(url, i);
        } else if (response.getEntity().getContentType().getValue().trim().equals("text/html") && i == 10) {
            System.out.println("Download failed!");
        }
    }
    client.close();
}

public static void main(String[] args) throws IOException, InterruptedException {
    PostRequest("https://servizissiir.regione.emilia-romagna.it/FlussiMTS/MobiliterServlet" , 1);
    GetRequest("https://www.ilmeteo.it/portale/archivio-meteo/Bondeno/2020/Agosto?format=csv", 1);
    }

`

我还写了 HTTP Get 解决方案

要运行代码，需要以下 jar 文件： httpclient-4.5.6.jar httpcore-4.4.10.jar commons-codec-1.10.jar commons-logging-1.2.jar

参数和http方法可以通过浏览器的开发者工具找到