通过代理 java 下载文件

Question

我通过代理从www.example.com/example.pdf 之类的网址下载文件并将其保存在java 中的文件系统上时遇到问题。有人对这如何工作有想法吗？如果我得到 InputStream，我可以简单地将其保存到文件系统：

final ReadableByteChannel rbc = Channels.newChannel(httpUrlConnetion.getInputStream());    
final FileOutputStream fos = new FileOutputStream(file);
fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
fos.close();

但是如何通过代理获取 url 的输入流？ 如果我这样做：

SocketAddress addr = new InetSocketAddress("my.proxy.com", 8080);
Proxy proxy = new Proxy(Proxy.Type.HTTP, addr);
URL url = new URL("http://my.real.url.com/");
URLConnection conn = url.openConnection(proxy);

我遇到了这个异常：

java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(Unknown Source)
    at java.net.SocketInputStream.read(Unknown Source)
    at java.io.BufferedInputStream.fill(Unknown Source)
    at java.io.BufferedInputStream.read1(Unknown Source)
    at java.io.BufferedInputStream.read(Unknown Source)
    at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
    at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
    at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
    at app.model.mail.crawler.newimpl.FileLoader.getSourceOfSiteViaProxy(FileLoader.java:167)
    at app.model.mail.crawler.newimpl.FileLoader.process(FileLoader.java:220)
    at app.model.mail.crawler.newimpl.FileLoader.run(FileLoader.java:57)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

使用这个：

final HttpURLConnection httpUrlConnetion = (HttpURLConnection) website.openConnection(proxy);
httpUrlConnetion.setDoOutput(true);
httpUrlConnetion.setDoInput(true);
httpUrlConnetion.setRequestProperty("Content-type", "text/xml");
httpUrlConnetion.setRequestProperty("Accept", "text/xml, application/xml");
httpUrlConnetion.setRequestMethod("POST");
httpUrlConnetion.connect();

我可以下载一个网站的源代码，它是 html，但不是文件，也许有人可以帮助我设置下载文件的属性。

Answer 1

以下内容与其他答案不同，对我有用：在连接之前设置这些属性：

            System.getProperties().put("http.proxySet", "true");
            System.getProperties().put("http.proxyHost", "my.proxy.com");
            System.getProperties().put("http.proxyPort", "8080"); //port is String, not int

然后，打开 URLConnection 并尝试下载文件。

Answer 2

可以使用 Apache httpclient 库来解决代理的大部分问题。编译下面的代码，可以使用下面的maven：

马文：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>stackoverflow.test</groupId>
  <artifactId>proxyhttp</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>proxy</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.httpcomponents</groupId>
      <artifactId>httpclient</artifactId>
      <version>4.5.1</version>
    </dependency>
  </dependencies>
</project>

Java 代码：

import org.apache.http.HttpHost;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;

/**
 * How to send a request via proxy.
 *
 * @since 4.0
 */
public class ClientExecuteProxy {

    public static void main(String[] args)throws Exception {
        CloseableHttpClient httpclient = HttpClients.createDefault();
        try {
            HttpHost target = new HttpHost("www.google.com", 80, "http");
            HttpHost proxy = new HttpHost("127.0.0.1", 8889, "http");

            RequestConfig config = RequestConfig.custom()
                    .setProxy(proxy)
                    .build();
            HttpGet request = new HttpGet("/");
            request.setConfig(config);

            System.out.println("Executing request " + request.getRequestLine() + " to " + target + " via " + proxy);

            CloseableHttpResponse response = httpclient.execute(target, request);
            try {
                System.out.println("----------------------------------------");
                System.out.println(response.getStatusLine());
                System.out.println(EntityUtils.toString(response.getEntity()));
            } finally {
                response.close();
            }
        } finally {
            httpclient.close();
        }
    }

}

Answer 3

以编程方式设置代理：

SocketAddress addr = new InetSocketAddress("my.proxy.com", 8080);
Proxy proxy = new Proxy(Proxy.Type.HTTP, addr);
URL url = new URL("http://my.real.url.com/");
URLConnection conn = url.openConnection(proxy);

然后您可以将上面的代码与最后一行返回的URLConnection 一起使用。如果您愿意，也可以使用 SOCKS 代理，或强制不使用代理。

这是从this Oracle documentation 拍摄的（并略有编辑）。

Answer 4

另一种方法是在 httpUrlConnection 的每个实例“内部”实现代理。那就是：

1. 不要连接到您想要的真实 URL。首先，连接到代理 IP 和端口，但使用 http GET 方法引用您想要的 URL。
2. 使用 setRequestProperty 将 host 设置为您的 URL 和您可能需要的任何其他标头。

如果有效，连接会透明地将文件发送给您。

我有一些代码可以使用 Sockets。

try {
    Socket sock = new Socket("10.0.241.1", 3128); //proxy IP and port
    InputStream is = sock.getInputStream();
    OutputStream os = sock.getOutputStream();
    String str = "GET http://www.uol.com.br HTTP/1.1\r\n"; //GET your site
    str += "Host: www.uol.com.br\r\n"; //again, Host of your site
    str += "Proxy-Authorization: Basic ZWR1YXJkby5wb2NvOmM1NmQyMw==\r\n"; //if password is needed
    str += "\r\n";
    os.write(str.getBytes());
    byte[] bb = new byte[1024];
    int L = 0;
    while ((L = is.read(bb)) != -1) {
        //write bytes to file stream...
    }
} catch (Exception ex) {
    //exception handling...
}

“当一个人可以使用 httpUrlConnection 时，为什么有人会使用纯套接字？”，你说。嗯，那个时候，我还不知道httpUrlConnection。