如何将大文件下载到字节数组中

Question

我正在尝试使用 spring Rest Template 下载大文件：

    private Result download(Long id) throws Exception {
    Result result = new Result();
    try {
        RestTemplate restTemplate = restTemplateFactory.getRestTemplate();
        HttpHeaders headers = new HttpHeaders();
        headers.setContentType(MediaType.APPLICATION_OCTET_STREAM);
        headers.set(AUTHORIZATION, authorizationBL.getAccessToken().getAccessToken());
        HttpEntity<MultiValueMap<String, String>> request = new HttpEntity<MultiValueMap<String, String>>(null, headers);
        Map<String, Long> valuesMap = new HashMap<String, Long>(1);
        valuesMap.put(ID, id);
        ResponseEntity<byte[]> response = restTemplate.exchange(downloadURL, HttpMethod.GET, request, byte[].class);
        byte[] stream = response.getBody();
        result.setData(stream);
        result.setStatus(Result.Status.SUCCESS);
    } catch (Exception e) {
        throw e;
    }
    return result;
}

当文件大小很大时，可以说超过 100 MB，那么我遇到了问题：

java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236) ~[na:1.8.0_60]
at java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:191) ~[na:1.8.0_60]
at org.springframework.http.converter.ByteArrayHttpMessageConverter.readInternal(ByteArrayHttpMessageConverter.java:59) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.http.converter.ByteArrayHttpMessageConverter.readInternal(ByteArrayHttpMessageConverter.java:38) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.http.converter.AbstractHttpMessageConverter.read(AbstractHttpMessageConverter.java:193) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:105) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:924) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:908) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:662) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:620) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:538) ~[spring-web-4.3.13.RELEASE.jar:4.3.13.RELEASE]

我将堆大小增加到 10g，但也没有帮助。

Answer 1

不要将文件下载到字节数组中，而是将其保存到磁盘。然后，您可以从程序中需要的任何其他位置读取它。例如，将您的方法更改为

/**
 * returns a File corresponding to the tmp file download for this resource
 */
private File download(Long id) throws IOException {
    File tmp = null;
    try {
        RestTemplate restTemplate = restTemplateFactory.getRestTemplate();
        HttpHeaders headers = new HttpHeaders();
        headers.setContentType(MediaType.APPLICATION_OCTET_STREAM);
        headers.set(AUTHORIZATION, authorizationBL.getAccessToken().getAccessToken());
        HttpEntity<MultiValueMap<String, String>> request = new HttpEntity<MultiValueMap<String, String>>(null, headers);
        ResponseEntity<byte[]> response = restTemplate.exchange(downloadURL, HttpMethod.GET, request, byte[].class);

        if (response.getStatusCode() == HttpStatus.OK) {
            tmp = Files.createTempFile("yourprefix", "yoursuffix");
            Files.write(tmp.toPath(), response.getBody());
        } else {
            // TODO handle this case
        }
    } catch (Exception e) {
        throw new IOException("An exception occurred while downloading", e);
    }
    return tmp;
}

一些观察：

1. valuesMap 未在任何地方使用（也已删除）
2. result 也被删除了

Answer 2

我能够使用流解决此问题，而不是将文件存储在 byte[] 中并使用IOUtils.copyLarge(inputStream, outputStream)。