通过多线程同时上传多个文件到 S3 抛出 java.nio.channels.ClosedChannelException: null答案

【问题标题】：Upload multiple files simultaneously via multi threading to S3 throwing java.nio.channels.ClosedChannelException: null通过多线程同时上传多个文件到 S3 抛出 java.nio.channels.ClosedChannelException: null
【发布时间】：2018-10-28 19:45:59
【问题描述】：

我需要同时上传多个文件到 AWS S3 。我使用 ExecuterService 在不同的线程上上传文件。当我启动 100 个线程时，它开始抛出异常：

com.amazonaws.SdkClientException: Unable to execute HTTP request: null
|   at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1114) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1064) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325) ~[aws-java-sdk-s3-1.11.248.jar!/:na]
|   at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272) ~[aws-java-sdk-s3-1.11.248.jar!/:na]
|   at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1749) ~[aws-java-sdk-s3-1.11.248.jar!/:na]
|   ... 9 common frames omitted
| Caused by: java.nio.channels.ClosedChannelException: null
|   at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110) ~[na:1.8.0_171]
|   at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:147) ~[na:1.8.0_171]
|   at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65) ~[na:1.8.0_171]
|   at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[na:1.8.0_171]
|   at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.8.0_171]
|   at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   at com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:107) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   at com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.read(MD5DigestCalculatingInputStream.java:128) ~[aws-java-sdk-s3-1.11.248.jar!/:na]
|   at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[na:1.8.0_171]
|   at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[na:1.8.0_171]
|   at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[na:1.8.0_171]
|   at com.amazonaws.internal.SdkBufferedInputStream.read(SdkBufferedInputStream.java:76) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:140) ~[httpcore-4.4.9.jar!/:4.4.9]
|   at com.amazonaws.http.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:160) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) ~[httpcore-4.4.9.jar!/:4.4.9]
|   at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:160) ~[httpclient-4.5.4.jar!/:4.5.4]
|   at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) ~[httpcore-4.4.9.jar!/:4.4.9]
|   at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:63) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) ~[httpcore-4.4.9.jar!/:4.4.9]
|   at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.4.jar!/:4.5.4]
|   at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.4.jar!/:4.5.4]
|   at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.4.jar!/:4.5.4]
|   at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.4.jar!/:4.5.4]
|   at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.4.jar!/:4.5.4]
|   at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1236) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056) ~[aws-java-sdk-core-1.11.248.jar!/:na]
|   ... 20 common frames omitted

我使用的 Java AWS Sdk 版本是1.11.248。

ExecuterSERvice 用作：

ExecutorService ioThreadPool = Executors.newFixedThreadPool(100);
    ConcurrentLinkedDeque<Future<?>> uploadTasks = new ConcurrentLinkedDeque<>();
    Future<?> future = ioThreadPool.submit(() -> {
        try {
            putObjectToS3();
        } catch (IOException e) {
            throw new InternalErrorException("IOException during file upload.", e);
        }
    });
    uploadTasks.push(future);
    CompletableFuture.allOf(uploadTasks.toArray(new CompletableFuture<?>[0])).join();

我还将 AWS 客户端中的 maxNumberOfConnection 设置为 150

谢谢

【问题讨论】：

标签： java multithreading amazon-s3 file-upload

【解决方案1】：

看来你需要使用ClientConfiguration实现重试机制如下：

AmazonS3Client s3Client = AmazonS3ClientBuilder.standard().withRegion(REGION).
.withClientConfiguration(getClientConfiguration()).build();

private ClientConfiguration getClientConfiguration() {
        ClientConfiguration clientConfiguration = new ClientConfiguration();
        clientConfiguration.setRetryPolicy(new RetryPolicy(null, null, MAX_TRIES, false);
        return clientConfiguration;
}

您可以在特定于您的应用程序的 s3 客户端构建器中添加/更新上述配置。我已经使用基本配置向您展示了它的用途。

现在，尝试使用此配置运行您的程序。

【讨论】：