AmazonS3.getObject(request).getObjectContent() 频繁出现 NoHttpResponseException

Question

我有一个帮助程序尝试从 S3 进行线程下载。很多时候（大约 1% 的请求）我收到一条关于 NoHttpResponseException 的日志消息，一段时间后从 S3ObjectInputStream 读取时会导致 SocketTimeoutException。

我做错了什么，还是只是我的路由器/互联网？或者这是 S3 所期望的？我没有注意到其他地方的问题。

  public void
fastRead(final String key, Path path) throws StorageException 
    {
        final int pieceSize = 1<<20;
        final int threadCount = 8;

        try (FileChannel channel = (FileChannel) Files.newByteChannel( path, WRITE, CREATE, TRUNCATE_EXISTING ))
        {
            final long size = s3.getObjectMetadata(bucket, key).getContentLength();
            final long pieceCount = (size - 1) / pieceSize + 1;

            ThreadPool pool = new ThreadPool (threadCount);
            final AtomicInteger progress = new AtomicInteger();

            for(int i = 0; i < size; i += pieceSize)
            {
                final int start = i;
                final long end = Math.min(i + pieceSize, size);

                pool.submit(() ->
                {
                    boolean retry;
                    do
                    {
                        retry = false;
                        try
                        {
                            GetObjectRequest request = new GetObjectRequest(bucket, key);
                            request.setRange(start, end - 1);
                            S3Object piece = s3.getObject(request);
                            ByteBuffer buffer = ByteBuffer.allocate ((int)(end - start));
                            try(InputStream stream = piece.getObjectContent())
                            {
                                IOUtils.readFully( stream, buffer.array() );
                            }
                            channel.write( buffer, start );
                            double percent = (double) progress.incrementAndGet() / pieceCount * 100.0;
                            System.err.printf("%.1f%%\n", percent);
                        }
                        catch(java.net.SocketTimeoutException | java.net.SocketException e)
                        {
                            System.err.println("Read timed out. Retrying...");
                            retry = true;
                        }
                    }
                    while (retry);

                });
            }

            pool.<IOException>await();
        }
        catch(AmazonClientException | IOException | InterruptedException e)
        {
            throw new StorageException (e);
        }
    }

2014-05-28 08:49:58 INFO com.amazonaws.http.AmazonHttpClient executeHelper Unable to execute HTTP request: The target server failed to respond
org.apache.http.NoHttpResponseException: The target server failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:95)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:66)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:713)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:518)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:385)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:233)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3569)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1130)
at com.syncwords.files.S3Storage.lambda$fastRead$0(S3Storage.java:123)
at com.syncwords.files.S3Storage$$Lambda$3/1397088232.run(Unknown Source)
at net.almson.util.ThreadPool.lambda$submit$8(ThreadPool.java:61)
at net.almson.util.ThreadPool$$Lambda$4/1980698753.call(Unknown Source)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:744)

Answer 1

更新：AWS SDK 已针对我在 GitHub 上创建的问题进行了更新。我不确定情况如何变化。这个答案的第二部分（批评getObject）可能（希望？）是错误的。

---

S3 旨在失败，而且经常失败。

幸运的是，AWS SDK for Java 具有用于重试请求的内置功能。不幸的是，它们没有涵盖下载 S3 对象时发生 SocketExceptions 的情况（它们确实在上传和执行其他操作时起作用）。因此，需要与问题中类似的代码（见下文）。

当该机制按预期工作时，您仍会在日志中看到消息。您可以选择通过过滤来自com.amazonaws.http.AmazonHttpClient 的INFO 日志事件来隐藏它们。 （AWS SDK 使用 Apache Commons Logging。）

根据您的网络连接和亚马逊服务器的健康状况，重试机制可能会失败。 lvlv指出，相关参数的配置方式是通过ClientConfiguration。我建议更改的参数是重试次数，默认为3。您可以尝试的其他事情是增加或减少连接和套接字超时（默认 50 秒，这不仅足够长，而且考虑到无论如何您都会经常超时，它可能太长了）和使用 TCP KeepAlive（默认关闭）。

ClientConfiguration cc = new ClientConfiguration()
    .withMaxErrorRetry (10)
    .withConnectionTimeout (10_000)
    .withSocketTimeout (10_000)
    .withTcpKeepAlive (true);
AmazonS3 s3Client = new AmazonS3Client (credentials, cc);

重试机制甚至可以通过设置RetryPolicy（同样在ClientConfiguration 中）来覆盖。它最有趣的元素是RetryCondition，默认情况下：

按以下顺序检查各种条件：

• 重试由 IOException 引起的 AmazonClientException 异常；
• 重试 AmazonServiceException 异常是 500 内部服务器 错误、503 服务不可用错误、服务限制错误或 时钟偏差错误。

参见 SDKDefaultRetryCondition javadoc 和 source

 

隐藏在 SDK 其他地方的半途而废的重试设施

内置机制（在整个 AWS 开发工具包中使用）不处理的是读取 S3 对象数据。

如果您调用AmazonS3.getObject (GetObjectRequest getObjectRequest, File destinationFile)，AmazonS3Client 会使用自己的重试机制。该机制在ServiceUtils.retryableDownloadS3ObjectToFile (source) 内部，它使用次优的硬连线重试行为（它只会重试一次，并且永远不会出现 SocketException！）。 ServiceUtils 中的所有代码似乎设计不佳 (issue)。

我使用的代码类似于：

  public void
read(String key, Path path) throws StorageException
    {
        GetObjectRequest request = new GetObjectRequest (bucket, key);

        for (int retries = 5; retries > 0; retries--) 
        try (S3Object s3Object = s3.getObject (request))
        {
            if (s3Object == null)
                return; // occurs if we set GetObjectRequest constraints that aren't satisfied

            try (OutputStream outputStream = Files.newOutputStream (path, WRITE, CREATE, TRUNCATE_EXISTING))
            {
                byte[] buffer = new byte [16_384];
                int bytesRead;
                while ((bytesRead = s3Object.getObjectContent().read (buffer)) > -1) {
                    outputStream.write (buffer, 0, bytesRead);
                }
            }
            catch (SocketException | SocketTimeoutException e)
            {
                // We retry exceptions that happen during the actual download
                // Errors that happen earlier are retried by AmazonHttpClient
                try { Thread.sleep (1000); } catch (InterruptedException i) { throw new StorageException (i); }
                log.log (Level.INFO, "Retrying...", e);
                continue;
            }
            catch (IOException e)
            {
                // There must have been a filesystem problem
                // We call `abort` to save bandwidth
                s3Object.getObjectContent().abort();
                throw new StorageException (e);
            }

            return; // Success
        }
        catch (AmazonClientException | IOException e)
        {
            // Either we couldn't connect to S3
            // or AmazonHttpClient ran out of retries
            // or s3Object.close() threw an exception
            throw new StorageException (e);
        }

        throw new StorageException ("Ran out of retries.");
    }

Answer 2

我以前也遇到过类似的问题。根据official example from AWS S3，我发现每次完成一个 S3Object 后，都需要 close() 将其释放回池中：

AmazonS3 s3Client = new AmazonS3Client(new ProfileCredentialsProvider());        
S3Object object = s3Client.getObject(
              new GetObjectRequest(bucketName, key));
InputStream objectData = object.getObjectContent();
// Process the objectData stream.
objectData.close();

感谢您添加链接。顺便说一句，我想增加ClientConfiguration 的最大连接、重试和超时（默认最大连接为50）也可能有助于解决问题，如下所示：

AmazonS3 s3Client = new AmazonS3Cient(aws_credential, 
                       new ClientConfiguration().withMaxConnections(100)
                                      .withConnectionTimeout(120 * 1000)
                                      .withMaxErrorRetry(15))