AmazonClientException：读取的数据长度与预期不同

Question

1.当我使用 AmazonS3Client 在 amazon s3 文件存储上上传文件时。 2.当我尝试一次上传多个文件时，它会给出异常：但同一个文件有多个线程。 我尝试了客户端配置，例如： 1.connectionTimeout=50000 毫秒 2.maxConnections=500 3.socketTimeout=50000 毫秒

Exception stacktrace:
com.amazonaws.AmazonClientException: Data read has a different length than the expected: dataLength=8192; expectedLength=79352; includeSkipped=false; in.getClass()=class com.amazonaws.internal.ResettableInputStream; markedSupported=true; marked=0; resetSinceLastMarked=false; markCount=1; resetCount=0
                    at com.amazonaws.util.LengthCheckInputStream.checkLength(LengthCheckInputStream.java:150)
                    at com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:110)
                    at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:73)
                    at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:151)
                    at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:73)
                    at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:98)
                    at com.amazonaws.http.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:153)
                    at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:98)
                    at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
                    at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:122)
                    at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:271)
                    at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:197)
                    at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:257)
                    at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:47)
                    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)

                at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:713)
                    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:518)
                    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
                    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
                    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:647)
                    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441)
                    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292)
                    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3655)
                    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1424)
                    at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:135)
                    at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:127)
                    at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:129)
                    at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:50)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:745)**

Answer 1

这个答案是 AWS Hanson 的人写的：

是否可能请求中指定的输入流已经被完全读取？

如果输入流是文件流，是否尝试过在请求中指定原始文件而不是文件的输入流？

Answer 2

用示例代码改进@iucasddaniel 的回答。

AmazonS3Client putObject：没有为流数据指定内容长度。 流内容将缓冲在内存中，并可能导致 内存错误。

解决方案 « 指定 Object Metadata content Length

File tempFile = "D://Test.mp4";
String bucketName = "YashFiles", filePath = "local/mp4/";

FileInputStream sampleStream = new FileInputStream( tempFile );
byte[] byteArray = IOUtils.toByteArray( sampleStream );
Long contentLength = Long.valueOf(byteArray.length);
sampleStream.close();

ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setContentLength(contentLength);

TransferManager tm = new TransferManager(credentials);

FileInputStream stream = new FileInputStream( tempFile );
PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName, filePath, stream,objectMetadata);
Upload myUpload = tm.upload(putObjectRequest);
    if (myUpload.isDone() == false) {
        System.out.println("Transfer: "+ myUpload.getDescription());
        System.out.println("  - State: "+ myUpload.getState());
        System.out.println("  - Progress: "+ myUpload.getProgress().getBytesTransferred());
    }
myUpload.waitForCompletion();

tm.shutdownNow();
stream.close();

org.apache.commons.io.FileUtils.forceDelete( tempFile );

---

Amazon S3：检查Key Exists and generating PresignedUrl

Answer 3

我在尝试执行 S3.putObject(MyObject); 时看到了该错误消息

我必须更新 objectMetadata.setContentLength( [你的内容的长度] );

例如：

String dataset= "Some value you want to add to S3 Bucket"; 
ObjectMetadata objectMetadata= new ObjectMetadata(); 
InputStream content= new ByteArrayInputStream(dataset.getBytes("UTF-8"));
objectMetadata.setContentLength(content.available()); 
objectMetadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYTION); 

Answer 4

...
byte[] f = IOUtils.toByteArray(inputStream); // This reads all bytes of the input stream
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(f.length);
metadata.setContentType(contentType); //Content type of the uploaded file
metadata.setHeader("filename", fileName);
s3.putObject(bucketName, key, new ByteArrayInputStream(f), metadata); // Here we create a new ByteArrayInputStream so S3 client get happy.