使用 c# 将数据传输到 aws s3 雪球

Question

我正在尝试使用雪球将所有数据导入 s3，但我需要跟踪每个文件的一些元数据。我有一些 c# 代码可以使用此元数据将文件直接上传到存储桶，这工作正常。我以为我可以使用 snowball s3 适配器连接到我的 snowball 并从我的 c# 代码上传数据，因为this link 说：

您可以将 S3 适配器与现有的 Amazon S3 接口一起使用如 AWS 开发工具包、AWS CLI 或您自己的自定义 Amazon S3 REST 客户。

我的适配器在http://localhost:8090 上运行，所以我尝试通过传入一个 AmazonS3Config 对象并将其设置为 Amazon3Client 的端点作为 ServiceUrl，但是当我尝试使用 PutObjectRequest 或TransferUtility，我得到以下异常：

 Amazon.Runtime.AmazonServiceException: A WebException with status
 NameResolutionFailure was thrown. ---> System.Net.WebException: The
 remote name could not be resolved: 'BUCKET.localhost' at
 System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
 at System.Net.HttpWebRequest.GetRequestStream() at
 Amazon.Runtime.Internal.HttpRequest.GetRequestContent() at
 Amazon.Runtime.Internal.HttpHandler`1.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.RedirectHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.Unmarshaller.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext
 executionContext)
 --- End of inner exception stack trace --- at Amazon.Runtime.Internal.WebExceptionHandler.HandleException(IExecutionContext
 executionContext, WebException exception) at
 Amazon.Runtime.Internal.ExceptionHandler`1.Handle(IExecutionContext
 executionContext, Exception exception) at
 Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext
 executionContext, Exception exception) at
 Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.Signer.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.CredentialsRetriever.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.RetryHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.S3.Internal.AmazonS3KmsHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.EndpointResolver.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.S3.Internal.AmazonS3PostMarshallHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.Marshaller.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.S3.Internal.AmazonS3PreMarshallHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.MetricsHandler.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.Internal.RuntimePipeline.InvokeSync(IExecutionContext
 executionContext) at
 Amazon.Runtime.AmazonServiceClient.Invoke(TRequest request,
 IMarshaller`2 marshaller, ResponseUnmarshaller unmarshaller) at
 Amazon.S3.AmazonS3Client.PutObject(PutObjectRequest request)

我尝试使用我的本地 IP 地址以及 127.0.0.1，但都得到了相同的错误。当我使用亚马逊 cli 将文档上传到雪球时，使用--endpoint 更改端点，它工作（使用本地主机或我的本地 ip），所以显然它只是关于 .net sdk 的东西。但我真的更喜欢使用我现有的 c# 代码，而不是编写一个命令行程序来执行此操作 - 从 cli 中它不会那么简单。

编辑：这是不起作用的 c# 代码：

var config = new AmazonS3Config
{
  ServiceURL = "http://localhost:8090"
};

var file = new FileInfo(FILENAME);

var pRequest = new PutObjectRequest
{
  BucketName = BUCKET,
  Key = S3FILENAME,
  StorageClass = S3StorageClass.StandardInfrequentAccess,
  FilePath = file.FullName
};

pRequest.Metadata.Add("x-amz-meta-local-date-modified", file.CreationTime.ToFileTimeUtc().ToString());

using (var client = new AmazonS3Client(config))
{
  client.PutObject(pRequest);
}

这是有效的命令行语句：

aws s3 cp FILENAME s3://BUCKET/S3FILENAME --endpoint http://localhost:8090

谁能帮帮我？

谢谢

Answer 1

问题是，AWS 开发工具包默认将您的 BucketName 附加到 ServiceURL，因此您最终会得到一个无效的 URL：http://BUCKET.localhost:8090。

您需要使用 ForcePathStyle 参数，它可以防止 BucketName 被添加到 ServiceURL：

var config = new AmazonS3Config
{
    ServiceURL = "http://localhost:8090",
    ForcePathStyle = true
};