【发布时间】：2019-05-29 08:19:31
【问题描述】：
我正在评估 AWS 数据库服务以选择最有效的服务，目标是每 5 分钟将 S3 存储桶中的 json 文件中的数据加载到 Redshift。
我目前正在尝试使用 AWS Data Pipeline 来实现 ETL 的自动化。我一直在关注这个 AWS 教程“使用 AWS Data Pipeline 控制台将数据复制到 Amazon Redshift”，一切都简单明了。
https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-copydata-redshift-create.html
我在 Redshift 上设置了一个集群，在 S3 上设置了一个存储桶，创建了所需的所有角色和策略以及所需的所有权限。
现在在创建管道后，按下激活会出现警告说
错误/警告：
Object:Ec2Instance
WARNING: Could not validate S3 Access for role. Please ensure role ('DataPipelineDefaultRole') has s3:Get*, s3:List*, s3:Put* and sts:AssumeRole permissions for DataPipeline.
现在我确定我的角色和资源角色有s3:Get*、s3:List*、s3:Put* 和sts:AssumeRole
事实上，他们都有FullAccess 来提供我需要的所有服务。
DataPipelineDefaultRole 政策：
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "iam:CreateServiceLinkedRole",
"Resource": "*",
"Condition": {
"StringLike": {
"iam:AWSServiceName": [
"elasticmapreduce.amazonaws.com",
"spot.amazonaws.com"
]
}
}
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"ec2:AuthorizeSecurityGroupIngress",
"sdb:Select*",
"sqs:ReceiveMessage",
"s3:Get*",
"sqs:GetQueue*",
"s3:CreateBucket",
"sns:Unsubscribe",
"s3:List*",
"datapipeline:EvaluateExpression",
"ec2:StartInstances",
"dynamodb:DescribeTable",
"sqs:Delete*",
"iam:ListAttachedRolePolicies",
"ec2:RevokeSecurityGroupEgress",
"dynamodb:GetItem",
"sns:Subscribe",
"iam:ListRolePolicies",
"s3:DeleteObject",
"sdb:BatchPutAttributes",
"iam:GetRole",
"dynamodb:BatchGetItem",
"redshift:DescribeClusterSecurityGroups",
"ec2:CreateTags",
"ec2:DeleteNetworkInterface",
"ec2:RunInstances",
"dynamodb:Scan",
"rds:DescribeDBSecurityGroups",
"ec2:StopInstances",
"ec2:CreateNetworkInterface",
"ec2:CancelSpotInstanceRequests",
"cloudwatch:*",
"sqs:PurgeQueue",
"iam:GetRolePolicy",
"dynamodb:UpdateTable",
"ec2:RequestSpotInstances",
"ec2:DeleteTags",
"sns:ListTopics",
"ec2:ModifyImageAttribute",
"iam:PassRole",
"sns:Publish",
"ec2:DescribeNetworkInterfaces",
"ec2:CreateSecurityGroup",
"rds:DescribeDBInstances",
"ec2:ModifyInstanceAttribute",
"ec2:AuthorizeSecurityGroupEgress",
"ec2:DetachNetworkInterface",
"ec2:TerminateInstances",
"iam:GetInstanceProfile",
"sns:GetTopicAttributes",
"datapipeline:DescribeObjects",
"dynamodb:Query",
"iam:ListInstanceProfiles",
"ec2:Describe*",
"ec2:DeleteSecurityGroup",
"redshift:DescribeClusters",
"sqs:CreateQueue",
"elasticmapreduce:*",
"s3:Put*"
],
"Resource": "*"
},
{
"Sid": "VisualEditor2",
"Effect": "Allow",
"Action": [
"iam:PassRole",
"s3:Get*",
"s3:List*",
"s3:Put*",
"sts:AssumeRole"
],
"Resource": [
"arn:aws:iam::*:role/DataPipelineDefaultResourceRole",
"arn:aws:iam::*:role/DataPipelineDefaultRole",
"arn:aws:s3:::*/*"
]
},
{
"Sid": "VisualEditor3",
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*",
"s3:Put*"
],
"Resource": "arn:aws:s3:::*"
},
{
"Sid": "VisualEditor4",
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*",
"s3:Put*"
],
"Resource": "*"
}
]
}
DataPipelineDefaultResourceRole 政策：
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"cloudwatch:*",
"datapipeline:*",
"dynamodb:*",
"ec2:Describe*",
"elasticmapreduce:AddJobFlowSteps",
"elasticmapreduce:Describe*",
"elasticmapreduce:ListInstance*",
"rds:Describe*",
"redshift:DescribeClusters",
"redshift:DescribeClusterSecurityGroups",
"s3:*",
"sdb:*",
"sns:*",
"sqs:*"
],
"Resource": [
"*"
]
}
]
}
我已经调查了一个多星期的问题，尝试了所有现有的解决方案，更新了信任关系，重新创建了角色，保留了默认角色，让 Data Pipeline 创建新角色并检查了安全组，仍然有同样的问题。
激活管道并查看日志Uri后，我确实找到了2个文件夹Ec2Instance和RedshiftLoadActivity，在Redshift日志文件中只有2行，另一个有更多[INFO]描述下载jar和 TaskRunner 的 S3 文件。
在日志中有[INFO]和这些[WARN]：
Ec2Instance：
private.com.amazonaws.services.s3.internal.S3V4AuthErrorRetryStrategy: Attempting to re-send the request to mylogbucket.s3.eu-central-1.amazonaws.com with AWS V4 authentication. To avoid this warning in the future, please use region-specific endpoint to access buckets located in regions that require V4 signing.
RedshiftLoadActivity：
private.com.amazonaws.services.s3.internal.S3V4AuthErrorRetryStrategy: Attempting to re-send the request to mylogbucket.s3.eu-central-1.amazonaws.com with AWS V4 authentication. To avoid this warning in the future, please use region-specific endpoint to access buckets located in regions that require V4 signing.
问题应该是角色和策略，但我确保 Redshift 和 S3 存储桶不是问题，因为我尝试在查询编辑器上使用 COPY 命令并按预期加载数据。
我目前仍陷在该错误中，并希望获得一些有关如何解决此问题的建议。
548 [错误] (TaskRunnerService-resource:df-0539055_@Ec2Instance_2019-05-30T13:38:35-0) amazonaws.datapipeline.database.ConnectionFactory：无法建立 连接到 jdbc:postgresql://redshift-cluster-1.coykb9.eu-central-1.redshift.amazonaws.com:5439/db 连接被拒绝。检查主机名和端口是否正确 postmaster 正在接受 TCP/IP 连接
【问题讨论】：
-
请附上您的政策声明。
-
那里，我添加了两个策略。
-
您似乎缺少 EC2 的承担角色。
-
我必须在 DataPipelineDefaultRole 的操作中添加“sts:AssumeRole”，对吗？我这样做并再次尝试，但仍然收到相同的警告。
-
不，我不这么认为。如果我没记错的话，我认为您需要将它添加到 EC2 节点的实例配置文件中。问题是我对DataPipepline服务不太熟悉，但一般来说EC2实例配置文件有sts:AssumeRole。
标签： amazon-web-services amazon-s3 amazon-redshift amazon-iam amazon-data-pipeline