【发布时间】：2020-02-21 06:45:46
【问题描述】：
当使用自定义 IAM 角色作为 ECS 任务定义的自定义执行角色时，由于无法初始化 CloudWatch 日志记录驱动程序，我们生成的服务将无法在我们的 ECS 实例上启动。具体来说，我们在 CloudWatch 中看到来自 ECS 代理的以下错误：
2019-10-24T21:43:10Z [INFO] TaskHandler: Adding event: TaskChange: [arn:aws:ecs:us-west-1:REDACTED -> STOPPED, Known Sent: NONE, PullStartedAt: 2019-10-24 21:43:08.499577397 +0000 UTC m=+187.475751716, PullStoppedAt: 2019-10-24 21:43:09.69279918 +0000 UTC m=+188.668973506, ExecutionStoppedAt: 2019-10-24 21:43:10.153954812 +0000 UTC m=+189.130129126, arn:aws:ecs:us-west-1:REDACTED wordpress -> STOPPED, Reason CannotStartContainerError: Error response from daemon: failed to initialize logging driver: CredentialsEndpointError: failed to load credentials
caused by: Get http://169.254.170.2/v2/credentials/REDACTED: dial tcp 169.254.170.2:80: connect: connection refused, Known Sent: NONE] sent: false
这个“连接被拒绝错误”曾经是一个超时错误，但我在阅读了类似的问题后尝试通过添加来自https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-install.html 的 iptables 条目来调试这个问题，即使这是一个 Amazon ECS 预置的 CoreOS EC2 实例（不是自定义一）。
基本上该链接和other issues similar to mine 推荐了以下内容，至少将错误更改为超时错误：
ubuntu:~$ sudo iptables -t nat -A PREROUTING -p tcp -d 169.254.170.2 --dport 80 -j DNAT --to-destination 127.0.0.1:51679
ubuntu:~$ sudo iptables -t nat -A OUTPUT -d 169.254.170.2 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 51679
请注意，当我们不在容器定义中使用自定义 IAM 执行角色时，此容器定义在正常情况下运行和工作完全正常；但是因为我试图在任务定义中添加一个 AWS SecretsManager 密钥；这需要我们定义一个可以访问密钥的自定义角色。
编辑：这是 ECS 实例的角色策略 JSON 和 cloud-config.yml：
JSON 策略角色：
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:Describe*",
"elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
"elasticloadbalancing:DeregisterTargets",
"elasticloadbalancing:Describe*",
"elasticloadbalancing:RegisterInstancesWithLoadBalancer",
"elasticloadbalancing:RegisterTargets"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ssm:GetParameters",
"secretsmanager:GetSecretValue",
"kms:Decrypt"
],
"Resource": [
"${var.aws_mysql_secret_arn}"
]
}
]
}
cloud-config.yml
coreos:
units:
- name: update-engine.service
command: stop
- name: amazon-ecs-agent.service
command: start
runtime: true
content: |
[Unit]
Description=AWS ECS Agent
Documentation=https://docs.aws.amazon.com/AmazonECS/latest/developerguide/
Requires=docker.socket
After=docker.socket
[Service]
Environment=ECS_CLUSTER=${ecs_cluster_name}
Environment=ECS_LOGLEVEL=${ecs_log_level}
Environment=ECS_VERSION=${ecs_agent_version}
Restart=on-failure
RestartSec=30
RestartPreventExitStatus=5
SyslogIdentifier=ecs-agent
ExecStartPre=-/bin/mkdir -p /var/log/ecs /var/ecs-data /etc/ecs
ExecStartPre=-/usr/bin/docker kill ecs-agent
ExecStartPre=-/usr/bin/docker rm ecs-agent
ExecStartPre=iptables -t nat -A PREROUTING -p tcp -d 169.254.170.2 --dport 80 -j DNAT --to-destination 127.0.0.1:51679
ExecStartPre=iptables -t nat -A OUTPUT -d 169.254.170.2 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 51679
ExecStartPre=/usr/bin/docker pull amazon/amazon-ecs-agent:$${ECS_VERSION}
ExecStart=/usr/bin/docker run --name ecs-agent \
--volume=/var/run/docker.sock:/var/run/docker.sock \
--volume=/var/log/ecs:/log \
--volume=/var/ecs-data:/data \
--volume=/sys/fs/cgroup:/sys/fs/cgroup:ro \
--volume=/run/docker/execdriver/native:/var/lib/docker/execdriver/native:ro \
--publish=127.0.0.1:51678:51678 \
--env=ECS_LOGFILE=/log/ecs-agent.log \
--env=ECS_LOGLEVEL=$${ECS_LOGLEVEL} \
--env=ECS_DATADIR=/data \
--env=ECS_CLUSTER=$${ECS_CLUSTER} \
--env=ECS_AVAILABLE_LOGGING_DRIVERS='["awslogs"]' \
--env=ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE=true \
--log-driver=awslogs \
--log-opt awslogs-region=${aws_region} \
--log-opt awslogs-group=${ecs_log_group_name} \
amazon/amazon-ecs-agent:$${ECS_VERSION}
【问题讨论】：
-
您的角色是否扩展了默认角色？
-
@Lamanus 我不确定如何扩展默认角色，因为在没有指定执行角色的情况下运行任务定义或 ECS 服务时看不到它的名称。在未明确指定时，AWS 似乎将其隐藏。
标签： amazon-web-services timeout amazon-iam amazon-ecs