【问题标题】:EC2 instances in private subnets cannot access amazon repository私有子网中的 EC2 实例无法访问亚马逊存储库
【发布时间】:2019-01-26 10:53:38
【问题描述】:

我正在尝试创建 ECS 集群,并且我手动构建了具有 3 个公共子网和 3 个私有子网的 VPC。所有 3 个公有子网都通过 0.0.0.0/0 附加了 IGW,所有 3 个私有子网都在路由表中附加了 NAT 网关,地址为 0.0.0.0/0。 3 个 NAT 网关中的每一个都分别位于每个公共子网中。

我已经使用我现在尝试使用的相同 CloudFormation 模板创建了另一个 ECS 集群,并且一切正常。

我比较了第一个和第二个 VPC 之间的设置(失败一个),所有设置(IGW、NAT 网关、路由表、NACL、SG)都相同,当然 IP 被调整为第二个 VPC 的 IP。当我尝试在第二个 VPC(失败一个)中创建 ECS 时,私有子网中的 EC2 实例无法连接到 Amazon 存储库,随后整个堆栈被回滚,因为来自 EC2 实例的信号从未发送到 Auto Scaling Group。

之后我检查了 EC2 实例的系统日志,但它们无法安装亚马逊代理。以下是日志摘录:

Starting cloud-init: Cloud-init v. 0.7.6 running 'modules:config' at Mon, 20 Aug 2018 06:38:04 +0000. Up 10.06 seconds.
Loaded plugins: priorities, update-motd, upgrade-helper


 One of the configured repositories failed (Unknown),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Disable the repository, so yum won't use it by default. Yum will then
        just ignore the repository until you permanently enable it again or use
        --enablerepo for temporary usage:

            yum-config-manager --disable <repoid>

     4. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true

Cannot find a valid baseurl for repo: amzn-main/latest
Could not retrieve mirrorlist http://repo.eu-central-1.amazonaws.com/latest/main/mirror.list error was
12: Timeout on http://repo.eu-central-1.amazonaws.com/latest/main/mirror.list: (28, 'Connection timed out after 5001 milliseconds')
Aug 20 06:38:20 cloud-init[2116]: util.py[WARNING]: Package upgrade failed
Aug 20 06:38:20 cloud-init[2116]: cc_package_update_upgrade_install.py[WARNING]: 1 failed with exceptions, re-raising the last one
Aug 20 06:38:20 cloud-init[2116]: util.py[WARNING]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_package_update_upgrade_install.pyc'>) failed
Generating SSH2 ED25519 host key: [  OK  ]

Starting sshd: [  OK  ]

ntpdate: Synchronizing with time server: [  OK  ]

Starting ntpd: [  OK  ]

Starting sendmail: [  OK  ]

Starting sm-client: [  OK  ]

Starting crond: [  OK  ]

Starting cgconfig service: [  OK  ]

Starting docker:    .[  OK  ]

Starting cloud-init: Cloud-init v. 0.7.6 running 'modules:final' at Mon, 20 Aug 2018 06:38:25 +0000. Up 29.91 seconds.
Loaded plugins: priorities, update-motd, upgrade-helper
Examining /var/tmp/yum-root-i85tqq/amazon-ssm-agent.rpm: amazon-ssm-agent-2.3.13.0-1.x86_64
Marking /var/tmp/yum-root-i85tqq/amazon-ssm-agent.rpm to be installed
Resolving Dependencies


 One of the configured repositories failed (Unknown),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Disable the repository, so yum won't use it by default. Yum will then
        just ignore the repository until you permanently enable it again or use
        --enablerepo for temporary usage:

            yum-config-manager --disable <repoid>

     4. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true

Cannot find a valid baseurl for repo: amzn-main/latest
Could not retrieve mirrorlist http://repo.eu-central-1.amazonaws.com/latest/main/mirror.list error was
12: Timeout on http://repo.eu-central-1.amazonaws.com/latest/main/mirror.list: (28, 'Connection timed out after 5000 milliseconds')
Loaded plugins: priorities, update-motd, upgrade-helper
[   53.291581] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[   53.297948] Bridge firewalling registered
[   53.304776] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[   53.318481] ip_tables: (C) 2000-2006 Netfilter Core Team
[   53.510300] Initializing XFRM netlink socket
[   53.515251] Netfilter messages via NETLINK v0.30.
[   53.518920] ctnetlink v0.93: registering with nfnetlink.
[   53.688086] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready


 One of the configured repositories failed (Unknown),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Disable the repository, so yum won't use it by default. Yum will then
        just ignore the repository until you permanently enable it again or use
        --enablerepo for temporary usage:

            yum-config-manager --disable <repoid>

     4. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true

Cannot find a valid baseurl for repo: amzn-main/latest
Could not retrieve mirrorlist http://repo.eu-central-1.amazonaws.com/latest/main/mirror.list error was
12: Timeout on http://repo.eu-central-1.amazonaws.com/latest/main/mirror.list: (28, 'Connection timed out after 5000 milliseconds')
Loaded plugins: priorities, update-motd, upgrade-helper


 One of the configured repositories failed (Unknown),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Disable the repository, so yum won't use it by default. Yum will then
        just ignore the repository until you permanently enable it again or use
        --enablerepo for temporary usage:

            yum-config-manager --disable <repoid>

     4. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true

Cannot find a valid baseurl for repo: amzn-main/latest
Could not retrieve mirrorlist http://repo.eu-central-1.amazonaws.com/latest/main/mirror.list error was
12: Timeout on http://repo.eu-central-1.amazonaws.com/latest/main/mirror.list: (28, 'Connection timed out after 5001 milliseconds')
/var/lib/cloud/instance/scripts/part-001: line 9: /opt/aws/bin/cfn-init: No such file or directory
/var/lib/cloud/instance/scripts/part-001: line 10: /opt/aws/bin/cfn-signal: No such file or directory
Aug 20 06:39:13 cloud-init[2286]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [127]
Aug 20 06:39:13 cloud-init[2286]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
Aug 20 06:39:13 cloud-init[2286]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_scripts_user.pyc'>) failed

我检查了 NACL,对于 Inbound 和 Outbound,一切都设置为 0.0.0.0/0 和 ALLOW。

对于第一个 VPC,我使用 ECS 优化的 AMI 和 t2.large(没有任何问题)和第二个 c5.xlarge(导致问题)。

什么仍然会导致 EC2 无法访问 Amazon 存储库?

编辑

后来我发现第二个 VPC 连接了 S3 端点。经过更多研究后,我在 LinkedIn 上发现了一篇不错的帖子:

Amazon Linux 存储库托管在 S3 上,因此 有必要在 S3 端点策略中允许对其进行访问。

所以当你启动 yum 时,它会利用本地 DNS 诡计的魔力 路由到内部 S3 端点

我继续更新我的 CloudFormation 模板并在下面的 LaunchConfiguration 中添加了其他策略,但这没有帮助:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": [
                "arn:aws:s3:::repo.eu-central-1.amazonaws.com",
                "arn:aws:s3:::repo.eu-central-1.amazonaws.com/*"
            ],
            "Effect": "Allow"
        }
    ]
}

端点策略如下所示:

{
    "Statement": [
        {
            "Action": "*",
            "Effect": "Allow",
            "Resource": "*",
            "Principal": "*"
        }
    ]
}

【问题讨论】:

    标签: amazon-web-services amazon-ec2 connectivity private-subnet


    【解决方案1】:

    在探索了 AWS 控制台的所有部分之后,我终于找到了导致问题的原因。正如我在对原始帖子的更新中所述,当 Endpoint 附加到 VPC 时,EC2 将尝试在内部解析包和存储库。我去检查了 Endpoint 的每个设置,发现只有添加到 Endpoint 的公共子网的路由表,并且在我也添加了私有子网之后,EC2 实例可以访问包和存储库。

    【讨论】:

    • 我遇到了一个非常相似的问题,yum 更新在部署在私有子网中的 ECS 优化 Amazon Linux 主机上失败。其他所有 yum 安装都可以正常工作。唯一的事情是,我们将私有子网路由表关联到 S3 网关端点。我们的公共子网与 S3 端点没有关联,添加它们并不能解决我们的问题。
    • 好的,刚刚想通了,我们设置 yum.conf 使用代理作为我们环境中安全状态的一部分。在 /etc/yum.repos.d/ 中将 proxy=_none_ 添加到 amzn2-core.repo 和 amzn2-extras.repo 为我解决了这个问题。这允许请求正确地通过 S3 VPC 端点路由,而不是将我们现有的代理与解析为 S3 的内部 IP 地址混淆。
    猜你喜欢
    • 2014-09-04
    • 1970-01-01
    • 2012-12-05
    • 2010-12-28
    • 2012-09-23
    • 2013-04-29
    • 1970-01-01
    • 2013-09-28
    • 2020-05-12
    相关资源
    最近更新 更多