AWS cdk python，具有每日触发器的胶水爬虫的哪个IAM角色？答案

【问题标题】：AWS cdk python, which IAM role for a glue crawler with a daily trigger?AWS cdk python，具有每日触发器的胶水爬虫的哪个IAM角色？
【发布时间】：2020-11-29 13:22:09
【问题描述】：

我正在尝试为 s3 部署 glue crawler。不幸的是，我无法找到允许爬虫运行的适当 IAM 角色。我需要的权限只是读取/写入 S3 和日志：PutLogsEvent，但不知何故我没有做对。这是我的代码，可以部署但crawler没有运行权限。

from aws_cdk import (
    aws_events as events,
    aws_lambda as lambda_,
    aws_events_targets as targets,
    aws_iam as iam,
    aws_glue as glue,
    core
)

class MyStack(core.Stack):

    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        # what should I put in the role exactly?
        glue_role = iam.Role(
            self, 'Role__arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole',
            assumed_by=iam.ServicePrincipal('glue.amazonaws.com'),
        )
        glue_trigger = glue.CfnTrigger(self, "glue-daily-trigger",
            name = "etl-trigger",
            schedule = "cron(5 * * * ? *)", # every hour at X.05, every day
            type="SCHEDULED",
            actions=[
                {
                    "jobName": "glue_crawler-daily"
                }
            ],
            start_on_creation=True
        )
        crawler_name = 'crawler_units_data'
        glue_crawler = glue.CfnCrawler(
            self, crawler_name,
            name=crawler_name,
            database_name='data_science',
            role=glue_role.role_arn,#'arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole',
            targets={"s3Targets": [{"path": "s3://random_s3/units/"}]},
        )
        glue_trigger.add_depends_on(glue_crawler)

我尝试了几件事并从 javascript 示例 like this one 翻译代码，但从 javascript 调用的方法不能 100% 映射到 python。

此角色（从 GUI 创建）正常工作并具有 2 个策略。

s3 读写策略

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::random_s3/units*"
            ]
        }
    ]
}

AWSGlueServicePolicy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "glue:*",
                "s3:GetBucketLocation",
                "s3:ListBucket",
                "s3:ListAllMyBuckets",
                "s3:GetBucketAcl",
                "ec2:DescribeVpcEndpoints",
                "ec2:DescribeRouteTables",
                "ec2:CreateNetworkInterface",
                "ec2:DeleteNetworkInterface",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVpcAttribute",
                "iam:ListRolePolicies",
                "iam:GetRole",
                "iam:GetRolePolicy",
                "cloudwatch:PutMetricData"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:CreateBucket"
            ],
            "Resource": [
                "arn:aws:s3:::aws-glue-*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::aws-glue-*/*",
                "arn:aws:s3:::*/*aws-glue-*/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::crawler-public*",
                "arn:aws:s3:::aws-glue-*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:*:*:/aws-glue/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags",
                "ec2:DeleteTags"
            ],
            "Condition": {
                "ForAllValues:StringEquals": {
                    "aws:TagKeys": [
                        "aws-glue-service-resource"
                    ]
                }
            },
            "Resource": [
                "arn:aws:ec2:*:*:network-interface/*",
                "arn:aws:ec2:*:*:security-group/*",
                "arn:aws:ec2:*:*:instance/*"
            ]
        }
    ]
}

【问题讨论】：

你可以试试glue_role = iam.Role(scope=self, id="Role__id",假定_by=aws_iam.ServicePrincipal("glue.amazonaws.com"), )。 ?如果仍然无法正常工作，还请使用附加到该角色的 IAM 政策详细信息更新您的问题
@PrabhakarReddy , 1. 我到底应该尝试什么？您建议的代码与我发布的代码完全相同。 2) 将策略详细信息附加到角色是什么意思？类似glue_role.add_to_policy..?
你能把 'glue_role' 改成 id="Role__id" id 这里的意思是你使用的 Glue 角色的完整 ARN。附加到您拥有的 Glue 角色的 IAM 策略
@PrabhakarReddy 我更新了这个问题。爬虫已创建，但我无法为其分配正确的 IAM 角色。正如 GUI 所建议的那样 This role must provide permissions similar to the AWS managed policy, AWSGlueServiceRole, plus access to your data stores.

标签： python-3.6 amazon-cloudformation amazon-iam aws-glue aws-cdk

【解决方案1】：

事实证明，我需要以不同的方式传递名称和策略

    glue_role = iam.Role(
        self, 'glue_role_id2323',
        role_name = 'Rolename',
        assumed_by=iam.ServicePrincipal('glue.amazonaws.com'),
        managed_policies=[iam.ManagedPolicy.from_aws_managed_policy_name('service-role/AWSGlueServiceRole')]
    )

【讨论】：