使用 CloudFormation 模板运行爬虫答案

【问题标题】：Run a crawler using CloudFormation template使用 CloudFormation 模板运行爬虫
【发布时间】：2021-01-25 18:31:17
【问题描述】：

此 CloudFormation 模板按预期工作并创建本文所需的所有资源：

Data visualization and anomaly detection using Amazon Athena and Pandas from Amazon SageMaker | AWS Machine Learning Blog

但是WorkflowStartTrigger 资源实际上并没有运行爬虫。如何使用 CloudFormation 模板运行爬虫？

Resources:
  MyRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          -
            Effect: "Allow"
            Principal:
              Service:
                - "glue.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      Path: "/"
      Policies:
        -
          PolicyName: "root"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              -
                Effect: "Allow"
                Action: "*"
                Resource: "*"
 
  MyDatabase:
    Type: AWS::Glue::Database
    Properties:
      CatalogId: !Ref AWS::AccountId
      DatabaseInput:
        Name: "dbcrawler123"
        Description: "TestDatabaseDescription"
        LocationUri: "TestLocationUri"
        Parameters:
          key1 : "value1"
          key2 : "value2"
 
  MyCrawler2:
    Type: AWS::Glue::Crawler
    Properties:
      Description: example classifier
      Name: "testcrawler123"
      Role: !GetAtt MyRole.Arn
      DatabaseName: !Ref MyDatabase
      Targets:
        S3Targets:
          - Path: 's3://nytaxi162/'
      SchemaChangePolicy:
        UpdateBehavior: "UPDATE_IN_DATABASE"
        DeleteBehavior: "LOG"
      TablePrefix: test-
      Configuration: "{\"Version\":1.0,\"CrawlerOutput\":{\"Partitions\":{\"AddOrUpdateBehavior\":\"InheritFromTable\"},\"Tables\":{\"AddOrUpdateBehavior\":\"MergeNewColumns\"}}}"


  WorkflowStartTrigger:
    Type: AWS::Glue::Trigger
    Properties:
      Description: Trigger for starting the Crawler
      Name: StartTrigger
      Type: ON_DEMAND
      Actions:
        - CrawlerName: "testcrawler123"

【问题讨论】：

在这个答案中，提到这是不可能的，我需要编写 shell/python 脚本。那是对的吗？ stackoverflow.com/questions/57967989/…

标签： amazon-cloudformation

【解决方案1】：

您应该能够通过创建附加到 lambda 的自定义资源来做到这一点，由此 lambda 实际执行启动爬虫的操作。您甚至应该可以让它等待爬虫完成其执行

【讨论】：

【解决方案2】：

CloudFormation 直接不运行爬虫，它只是创建它们。但是您可以创建一个计划以便在定义触发器时运行爬虫：

ScheduledJobTrigger:
  Type: 'AWS::Glue::Trigger'
  Properties:
    Type: SCHEDULED
    StartOnCreation: true
    Description: DESCRIPTION_SCHEDULED
    Schedule: cron(5 * * * ? *)
    Actions:
      - CrawlerName: "testcrawler123"
    Name: ETLGlueTrigger

如果需要在 CloudFormation 堆栈创建过程中运行爬虫，可以使用 Lambda。

【讨论】：