【问题标题】:AWS CodePipeline: pass Lambda function output to CloudFormationAWS CodePipeline:将 Lambda 函数输出传递给 CloudFormation
【发布时间】:2019-04-03 12:57:09
【问题描述】:

我想使用 CodePipeline 运行 CloudFormation 模板。此模板需要一个需要包含当前日期/时间的输入参数。不幸的是,CloudFormation 无法自行生成当前的 DateTime。

我的方法是首先运行一个简单的 Lambda 函数来创建当前时间戳并将其保存为 OutputArtifacts。随后的 CloudFormation 任务将该工件导入为 InputArtifacts,并从 DateTime 属性中获取值,并通过 ParameterOverrides 指令将其传递给 CloudFormation。

不幸的是,CodePipeline 一直说 DateTimeInput 参数无效(显然 GetArtifactAtt 查找失败)。 我假设 lambda 输出(python:print)没有正确保存为工件?

您知道如何正确传递 lambda 输出,或者您知道如何以更好的方式实现这一目标?

所有管道组件都使用 CloudFormation 定义为 YAML。以下是相关部分:

Lambda 函数:

Resources:
  ...
  GetDateTimeFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.lambda_handler
      Runtime: python2.7
      Timeout: '10'
      Role: !GetAtt GetDateTimeFunctionExecutionRole.Arn
      Code:
        ZipFile: |
                import datetime
                import boto3
                import json

                code_pipeline = boto3.client('codepipeline')

                def lambda_handler(event, context):
                  now = datetime.datetime.now().strftime("%Y%m%d%H%M%S")
                  responseData = {'DateTime':now}
                  print json.dumps(responseData)
                  response = code_pipeline.put_job_success_result(jobId=event['CodePipeline.job']['id'])
                  return response

这里是管道任务:

Resources:
...
  Pipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      ArtifactStore:
        Location: !Ref ArtifactStoreBucket
        Type: S3
      DisableInboundStageTransitions: []
      Name: !Ref PipelineName
      RoleArn: !GetAtt PipelineRole.Arn
      Stages:
        - Name: Deploy
          Actions:
            - Name: GetDateTime
              RunOrder: 1
              ActionTypeId:
                Category: Invoke
                Owner: AWS
                Provider: Lambda
                Version: '1'
              Configuration:
                 FunctionName: !Ref GetDateTimeFunction
              OutputArtifacts:
                - Name: GetDateTimeOutput
            - Name: CreateStack
              RunOrder: 2
              ActionTypeId:
                Category: Deploy
                Owner: AWS
                Provider: CloudFormation
                Version: '1'
              InputArtifacts:
                - Name: TemplateSource
                - Name: GetDateTimeOutput
              Configuration:
                ActionMode: REPLACE_ON_FAILURE
                Capabilities: CAPABILITY_IAM
                RoleArn: !GetAtt CloudFormationRole.Arn
                StackName: !Ref CFNStackname
                TemplatePath: !Sub TemplateSource::${CFNScriptfile}
                TemplateConfiguration: !Sub TemplateSource::${CFNConfigfile}
                ParameterOverrides: |
                  {
                    "DateTimeInput" : { "Fn::GetArtifactAtt" : [ "GetDateTimeOutput", "DateTime" ] }
                  }

更新:我天真地认为会有一个简单的方法。 现在我知道使用 lambda 交付一个简单的输出工件是一项更高级和手动的任务。

在 python 代码中,必须评估传递的 event 字典 (CodePipeline.job) 以进行查找:
- 预定义的 OutputArtifacts (S3 Bucket/Key) 和
- CodePipeline 提供的临时 S3 会话凭证。
然后必须通过这些凭据初始化 S3 客户端。 S3 put_object 之后需要运行。

https://docs.aws.amazon.com/codepipeline/latest/userguide/actions-invoke-lambda-function.html https://forums.aws.amazon.com/thread.jspa?threadID=232174

所以我的问题又是:你们知道如何以更好或更简单的方式实现这一目标吗?
我只想将当前日期和时间作为 CloudFormation 的输入参数,不想破坏自动化。

【问题讨论】:

    标签: lambda amazon-cloudformation aws-codepipeline artifact


    【解决方案1】:

    是的,我不知道自己需要手动处理输出工件。

    终于成功了:

      GetDateTimeFunction:
        Type: AWS::Lambda::Function
        Properties:
          Handler: index.lambda_handler
          Runtime: python2.7
          Timeout: '10'
          Role: !GetAtt GetDateTimeFunctionExecutionRole.Arn
          Code:
            ZipFile: |
                    from __future__ import print_function
                    from boto3.session import Session
                    from zipfile import ZipFile
                    import json
                    import datetime
                    import boto3
                    import botocore
                    import traceback
                    import os
                    import shutil
    
    
                    code_pipeline = boto3.client('codepipeline')
    
                    def evaluate(event):
                        # Extract attributes passed in by CodePipeline
                        job_id = event['CodePipeline.job']['id']
                        job_data = event['CodePipeline.job']['data']
    
                        config = job_data['actionConfiguration']['configuration']
                        credentials = job_data['artifactCredentials']
    
                        output_artifact = job_data['outputArtifacts'][0]
                        output_bucket = output_artifact['location']['s3Location']['bucketName']
                        output_key = output_artifact['location']['s3Location']['objectKey']
    
                        # Temporary credentials to access CodePipeline artifact in S3
                        key_id = credentials['accessKeyId']
                        key_secret = credentials['secretAccessKey']
                        session_token = credentials['sessionToken']
    
                        return (job_id, output_bucket, output_key, key_id, key_secret, session_token)
    
                    def create_artifact(data):
                        artifact_dir = '/tmp/output_artifacts/'+str(uuid.uuid4())
                        artifact_file = artifact_dir+'/files/output.json'
                        zipped_artifact_file = artifact_dir+'/artifact.zip'
                        try:
                            shutil.rmtree(artifact_dir+'/files/')
                        except Exception:
                            pass
                        try:
                            os.remove(zipped_artifact_file)
                        except Exception:
                            pass
                        os.makedirs(artifact_dir+'/files/')
                        with open(artifact_file, 'w') as outfile:
                            json.dump(data, outfile)
                        with ZipFile(zipped_artifact_file, 'w') as zipped_artifact:
                            zipped_artifact.write(artifact_file, os.path.basename(artifact_file))
    
                        return zipped_artifact_file
    
                    def init_s3client (key_id, key_secret, session_token):
                        session = Session(aws_access_key_id=key_id, aws_secret_access_key=key_secret, aws_session_token=session_token)
                        s3client = session.client('s3', config=botocore.client.Config(signature_version='s3v4'))
                        return s3client
    
                    def lambda_handler(event, context):
                        try:
                            (job_id, output_bucket, output_key, key_id, key_secret, session_token)=evaluate(event)
                            (s3client)=init_s3client(key_id, key_secret, session_token)
    
                            now=datetime.datetime.now().strftime('%Y-%m-%d_%H:%M:%S')
                            data={"DateTime":now}
    
                            (zipped_artifact_file)=create_artifact(data)
    
                            s3client.upload_file(zipped_artifact_file, output_bucket, output_key, ExtraArgs={"ServerSideEncryption": "AES256"})
    
                            # Tell CodePipeline we succeeded
                            code_pipeline.put_job_success_result(jobId=job_id)
    
                        except Exception as e:
                            print("ERROR: " + repr(e))
                            message=repr(e)
                            traceback.print_exc()
                            # Tell CodePipeline we failed
                            code_pipeline.put_job_failure_result(jobId=job_id, failureDetails={'message': message, 'type': 'JobFailed'})
    
                        return "complete"
    
    
      Pipeline:
        Type: AWS::CodePipeline::Pipeline
        Properties:
          ArtifactStore:
            Location: !Ref ArtifactStoreBucket
            Type: S3
          DisableInboundStageTransitions: []
          Name: !Ref PipelineName
          RoleArn: !GetAtt PipelineRole.Arn
          Stages:
            - Name: S3Source
              Actions:
                - Name: TemplateSource
                  RunOrder: 1
                  ActionTypeId:
                    Category: Source
                    Owner: AWS
                    Provider: S3
                    Version: '1'
                  Configuration:
                    S3Bucket: !Ref ArtifactStoreBucket
                    S3ObjectKey: !Ref SourceS3Key
                  OutputArtifacts:
                    - Name: TemplateSource
            - Name: Deploy
              Actions:
                - Name: GetDateTime
                  RunOrder: 1
                  ActionTypeId:
                    Category: Invoke
                    Owner: AWS
                    Provider: Lambda
                    Version: '1'
                  Configuration:
                     FunctionName: !Ref GetDateTimeFunction
                  OutputArtifacts:
                    - Name: GetDateTimeOutput
                - Name: CreateStack
                  RunOrder: 2
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Provider: CloudFormation
                    Version: '1'
                  InputArtifacts:
                    - Name: TemplateSource
                    - Name: GetDateTimeOutput
                  Configuration:
                    ActionMode: REPLACE_ON_FAILURE
                    Capabilities: CAPABILITY_IAM
                    RoleArn: !GetAtt CloudFormationRole.Arn
                    StackName: !Ref CFNStackname
                    TemplatePath: !Sub TemplateSource::${CFNScriptfile}
                    TemplateConfiguration: !Sub TemplateSource::${CFNConfigfile}
                    ParameterOverrides: |
                      {
                        "DateTimeInput" : { "Fn::GetParam" : ["GetDateTimeOutput", "output.json", "DateTime"]}
                      }
    

    这样一个微不足道的任务的开销相当大;-)

    【讨论】:

      【解决方案2】:

      您应该使用“Fn::GetParam”而不是“Fn::GetArtifactAtt”。根据 CloudFormation 文档,“Fn::GetArtifactAtt”只能获取工件的属性,如 BucketName、ObjectKey 和 URL。 "Fn::GetParam" 可以从工件中的 json 文件中获取值。因此,如果您可以将工件“GetDateTimeOutput”生成为 zip 文件,其中包括一个 JSON 文件(例如 param.json),其内容如下

      { “日期时间”:“2018/10/31 13:32:00” }

      然后你可以使用 { "Fn::GetParam" : [ "GetDateTimeOutput", "param.json", "DateTime" ] } 来获取时间。

      您可以修改您的 Lambda 函数来执行此操作,或使用 CodeBuild 操作。 CodeBuild 负责创建 zip,您只需指定构建命令即可在输出文件夹中创建 JSON 文件。您可以在以下文档中找到有关如何在 CodePipeline 中使用 CodeBuild 的更多信息。

      CloudFormation 文档 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-delivery-codepipeline-parameter-override-functions.html#w2ab1c13c17b9

      CodeBuild 文档 https://docs.aws.amazon.com/codebuild/latest/userguide/how-to-create-pipeline.html

      【讨论】:

        猜你喜欢
        • 2021-03-27
        • 2020-11-06
        • 2017-11-18
        • 1970-01-01
        • 2020-06-28
        • 2017-12-30
        • 1970-01-01
        • 2018-02-25
        • 1970-01-01
        相关资源
        最近更新 更多