How to use CloudFormation to update AWS Glue Jobs

【问题标题】：How to use CloudFormation to update AWS Glue JobsHow to use CloudFormation to update AWS Glue Jobs
【发布时间】：2022-11-20 20:56:13
【问题描述】：

We have many AWS Glue jobs and we are only updating the job code, which are scripts stored in S3. The problem is CloudFormation couldn't tell when and when not to update our Glue jobs because all CloudFormation template parameters remain the same after script changes, even the script location is pointing to the same S3 object.

【问题讨论】：

How is this related to terraform?
What do you mean by updating a job? It's source code?

标签： amazon-web-services amazon-cloudformation aws-glue cicd

【解决方案1】：

As this is similar to Lambda code package, you can use parameterize the Glue script path and then have different versions of Glue script file

Glue CFT has "Command" parameter taking in "JobCommand" Type value which includes "ScriptLocation" attribute, make this as a CFT parameter and have the script dynamic

{
  "Name" : String,
  "PythonVersion" : String,
  "ScriptLocation" : String
}

You can probably setup a CI/CD pipeline using AWS CodePipeline or your 3rd party CI/CD tool with the below steps

Pull the new code from your SCM like Github to deploy S3
Update CloudFormation stack with new S3 script path (with versions like v1, v2 etc...)

【讨论】：