AWS Batch：使用云形成将 efs 卷挂载到容器中

Question

我有一个带有作业定义的 AWS Batch 计算环境。

我使用 Cloud Formation 创建了这一切。

现在我想在此作业定义中添加一个 EFS 卷（名称：EFS-000，文件系统 ID：fs-9999999）和一个 MountPoint。

我读过

• https://docs.aws.amazon.com/AmazonECS/latest/developerguide/tutorial-efs-volumes.html
• https://aws.amazon.com/es/premiumsupport/knowledge-center/batch-mount-efs

在第一个链接中，我们有一个任务定义示例（AWS ECS 而不是 AWS Batch 概念）

{
    "containerDefinitions": [
        {
            "memory": 128,
            "portMappings": [
                {
                    "hostPort": 80,
                    "containerPort": 80,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "mountPoints": [
                {
                    "containerPath": "/usr/share/nginx/html",
                    "sourceVolume": "efs-html"
                }
            ],
            "name": "nginx",
            "image": "nginx"
        }
    ],
    "volumes": [
        {
            "name": "efs-html",
            "efsVolumeConfiguration": {
                "fileSystemId": "fs-1324abcd",
                "transitEncryption": "ENABLED"
            }
        }
    ],
    "family": "efs-tutorial"
}

似乎很容易将正确的代码添加到我的 Cloud Formation 配方中（我选择了 yaml 语法）。我在 ContainerDefinition 中添加了...

Volumes:
  - Name: SRV 
    EfsVolumeConfiguration:
      FileSystemId: fs-9999999
      TransitEncryption: ENABLED

但是当我运行 Cloud Formation 配方时，我得到......

The following resource(s) failed to update: [ContentJob1].

Property validation failure: [Encountered unsupported properties in {/ContainerProperties/Volumes/0}: [EfsVolumeConfiguration]]

如果 EfsVolumeConfiguration 不是有效属性...

如何使用 Cloud Formation 将 EFS 卷添加到 AWS 批处理作业定义？

Answer 1

在 CloudFormation 支持 EFS 之前，我一直使用这个解决方案。但它仍然有用。

我创建了一个启动模板

  LaunchTemplate:
    Type: AWS::EC2::LaunchTemplate
    Properties: 
      LaunchTemplateName: !Join ['', [!Ref ServiceName, '-EC2-', LaunchTemplate]]
      LaunchTemplateData: 
        UserData:
          Fn::Base64: !Sub |
            MIME-Version: 1.0 
            Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

            --==MYBOUNDARY==
            Content-Type: text/cloud-config; charset="us-ascii"

            runcmd:
            - mkdir /mnt/efs-misc
            - mkdir /mnt/efs-jobs
            - echo "${EfsJobs}:/ /mnt/efs-jobs nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=300,retrans=2,noresvport 0 0" >> /etc/fstab
            - echo "${EfsMisc}:/ /mnt/efs-misc nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=300,retrans=2,noresvport 0 0" >> /etc/fstab
            - mount -a
            --==MYBOUNDARY==--
        InstanceMarketOptions:
          MarketType: spot
          SpotOptions:
            MaxPrice: 0.096

然后我创建了一个使用这个模板的计算环境。

  BatchCompute:
    Type: 'AWS::Batch::ComputeEnvironment'
    DependsOn: LaunchTemplate
    Properties:
      ComputeEnvironmentName: !Join ['', [!Ref ServiceName, '-EC2']]
      ComputeResources:
        AllocationStrategy: SPOT_CAPACITY_OPTIMIZED
        DesiredvCpus: 0
        Ec2KeyPair: !Join ['-', [MyComputeEnv, !Ref EnvironmentLow]]
        InstanceRole: !GetAtt BatchInstanceProfile.Arn
        InstanceTypes:
          - c4.large
          - c5.large
        LaunchTemplate: 
          LaunchTemplateId: !Ref LaunchTemplate
          Version: $Latest
        MaxvCpus: 256
        MinvCpus: 0
        DesiredvCpus: 0
        SecurityGroupIds:
          - !Ref DefaultSecurityGroup
        Subnets: !Split [',', !Join [',', [!Ref SubnetA, !Ref SubnetB, !Ref SubnetC ]]]
        Type: SPOT
        SpotIamFleetRole: SpotFleetTagServiceRole
      ServiceRole: !Ref BatchServiceRole
      State: ENABLED
      Type: Managed

最后，我使用了 AWS::Batch::JobDefinition 的 ContainerProperties/[Volumes|MountPoints] 来添加我的 EFS 卷。

  ContentJob0:
    Type: 'AWS::Batch::JobDefinition'
    Properties:
      Type: Container
      ContainerProperties:
        Command: 
          - ls
        Environment:
          - Name: AAA
            Value: AAA
        ExecutionRoleArn: !GetAtt BatchJobRole.Arn
        Image: !Join ['', [!Ref 'AWS::AccountId','.dkr.ecr.', !Ref 'AWS::Region', '.amazonaws.com/', !Ref Image ]]
        LogConfiguration:
          LogDriver: awslogs
        JobRoleArn: !Ref BatchContainerRole
        Memory: 2048
        Vcpus: 2
        Volumes:
          - Name: MISC
            Host:
              SourcePath: '/mnt/efs-0'
          - Name: JOBS
            Host:
              SourcePath: '/mnt/efs-1'
        MountPoints:
          - SourceVolume: MISC
            ContainerPath: '/mnt/efs-0'
          - SourceVolume: JOBS
            ContainerPath: '/mnt/efs-1'
      JobDefinitionName: !Join ['-', ['MyJob', !Ref ServiceName, 'EC2', '0']]
      PlatformCapabilities:
        - EC2
      PropagateTags: true
      RetryStrategy: 
        Attempts: 3
      Timeout: 
        AttemptDurationSeconds: 300

由于我需要在 LaunchTemplate 中设置 SPOT 实例和其他东西的价格，这对我来说是一个很好的解决方案。

        InstanceMarketOptions:
          MarketType: spot
          SpotOptions:
            MaxPrice: 0.096

Answer 2

AWS 批处理 added EFS Volume support on April 1, 2021.

用户文档位于https://docs.aws.amazon.com/batch/latest/userguide/efs-volumes.html

博客文章可在https://aws.amazon.com/blogs/hpc/introducing-support-for-per-job-amazon-efs-volumes-in-aws-batch/获取