通过 Cloud Formation 创建 Amazon Elasticsearch Service 时出现 CloudWatch 资源访问策略错误答案

【问题标题】：CloudWatch resource access policy error while creating Amazon Elasticsearch Service via Cloud Formation通过 Cloud Formation 创建 Amazon Elasticsearch Service 时出现 CloudWatch 资源访问策略错误
【发布时间】：2020-11-04 18:49:06
【问题描述】：

我正在尝试创建一个启用了LogPublishingOptions 的弹性搜索域。虽然启用 LogPublishingOptions ES 表示它没有足够的权限在 Cloudwatch 上创建 LogStream。

我尝试创建一个具有角色的策略并将该策略附加到 ES 引用的 LogGroup 但它不起作用。以下是我的弹性搜索云形成模板，

AWSTemplateFormatVersion: 2010-09-09

Resources:
  MYLOGGROUP:
    Type: 'AWS::Logs::LogGroup'
    Properties:
      LogGroupName: index_slow

  MYESROLE:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: es.amazonaws.com
            Action: 'sts:AssumeRole'
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/AmazonESFullAccess'
        - 'arn:aws:iam::aws:policy/CloudWatchFullAccess'
      RoleName: !Join
        - '-'
        - - es
          - !Ref 'AWS::Region'

  PolicyDocESIndexSlow :
    Type: AWS::IAM::Policy
    Properties:
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: 
             - logs:PutLogEvents
             - logs:CreateLogStream
            Resource: 'arn:aws:logs:*'
      PolicyName: !Ref MYLOGGROUP
      Roles:
        - !Ref MYESROLE

  MYESDOMAIN:
    Type: AWS::Elasticsearch::Domain
    Properties:
      DomainName: 'es-domain'
      ElasticsearchVersion: '7.4'
      ElasticsearchClusterConfig:
        DedicatedMasterCount: 3
        DedicatedMasterEnabled: True
        DedicatedMasterType: 'r5.large.elasticsearch'
        InstanceCount: '2'
        InstanceType: 'r5.large.elasticsearch'
      EBSOptions:
        EBSEnabled: True
        VolumeSize: 10
        VolumeType: 'gp2'
      AccessPolicies:
        Version: 2012-10-17
        Statement:
          - Effect: Deny
            Principal:
              AWS: '*'
            Action: 'es:*'
            Resource: '*'
      AdvancedOptions:
        rest.action.multi.allow_explicit_index: True
      LogPublishingOptions:
        INDEX_SLOW_LOGS:
          CloudWatchLogsLogGroupArn: !GetAtt
            - MYLOGGROUP
            - Arn
          Enabled: True
      VPCOptions:
        SubnetIds:
          - !Ref MYSUBNET
        SecurityGroupIds:
          - !Ref MYSECURITYGROUP
  MYVPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.0.0.0/16
  MYSUBNET:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref MYVPC
      CidrBlock: 10.0.0.0/16
  MYSECURITYGROUP:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: security group for elastic search domain
      VpcId: !Ref MYVPC
      GroupName: 'SG for ES'
      SecurityGroupIngress:
        - FromPort: '443'
          IpProtocol: tcp
          ToPort: '443'
          CidrIp: 0.0.0.0/0

在执行时，它会创建除 MYESDOMAIN 之外的所有资源。它说

为 CloudWatch Logs 日志组 index_slow 指定的资源访问策略未授予 Amazon Elasticsearch Service 创建日志流的足够权限。请检查资源访问策略。（服务：AWSElasticsearch；状态码：400；错误码：ValidationException）

知道这里缺少什么吗？

【问题讨论】：

标签： amazon-web-services elasticsearch amazon-cloudformation amazon-iam aws-cloudformation-custom-resource

【解决方案1】：

我认为这里对于应该更新/设置哪些策略以启用 ES 写入日志组存在一些混淆。

我认为您应该将 PolicyDocESIndexSlow 策略应用于 CloudWatch Logs。

据我记忆，这无法在 CloudFormation 中完成。您必须使用put-resource-policy、相应的 API 调用或控制台，如下所示：

Viewing Amazon Elasticsearch Service Slow Logs

【讨论】：

像魅力一样工作！

【解决方案2】：

最终的代码会是这样的，

部署 lambda_function.py

import logging
import time

import boto3
import json
from crhelper import CfnResource

logger = logging.getLogger(__name__)
helper = CfnResource(json_logging=False, log_level='DEBUG', boto_level='CRITICAL', sleep_on_delete=120)

try:
    # Init code goes here
    pass
except Exception as e:
    helper.init_failure(e)


@helper.create
@helper.update
def create(event, _):
    logger.info("Got Create/Update")

    my_log_group_arn = event['ResourceProperties']['MYLOGGROUPArn']

    client = boto3.client('logs')

    policy_document = dict()
    policy_document['Version'] = '2012-10-17'
    policy_document['Statement'] = [{
        'Sid': 'ESLogsToCloudWatchLogs',
        'Effect': 'Allow',
        'Principal': {
            'Service': [
                'es.amazonaws.com'
            ]
        },
        'Action': 'logs:*',
    }]

    policy_document['Statement'][0]['Resource'] = my_log_group_arn 
    client.put_resource_policy(policyName='ESIndexSlowPolicy', policyDocument=json.dumps(policy_document))

    helper.Data['success'] = True
    helper.Data['message'] = 'ES policy deployment successful'

    # To return an error to Cloud Formation you raise an exception:
    if not helper.Data["success"]:
        raise Exception('Error message to cloud formation')

    return "MYESIDDEFAULT"


@helper.delete
def delete(event, _):
    logger.info("Got Delete")
    # Delete never returns anything. Should not fail if the underlying resources are already deleted.
    # Desired state.

    try:
        client = boto3.client('logs')
        client.delete_resource_policy(policyName='ESIndexSlowPolicy')

    except Exception as ex:
        logger.critical(f'ES policy delete failed with error [{repr(ex)}]')


def lambda_handler(event, context):
    helper(event, context)

以及CF模板中的一些附加组件

 MYLAMBDAROLE:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/AWSLambdaFullAccess'
        - 'arn:aws:iam::aws:policy/AmazonS3FullAccess'
        - 'arn:aws:iam::aws:policy/AmazonESFullAccess'
        - 'arn:aws:iam::aws:policy/CloudWatchFullAccess'
      RoleName: !Join
        - '-'
        - - lambda-role
          - !Ref 'AWS::Region'

  MYLAMBDADEPLOY:
    Type: 'AWS::Lambda::Function'
    Properties:
      Code:
        S3Bucket: es-bucket-for-lambda-ta86asdf596
        S3Key: es.zip
      FunctionName: deploy_es
      Handler: lambda_function.lambda_handler
      MemorySize: 128
      Role: !GetAtt
        - MYLAMBDAROLE
        - Arn
      Runtime: python3.8
      Timeout: 60

  MYESSETUP:
    Type: 'Custom::MYESSETUP'
    Properties:
      ServiceToken: !GetAtt
        - MYLAMBDADEPLOY
        - Arn
      MYLOGGROUPArn: !GetAtt
        - MYLOGGROUP
        - Arn
    DependsOn:
      - MYLAMBDADEPLOY
      - MYLOGGROUP

只需在下面添加DependsOn 到 MYESDOMAIN

DependsOn:
  - MYESSETUP

【讨论】：

【解决方案3】：

2021 年更新

有一个名为 AWS::Logs::ResourcePolicy 的 CloudFormation 资源允许在 CF 中为 CloudWatch Logs 定义策略。我发现的主要问题是它只接受一个真正的字符串作为值。尝试使用 Ref、Join 等组合字符串一直被拒绝。如果有人能完成这项工作，那就太棒了。

用 YAML 编写更容易，因为 JSON 需要转义所有 " 字符。

OSLogGroupPolicy:
    Type: AWS::Logs::ResourcePolicy
    Properties:
      PolicyName: AllowES
      PolicyDocument: '{"Version": "2012-10-17","Statement":[{"Effect":"Allow","Principal": {"Service": ["es.amazonaws.com"]},"Action":["logs:PutLogEvents","logs:CreateLogStream"],"Resource":"*"}]}'

【讨论】：