【问题标题】:Error creating application autoscaling target on AWS when using Terraform - Defining `scalable_resource` for custom `aws_appautoscaling_target`使用 Terraform 在 AWS 上创建应用程序自动缩放目标时出错 - 为自定义 `aws_appautoscaling_target` 定义`scalable_resource`
【发布时间】:2020-07-25 13:37:44
【问题描述】:

目标

我正在为 Kinesis 数据流实施自动缩放解决方案。

aws-samples/aws-application-auto-scaling-kinesis 存储库中详细记录了我正在关注的一种可能的解决方案。但是,示例代码使用了 yaml 模板。我希望使用 定义相同的内容。

到目前为止的故事

尝试为my_custom_resource 创建缩放目标时,

resource "aws_appautoscaling_target" "my_custom_resource" {
  resource_id        = "https://${aws_api_gateway_rest_api.my_api.id}.execute-api.${var.region}.amazonaws.com/prod/scalableTargetDimensions/${var.stream}"
  scalable_dimension = "custom-resource:ResourceType:Property"
  service_namespace  = "custom-resource"
}

所有属性都是按照AWS Auto-Scaling docs构建的

在链接的 AWS 存储库中使用 CloudFormation 创建相同的资源:

KinesisAutoScaling:
  Type: AWS::ApplicationAutoScaling::ScalableTarget
  DependsOn: LambdaScaler
  Properties:
    ResourceId: !Sub https://${MyApi}.execute-api.${AWS::Region}.amazonaws.com/prod/scalableTargetDimensions/${MyKinesisStream}
    ScalableDimension: 'custom-resource:ResourceType:Property'
    ServiceNamespace: custom-resource

注意:为简洁起见,省略了不相关的属性。

问题

terraform apply 产生以下错误:

错误:创建应用程序自动缩放目标时出错: ValidationException:资源验证失败: https://k5df89sd23.execute-api.us-west-1.amazonaws.com/prod/scalableTargetDimensions/my-test-stream, 可扩展维度:自定义资源:资源类型:属性。 原因:找不到可扩展的资源 在 application-autoscaling.tf 第 9 行,资源“aws_appautoscaling_target”“my_custom_resource”中: 9:资源“aws_appautoscaling_target”“my_custom_resource”{

terraform 定义中可能有什么问题?


我知道 Terraform 支持使用 aws_cloudformation_stack 的 CloudFormation 模板 - 我真的希望避免这种解决方法。

【问题讨论】:

  • @matt-schuchard - 你认为我的资源定义没有缺陷吗?我对 terraform 很陌生,所以不能确定。
  • 没关系;只是仔细检查了文档,您对 resource_id 的值不正确:terraform.io/docs/providers/aws/r/…
  • @MattSchuchard - 在该页面上,如果您向下滚动到属性并点击resource_id 中提到的链接,则会转到 AWS 文档。那里有一个自定义资源部分,这是我在到达我的resource_id 时所遵循的。还是错了吗?

标签: cloudformation terraform amazon-web-services terraform amazon-cloudformation aws-auto-scaling


【解决方案1】:

免责声明:我知道这个问题已经问过一段时间了,但我还是要回答以防万一它可能对其他人有所帮助......它肯定会对我有所帮助


假设您正在尝试重现 link you put in the question 的内容,那么我认为可能是错误的:

当然是部分答案:aws_appautoscaling_target.my_custom_resource.resource_id

如果您创建了一个 aws_api_gateway_deployment 资源来重现 aws-sampleMyApi 部分,那么您很幸运!
你可以用这个:

resource "aws_appautoscaling_target" "my_custom_resource" {
  # ...
  resource_id = "${aws_api_gateway_deployment.gateway.invoke_url}/scalableTargetDimensions/${var.stream}"
  # ...

如果您想了解如何让这个 ^ 工作的详细信息,请参阅下文...

但请记住,这可能还不够!


可能也是解决方案的一部分:创建aws_appautoscaling_target 所需的内部链接和权限

始终假设您正在关注same example,并且您已经使用terraform 实现了大部分 所有...

还有

免责声明:我不确定此答案其余部分的确切原因。我怀疑这与AWS Auto Scaling服务的内部API的要求有关。


TL;DR:在注册自动缩放目标之前,一切都需要连接并正常工作

这部分:

README.mdthis other project that covers the integration you're trying to achieve 暗示了这一点。

长版;我在实施中必须解决的问题:

  1. Lambda 响应格式
  2. API 网关设置和接线
  3. 正确的 IAM 权限
  4. Lambda 和 APIGateway 工作正常

1。确保你有正确的 lambda 响应

lambda 的返回值的返回格式对于 aws_appautoscaling_target 的创建至关重要。
它需要完全

    returningJson = {
      "actualCapacity": float(actualCapacity),
      "desiredCapacity": float(desiredCapacity),
      "dimensionName": resourceName,
      "resourceName": resourceName,
      "scalableTargetDimensionId": resourceName,
      "scalingStatus": scalingStatus,
      "version": "MyVersion"
    }
    
    try:
        returningJson['failureReason'] = failureReason
    except:
        pass

(这是示例中定义的方式)...
在我的实现中,我曾尝试过它(在一切部署和完成之前),我认为我可以从 GET 调用中获取更多数据,用于度量和监控......
原来,最后一切都搞定了,我只需要恢复返回函数,就全部连接成功了。

2。 API 网关设置和接线

这部分给我带来了麻烦。我认为 Auto Scaling API 连接并找到目标以便注册它是完全正确的(这就是您要创建的资源所做的)
@987654326 @ 是一个有效的 openapi.yaml 定义。
我建议把它放在一个像这样的文件中:

openapi.yaml.template

swagger: '2.0'
info:
  title: "${NAME}"
paths:
  '/scalableTargetDimensions/{scalableTargetDimensionId}':
    get:
      tags:
        - ScalableTargets
      x-tags:
        - tag: ScalableTargets
      security:
        - sigv4: []
      x-amazon-apigateway-any-method:
        produces:
          - application/json
        consumes:
          - application/json
      x-amazon-apigateway-integration:
        httpMethod: POST
        type: aws_proxy
        uri: ${INTEGRATION_URI}
        responses: {}
    patch:
      tags:
        - ScalableTargets
      x-tags:
        - tag: ScalableTargets
      security:
        - sigv4: []
      x-amazon-apigateway-any-method:
        security:
          - sigv4: []
        produces:
          - application/json
        consumes:
          - application/json
      x-amazon-apigateway-integration:
        httpMethod: POST
        type: aws_proxy
        uri: ${INTEGRATION_URI}
        responses: {}
securityDefinitions:
  sigv4:
    type: apiKey
    name: Authorization
    in: header
    x-amazon-apigateway-authtype: awsSigv4

然后你可以这样使用它:

# API Gateway
resource "aws_api_gateway_rest_api" "gateway" {
  name = var.rest_api_name
  body = templatefile("${path.module}/openapi.yaml.template",
    {
      NAME            = var.rest_api_name,
      INTEGRATION_URI = var.integration_uri
    }
  )
}

resource "aws_api_gateway_deployment" "gateway" {
  depends_on = [
    aws_api_gateway_rest_api.gateway,
  ]

  lifecycle {
    create_before_destroy = true
  }

  rest_api_id = aws_api_gateway_rest_api.gateway.id
  stage_name  = var.stage_name
}

3。正确的 IAM 权限

cloudformation 模板中定义的权限以及您可以使用 terraform 创建的权限不是精确匹配,并且似乎需要一些调整才能使集成工作...(我怀疑这与某些 AWS 魔法有关,但我发现转移到 terraform 总体上相对容易)
所以这是我最终创建的rolepolicies

# Lambda
resource "aws_lambda_function" "lambda" {
  # ...
  role = aws_iam_role.kinesis_autoscaler_lambda_role.arn
  # ...
}
resource "aws_iam_role" "kinesis_autoscaler_lambda_role" {
  name               = "${var.env}-kinesis-scaler-lambda-role"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_lambda_permission" "kinesis_api" {
  statement_id  = "AllowKinesisAPIInvoke"
  function_name = aws_lambda_function.lambda.function_name
  action        = "lambda:InvokeFunction"
  principal     = "apigateway.amazonaws.com"
  source_arn    = "${aws_api_gateway_deployment.gateway.execution_arn}/GET/scalableTargetDimensions/{scalableTargetDimensionId}"
}

resource "aws_lambda_permission" "kinesis_api_patch" {
  statement_id  = "AllowKinesisAPIPatchInvoke"
  function_name = aws_lambda_function.lambda.function_name
  action        = "lambda:InvokeFunction"
  principal     = "apigateway.amazonaws.com"
  source_arn    = "${aws_api_gateway_deployment.gateway.execution_arn}/PATCH/scalableTargetDimensions/{scalableTargetDimensionId}"
}

# Permissions
resource "aws_iam_policy" "lambda_access_stream" {
  name = "${var.stream_name}-access-stream-policy"
                                      
  policy = <<POLICY                 
{                           
  "Version": "2012-10-17",                                          
  "Statement": [                                              
    {                                               
      "Sid": "KinesisConsumerAccess",     
      "Effect": "Allow",                                  
      "Action": [                                     
        "kinesis:DescribeStreamConsumer"               
      ],                                               
      "Resource": "${aws_kinesis_stream.stream.arn}/consumer/*:*"
    },                                        
    {                                                                                                                  
      "Sid": "KinesisStreamAccess",      
      "Effect": "Allow",                                                  
      "Action": [                                             
        "kinesis:DescribeLimits",                               
        "kinesis:DescribeStream",                                                                                     
        "kinesis:DescribeStreamConsumer",               
        "kinesis:DescribeStreamSummary",                                                                               
        "kinesis:UpdateShardCount"                      
      ],                                                      
      "Resource": "${aws_kinesis_stream.stream.arn}" 
    },                               
    {                                                              
      "Sid": "SSMParameterStoreGet",                                                                                
      "Effect": "Allow",                                  
      "Action": [    
        "ssm:GetParameter"           
      ],                                           
      "Resource": [                      
        "${aws_ssm_parameter.number_of_shards.arn}",    
      ]                                                           
    },                                             
    {                                                     
      "Sid": "SSMParameterStorePut",                      
      "Effect": "Allow",                                                                                               
      "Action": [                            
        "ssm:PutParameter"                                    
      ],                                
      "Resource": [                                                         
        "${aws_ssm_parameter.number_of_shards.arn}"
      ]                          
    }                                  
  ]                                 
}                                    
POLICY                                
}                  
# I ended up splitting the policies for `module` reasons...                              
resource "aws_iam_policy_attachment" "attach_lambda_stream_access" {
  name = "${aws_iam_role.kinesis_autoscaler_lambda_role.name}_attach_lambda_stream_access"
  roles = [                                         
    aws_iam_role.kinesis_autoscaler_lambda_role.name                  
  ]                                                       
  policy_arn = aws_iam_policy.lambda_access_stream.arn
}       

最后一个很重要。这是我为使 lambda 工作所做的最后一次调整的结果。

resource "aws_iam_policy" "lambda_access_scaling" {
  name = "${var.stream_name}-lambda-access-scaling-policy"

  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "FindScalingPolicyARNAndAlarms",
      "Effect": "Allow",
      "Action": [
        "application-autoscaling:DescribeScalingPolicies",
        "cloudwatch:DescribeAlarms"
      ],
      "Resource": "*"
    },
    {
      "Sid": "UpdateAlarms",
      "Effect": "Allow",
      "Action": [
        "cloudwatch:PutMetricAlarm",
        "cloudwatch:DeleteAlarms"
      ],
      "Resource": [
        "${aws_cloudwatch_metric_alarm.alarm_out.arn}",
        "${aws_cloudwatch_metric_alarm.alarm_in.arn}"
      ]
    }
  ]
}
POLICY
}

resource "aws_iam_policy_attachment" "attach_scaling_access" {
  name = "${aws_iam_role.kinesis_autoscaler_lambda_role.name}_attach_scaling_access"
  roles = [
    aws_iam_role.kinesis_autoscaler_lambda_role.name
  ]
  policy_arn = aws_iam_policy.lambda_access_scaling.arn
}

另外,您希望拥有custom_appautoscaling...的正确权限...

# For reference:  
resource "aws_appautoscaling_target" "kinesis_stream" {
  min_capacity       = var.min_number_of_shard
  max_capacity       = var.max_number_of_shard
  resource_id        = "${aws_api_gateway_deployment.gateway.invoke_url}/scalableTargetDimensions/${var.stream_name}"           
  role_arn           = aws_iam_role.custom_appautoscaling_service_role.arn
  scalable_dimension = "custom-resource:ResourceType:Property"
  service_namespace  = "custom-resource"
                                                                   
  depends_on = [                                        
    aws_iam_policy_attachment.attach_base_policy,
  ]                                                     
                 
  lifecycle {                                        
    ignore_changes = [
      # This is because the "assume_role_policy" becomes the actual
      # Role "AWSServiceRoleForApplicationAutoScaling_CustomResource"                                               
      # at runtime and is always attemted to be recreated 
      role_arn,
    ]              
  }
}                         
          
# Actual policies:
 
resource "aws_iam_role" "custom_appautoscaling_service_role" {
  name               = "${var.stream_name}-assume-custom-resource"
  assume_role_policy = <<-EOF                   
  {        
    "Version": "2012-10-17",                            
    "Statement": [
      {                                      
        "Action": "sts:AssumeRole",
        "Principal": {
          "Service": "custom-resource.application-autoscaling.amazonaws.com"
        },                                       
        "Effect": "Allow",
        "Sid": ""  
      }
    ]                     
  }             
  EOF
}                           
                        
resource "aws_iam_policy" "base_policy" {
  name = "${var.stream_name}_base_policy"
                                 
  policy = <<POLICY
{                  
  "Version": "2012-10-17",                             
  "Statement": [                                     
    {  
      "Sid": "DescribeAlarms",
      "Effect": "Allow",
      "Action": [
        "cloudwatch:DescribeAlarms"
      ],
      "Resource": "*"
    },                                                             
    {                                                   
      "Sid": "InvokeApiGateway",
      "Effect": "Allow",                                
      "Action": [
        "execute-api:Invoke*"                        
      ],
      "Resource": [
        "${aws_api_gateway_deployment.gateway.execution_arn}/scalableTargetDimensions/${var.stream_name}"
      ]                                                   
    }
  ]                
}
POLICY                    
}               
     
resource "aws_iam_policy_attachment" "attach_base_policy" {
  name = "${var.stream_name}_attach_base_policy"
  roles = [      
    aws_iam_role.custom_appautoscaling_service_role.name  
  ]                                
  policy_arn = aws_iam_policy.base_policy.arn
}                    
      
resource "aws_iam_policy" "alarms_modification" {
  name = "${var.stream_name}_alarms_modification"
                        
  policy = <<POLICY
{                                   
  "Version": "2012-10-17",       
  "Statement": [
    {              
      "Sid": "UpdateAlarms",                           
      "Effect": "Allow",                             
      "Action": [
        "cloudwatch:PutMetricAlarm",
        "cloudwatch:DeleteAlarms"
      ],
      "Resource": [
        "${aws_cloudwatch_metric_alarm.alarm_out.arn}",
        "${aws_cloudwatch_metric_alarm.alarm_in.arn}"
      ]                                                       
    }                                                   
  ]        
}                       
POLICY
}                                                      
 
resource "aws_iam_policy_attachment" "attach_alarms_modification" {                        
  name = "${var.stream_name}_attach_alarms_modification"
  roles = [                                            
    aws_iam_role.custom_appautoscaling_service_role.name
  ]
  policy_arn = aws_iam_policy.alarms_modification.arn
}              

4。 Lambda 和 APIGateway 工作正常

最后,如果所有这些仍然不起作用,您可能需要对 lambda 和 APIGateway 进行故障排除...
我最终在 lambda 控制台中使用了 apigateway-aws-proxy Event Template。稍微编辑一下字段,使其看起来像这样:

{
  "body": {
    "desiredCapacity": "1"
  },
  "resource": "/{proxy+}",
  "path": "/scalableTargetDimensions/my-custom-stream",
  "httpMethod": "PATCH",
...
  "requestContext" {
    ...
    "path": "/<STAGE_NAME_SEE_API_GATEWAY_DEPLOYMENT_RESOURCE>/scalableTargetDimensions/my-custom-stream",
    "resourcePath": "/{proxy+}",
    "httpMethod": "PATCH",
}

对于这一部分,我还创建了一个具有base64encoded 主体的内容(因为我猜它是通过 APIGateway 传递的方式?)...
无论如何,我最终对 lambda 进行了微调,以便它同时接受两者,这样就可以知道 lambda 是否真的正常工作。

{
  "body": "eyJkZXNpcmVkQ2FwYWNpdHkiOiIyIn0=",
  "resource": "/{proxy+}",
  "path": "/scalableTargetDimensions/my-custom-stream",
  "httpMethod": "PATCH",
  "isBase64Encoded": true,
...

有了这个,您应该能够对 lambda 进行故障排除,确保正确授予所有权限。
要对 APIGateway 进行故障排除,您始终可以使用 postman。它可以很好地处理对 AWS 的身份验证,因此如果您有足够访问您创建的 APIGateway 资源的凭据,您应该能够执行一些 GETPATCH 来手动触发 APIGateway 并测试它集成的一部分。

【讨论】:

  • 首先,感谢您的付出!我已经设法通过使用 CloudFormation 而不是 Terraform 克服了这个问题,我认为我遇到的问题是因为您在回答中提到的 IAM 权限。我要试试你的建议。但是,为了您的努力和回答质量,我已经悬赏了这个问题,我会在超时到期时给您!干得好!
猜你喜欢
  • 1970-01-01
  • 2020-02-27
  • 2020-11-08
  • 2021-10-08
  • 1970-01-01
  • 1970-01-01
  • 2018-06-09
  • 2021-04-27
  • 1970-01-01
相关资源
最近更新 更多