从 Lambda 创建 Elasticsearch 映射时出现“授权异常”

Question

我根本无法启用我的 Lambda 函数来连接以在我的 AWS Elasticsearch 服务器上创建映射。我正在使用 Node 和 elasticsearch 包，我收到以下错误（在 CloudWatch 日志中）：

ERROR   Uncaught Exception
{
    "errorType": "Error [ERR_UNHANDLED_ERROR]",
    "errorMessage": "Unhandled error. ({ message: 'Unable to create Elasticsearch mapping for Log',\n  data:\n   'Error: Authorization Exception\\n    at respond (/var/task/node_modules/elasticsearch/src/lib/transport.js:308:15)\\n    at checkRespForFailure (/var/task/node_modules/elasticsearch/src/lib/transport.js:267:7)\\n    at done (/var/task/node_modules/http-aws-es/connector.js:48:7)\\n    at IncomingMessage.cleanUp (/var/task/node_modules/http-aws-es/src/node.js:20:7)\\n    at IncomingMessage.emit (events.js:203:15)\\n    at IncomingMessage.EventEmitter.emit (domain.js:448:20)\\n    at endReadableNT (_stream_readable.js:1129:12)\\n    at process._tickCallback (internal/process/next_tick.js:63:19)' })",
    "code": "ERR_UNHANDLED_ERROR",
    "stack": [
        "Error [ERR_UNHANDLED_ERROR]: Unhandled error. ({ message: 'Unable to create Elasticsearch mapping for Log',",
        "  data:",
        "   'Error: Authorization Exception\\n    at respond (/var/task/node_modules/elasticsearch/src/lib/transport.js:308:15)\\n    at checkRespForFailure (/var/task/node_modules/elasticsearch/src/lib/transport.js:267:7)\\n    at done (/var/task/node_modules/http-aws-es/connector.js:48:7)\\n    at IncomingMessage.cleanUp (/var/task/node_modules/http-aws-es/src/node.js:20:7)\\n    at IncomingMessage.emit (events.js:203:15)\\n    at IncomingMessage.EventEmitter.emit (domain.js:448:20)\\n    at endReadableNT (_stream_readable.js:1129:12)\\n    at process._tickCallback (internal/process/next_tick.js:63:19)' })",
        "    at Object.emit (events.js:187:17)",
        "    at limiter.removeTokens (/var/task/node_modules/@my-company/my-package/lib/logger.js:158:10)",
        "    at afterTokensRemoved (/var/task/node_modules/limiter/lib/rateLimiter.js:87:7)",
        "    at process._tickCallback (internal/process/next_tick.js:61:11)"
    ],
    "context": {
        "message": "Unable to create Elasticsearch mapping for Log",
        "data": "Error: Authorization Exception\n    at respond (/var/task/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/var/task/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at done (/var/task/node_modules/http-aws-es/connector.js:48:7)\n    at IncomingMessage.cleanUp (/var/task/node_modules/http-aws-es/src/node.js:20:7)\n    at IncomingMessage.emit (events.js:203:15)\n    at IncomingMessage.EventEmitter.emit (domain.js:448:20)\n    at endReadableNT (_stream_readable.js:1129:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"
    }
}

似乎 Lambda 能够连接到 Elasticsearch 服务器，但它收到了授权异常错误（我相信是 403）。

我正在使用无服务器框架。我的无服务器配置如下所示：

provider:
  name: aws
  runtime: nodejs10.x
  stage: ${opt:stage, 'dev'}
  region: us-east-2
  memorySize: 512
  timeout: 30
  vpc:
    securityGroupIds:
      - ${ssm:/my-company/security-group/lambda}
    subnetIds:
      - ${ssm:/my-company/subnet/lambda1/id}
      - ${ssm:/my-company/subnet/lambda2/id}
      - ${ssm:/my-company/subnet/lambda3/id}

functions:
  myFunctionName:
    handler: src/my-function-name.handler
    events:
      - sqs:
          arn: ${ssm:/my-company/${self:provider.stage}/update-print-image-status-queue-arn}

package:
  exclude:
    - .circleci/**
    - .terraform/**
    - test/**

Lambda 使用的 IAM 角色拥有完整的 Elasticsearch 访问权限：

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "sqs:DeleteMessage",
                "logs:CreateLogStream",
                "sqs:ReceiveMessage",
                "sqs:GetQueueAttributes"
            ],
            "Resource": [
                "arn:aws:logs:us-east-2:<AWS account ID>:log-group:/aws/lambda/<project-name>-dev*:*",
                "arn:aws:sqs:us-east-2:<AWS account ID>:update-print-image-status-queue-dev"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "logs:PutLogEvents",
            "Resource": "arn:aws:logs:us-east-2:<AWS account ID>:log-group:/aws/lambda/<project-name>-dev*:*:*"
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": "es:*",
            "Resource": "*"
        }
    ]
}

VPC 和子网肯定存在，并且预期的子网与 Lambda 相关联。 Lambda 能够毫无问题地访问和查询我的 MongoDB EC2 服务器。 Elasticsearch 服务器与 MongoDB EC2 服务器位于同一 VPC 和子网中。

这是我从 Node 初始化 ES 客户端的方式：

const AWS = require('aws-sdk');
const elasticsearch = require('elasticsearch');

client = elasticsearch.Client({
  host: process.env.ELASTICSEARCH_HOSTS,
  connectionClass: require('http-aws-es'),
  awsConfig: new AWS.Config({
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    region: process.env.ELASTICSEARCH_REGION,
  })
});

ELASTICSEARCH_HOSTS 和 ELASTICSEARCH_REGION 都被定义为具有正确值的 Lambda 环境变量。

客户端随后被 mongoosastic 使用。

同样的代码在 EC2 服务器（与 MongoDB EC2 服务器位于相同的 VPC 和子网中）上运行良好，无需任何更改。

Elasticsearch 服务（由 AWS 管理；版本 6.3）具有以下策略：

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-2:<aws account ID>:domain/<my domain>/*"
    }
  ]
}

所以，我很困惑为什么我会看到“授权异常”。有什么想法吗？

Answer 1

我相信我已经解决了这个问题。以下作品：

const AWS = require('aws-sdk');
const elasticsearch = require('elasticsearch');

client = elasticsearch.Client({
  host: process.env.ELASTICSEARCH_HOSTS,
  connectionClass: require('http-aws-es')
});

无需指定访问密钥。我有弹性搜索服务器的安全组设置，因此 Lambda 安全组具有入口访问权限。

Answer 2

• 我没有使用 node 完成此操作，但我在 python 中有一个工作代码希望对您有所帮助
• 我正在使用 AWS4Auth，但我遇到了类似的身份验证问题。
• 节点示例https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-request-signing.html#es-request-signing-node

    import gitlab
    import logging
    from elasticsearch import Elasticsearch, RequestsHttpConnection
    from requests_aws4auth import AWS4Auth
    import boto3
    #from aws_requests_auth.aws_auth import AWSRequestsAuth

    LOGGER = logging.getLogger()
    ES_HOST = {'host':'search-testelasticsearch-xxxxxxxxx.eu-west-2.es.amazonaws.com', 'port': 443}


    def lambda_handler(event, context):
        LOGGER.info('started')
        gl = gitlab.Gitlab('xxxxxxx', private_token='xxxxxx')
        group = gl.groups.get('Thunderbird')
        project = gl.projects.get(92, lazy=True)
        mrs = project.mergerequests.list(state='opened', order_by='updated_at')
        first_mr = mrs[0]
        dump = {
          'title': first_mr.title,
          'state': first_mr.state,
          'name': first_mr.author.get('name'),
          'comments': first_mr.user_notes_count
        }

        dump2={
            'number_of_mrs': 9
        }

        service = 'es'
        credentials = boto3.Session().get_credentials()
        awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, "eu-west-2", service, session_token=credentials.token)
        es = Elasticsearch(hosts=[ES_HOST], http_auth = awsauth, use_ssl = True, verify_certs = True, connection_class = RequestsHttpConnection)
        DAVID_INDEX = 'david_test_index'
        response = es.index(index=DAVID_INDEX, doc_type='is_this_important?', body=dump2, id='4')```