Elasticsearch _update_by_query in logstash error [HTTP Output Failure] Encountered non-2xx HTTP code 400答案

【问题标题】：Elasticsearch _update_by_query in logstash error [HTTP Output Failure] Encountered non-2xx HTTP code 400Elasticsearch _update_by_query in logstash error [HTTP Output Failure] Encountered non-2xx HTTP code 400
【发布时间】：2021-06-05 17:41:07
【问题描述】：

工作 _update_by_query 调用 -

POST /s1test-demo7/_update_by_query
{
  "script": {
      "source": "ctx._source.externaldata = params.externaldata",
      "lang": "painless",
      "params": {
        "externaldata":{
          "field1": "1",
          "field2": "abc"  
        }
      }
    },
  "query": {
    "bool": {
      "must": [
        { "match": { "h.req-id": "Test9GrpA"} },
        { "match": { "h.process-code": "DemoS99"} }
      ]
    }
  }
}

此 API 实际上在匹配现有文档的查询中正确地添加了带有 2 个内部字段的“externaldata”字段作为嵌套 json。
我需要有关 logstash 管道的帮助来做同样的事情 -

input {
    file {
        id => "updatedata"
        path => "D:/p3-test1.json"
        start_position => "beginning"
        sincedb_path => "D:/sdb/p3-test1.sdb"
    }
}
filter {
    json {
        source => "message"
    }
    mutate {
        add_field => {
            "[script][lang]" => "painless"
            "[script][source]" => "ctx._source.externaldata = params.externaldata"
            "[script][params][externaldata][field1]" => "%{field1}"
            "[script][params][externaldata][field2]" => "%{field2}"
            "[query][bool][must][match][h.req-id]" => "%{req-id}"
            "[query][bool][must][match][h.process-code]" => "%{process-code}"
        }
    }
}
output {
    stdout {
        codec => rubydebug
    }
    http {
        url => "http://localhost:9200/s1test-demo7/_update_by_query"
        headers => { "Authorization" => "Basic ZWxhc3RpYzplbGFzdGlj" }
        http_method => "post"
        format => "json"
    }
}

logstash 管道必须包含身份验证，否则我会收到 HTTP 错误 401。但我不确定这里的语法是否正确。文档和 elasticsearch 论坛帖子在 http 输出插件标头中有所不同。
Logstash 输出 -

[2021-05-29T20:14:13,226][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
            "host" => "mypc",
        "@version" => "1",
          "req-id" => "Test9GrpA",
          "field2" => "default",
      "@timestamp" => 2021-05-29T14:44:13.521Z,
          "field1" => "1",
           "query" => {
        "bool" => {
            "must" => {
                "match" => {
                    "h.process-code" => "DemoS99",
                          "h.req-id" => "Test9GrpA"
                }
            }
        }
    },
         "message" => "{\"req-id\":\"Test9GrpA\",\"process-code\":\"DemoS99\",\"field1\":\"1\",\"field2\":\"default\"}\r",
    "process-code" => "DemoS99",
          "script" => {
          "lang" => "painless",
        "params" => {
            "externaldata" => {
                "field2" => "default",
                "field1" => "1"
            }
        },
        "source" => "ctx._source.externaldata = params.externaldata"
    },
            "path" => "D:/p3-test1.json"
}
[2021-05-29T20:14:14,268][ERROR][logstash.outputs.http    ][main][2bc46338fec26d73c819043dac159f1d12397fefc800c84c7d2e0f0d16b278c2] [HTTP Output Failure] Encountered non-2xx HTTP code 400 {:response_code=>400, :url=>"http://localhost:9200/s1test-demo7/_update_by_query", :event=>#<LogStash::Event:0x2badd231>}

请帮忙。

【问题讨论】：

标签： elasticsearch logstash logstash-configuration

【解决方案1】：

可以将脚本部分转移到“http”输出插件，而不是在过滤器部分使用“painless”脚本。

output {
    stdout {
        codec => rubydebug
    }
    if "externallogs" in [tags] {
        http {
            url => "http://localhost:9200/s1test-demo7/_update_by_query"
            headers => { "Authorization" => "Basic ZWxhc3RpYzplbGFzdGlj" }
            http_method => "post"
            format => "message"
            content_type => "application/json"
            message => '{  
                    "script": {
                      "source": "ctx._source.externaldata = params.externaldata",
                      "lang": "painless",
                      "params": {
                        "externaldata":{
                          "field1": "%{field1}",
                          "field2": "%{field2}"  
                        }
                      }
                    },
                  "query": {
                    "bool": {
                      "must": [
                        { "match": { "h.req-id": "%{req-id}"} },
                        { "match": { "h.process-code": "%{process-code}"} }
                      ]
                    }
                  }
                }'
        }
    }
}

【讨论】：