如何更新映射答案

【问题标题】：How to update Mapping如何更新映射
【发布时间】：2018-10-08 22:29:51
【问题描述】：

如何更新索引映射以包含以下字段 doc_as_upsert : true

我从 s3 摄取 cloudtrail 日志的 logstash 在日志上显示以下内容 Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"cloudtrail-2018.10.08", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x251f932>], :response=>{"index"=>{"_index"=>"cloudtrail-2018.10.08", "_type"=>"doc", "_id"=>"t2mmVWYBVQr-RbWuAQIS", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [requestParameters.disableApiTermination]", "caused_by"=>{"type"=>"json_parse_exception", "reason"=>"Current token (START_OBJECT) not of boolean type\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@133a6c; line: 1, column: 1509]"}}}}}

映射是动态的，而且很长，所以不能在这里全部放好，但这是我最适合的

{
  "cloudtrail-2018.10.08": {
    "mappings": {
      "_default_": {
        "dynamic_templates": [
          {
            "message_field": {
              "path_match": "message",
              "match_mapping_type": "string",
              "mapping": {
                "norms": false,
                "type": "text"
              }
            }
          },
          {
            "string_fields": {
              "match": "*",
              "match_mapping_type": "string",
              "mapping": {
                "fields": {
                  "keyword": {
                    "ignore_above": 256,
                    "type": "keyword"
                  }
                },
                "norms": false,
                "type": "text"
              }
            }
          }
        ],
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "keyword"
          },
          "geoip": {
            "dynamic": "true",
            "properties": {
              "ip": {
                "type": "ip"
              },
              "latitude": {
                "type": "half_float"
              },
              "location": {
                "type": "geo_point"
              },
              "longitude": {
                "type": "half_float"
              }
            }
          }
        }
      },
      "doc": {
        "dynamic_templates": [
          {
            "message_field": {
              "path_match": "message",
              "match_mapping_type": "string",
              "mapping": {
                "norms": false,
                "type": "text"
              }
            }
          },
          {
            "string_fields": {
              "match": "*",
              "match_mapping_type": "string",
              "mapping": {
                "fields": {
                  "keyword": {
                    "ignore_above": 256,
                    "type": "keyword"
                  }
                },
                "norms": false,
                "type": "text"
              }
            }
          }
        ],
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "keyword"
          },
          "additionalEventData": {
            "properties": {
              "configRuleArn": {
                "type": "text",
                "norms": false,
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "configRuleInputParameters": {
                "type": "text",
                "norms": false,
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "configRuleName": {
                "type": "text",
                "norms": false,
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "managedRuleIdentifier": {
                "type": "text",
                "norms": false,
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "notificationJobType": {
                "type": "text",
                "norms": false,
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "vpcEndpointId": {
                "type": "text",
                "norms": false,
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "apiVersion": {
            "type": "date"
          },
          "awsRegion": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "errorCode": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "errorMessage": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "eventID": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "eventName": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "eventSource": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "eventType": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "eventVersion": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }

当我尝试使用这些更新映射时出现以下错误

PUT cloudtrail-*/_mapping/_doc
{
  "properties": {
    "doc_as_upsert": true
    }
  }

我得到错误

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "Expected map for property [fields] on field [doc_as_upsert] but got a class java.lang.String"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "Expected map for property [fields] on field [doc_as_upsert] but got a class java.lang.String"
  },
  "status": 400
}

【问题讨论】：

您看到的错误表明 Elasticsearch 无法解析您发送的 JSON。可能是您以它不期望的格式发送数据。如果可能，请发布您的映射和 ES 抱怨的 JSON 文档。
感谢问题中的更新信息。您是否希望 Elasticsearch 在默认情况下对您发送用于索引的任何文档进行“更新插入”？恐怕这做不到。如下图所示，doc_as_upsert 是_update API 的属性，而不是PUT _mapping 的属性。如果我没听懂，请回复。

标签： elasticsearch logstash kibana-6 amazon-cloudtrail

【解决方案1】：

doc_as_upsert 是一个标志，用于告诉 elasticsearch 你想用 doc 的内容作为 upsert 值来更新文档。它与更新索引映射无关。

假设您要更新 id 为 1 的文档并索引 test（更新名称）。

POST test/_doc/1/_update
{
    "doc" : {
    "name" : "new_name"
    },
"doc_as_upsert" : true
}

【讨论】：

以下也可以吗？ ` PUT cloudtrail-*/_mapping/_doc { "properties": { "doc_as_upsert": true } } } `
在映射中指定字段的数据类型，以便 Elasticsearch 能够正确索引该字段。所以不要在映射中指定值，只需将数据类型设置为“布尔值”即可。
仍然出现错误，这是我运行的 ` PUT cloudtrail-*/_mapping/_doc { "properties": { "doc_as_upsert": { "type": "true" } } } ` 和错误` { "error": { "root_cause": [ { "type": "mapper_parsing_exception", "reason": "没有在字段 [doc_as_upsert] 上声明的类型 [true] 的处理程序" } ], "type": "mapper_parsing_exception" ，“原因”：“没有在字段 [doc_as_upsert] 上声明的类型 [true] 的处理程序”}，“状态”：400 } `
我以为您想添加一个新字段并插入其映射。我编辑答案。这就是 doc_as_upsert 的工作方式。如果你还想要什么，请告诉我。