logstash 输出带有序列号的弹性搜索索引答案

【问题标题】：logstash output elasticsearch index with sequence numberlogstash 输出带有序列号的弹性搜索索引
【发布时间】：2022-01-18 07:41:01
【问题描述】：

我正在使用带有 Logstash 7.10 的 AWS Elastic Search（版本 7.10）。目的是将内容从 logstash 发送到弹性搜索，并使用策略在特定大小或时间后翻转索引。

policy: {
    "policy_id": "Rollover_Policy",
    "description": "roller index",
    "last_updated_time": 1634910129219,
    "schema_version": 1,
    "error_notification": null,
    "default_state": "hot",
    "states": [
        {
            "name": "hot",
            "actions": [
                {
                    "rollover": {
                        "min_size": "1mb"
                    }
                }
            ],
            "transitions": [
                {
                    "state_name": "warm"
                }
            ]
        },
        {
            "name": "warm",
            "actions": [
                {
                    "replica_count": {
                        "number_of_replicas": 1
                    }
                }
            ],
            "transitions": [
                {
                    "state_name": "delete",
                    "conditions": {
                        "min_index_age": "1h"
                    }
                }
            ]
        },
        {
            "name": "delete",
            "actions": [
                {
                    "delete": {}
                }
            ],
            "transitions": []
        }
    ],
    "ism_template": [
        {
            "index_patterns": [
                "products*"
            ],
            "priority": 100,
            "last_updated_time": 1634910129219
        }
    ]
}

虽然我尝试在 logstash 输出插件中将 ilm_enabled 设置为 true，但它无法连接弹性搜索 xpack API。

注意：AWS 弹性搜索不支持 xpack 和 ILM。

elasticsearch {  
        hosts => "${elasticsearch_endpoint}"
        user => "${elasticsearch_user}"
        password => "${elasticsearch_password}"
        ilm_enabled => true
        ilm_rollover_alias => "products"
        ilm_pattern => "{now/d}-000001"
        ilm_policy => "Rollover_Policy"
}

所以我已将 ilm_enabled 标志更改为 false 并尝试以下选项

elasticsearch {
        hosts => "${elasticsearch_endpoint}"
        user => "${elasticsearch_user}"
        password => "${elasticsearch_password}"
        ilm_enabled => false
        index => "products-%{+YYYY.MM.dd}-000001"
}

现在的问题是，即使在翻转之后，logstash 仍然将文档发送到 001 索引而不是新索引。如果我不在索引名称中给出 -000001，则翻转失败。

【问题讨论】：

标签： amazon-web-services elasticsearch logstash

【解决方案1】：

在弹性中使用以下 REST 请求创建索引。由于索引名称具有日期模式，翻转将创建具有当前日期的新索引。

PUT %3Cproducts-%7Bnow%2Fd%7D-000001%3E
{
  "settings":{
    "number_of_shards":1,
    "number_of_replicas":1
  },
  "aliases": {
    "products":  {
      "is_write_index": true
    }
  }

为索引模式创建一个模板以及翻转别名

PUT _index_template/products_logs
{
  "index_patterns": [
    "products*"
  ],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "opendistro": {
        "index_state_management": {
          "rollover_alias": "products"
        }
      }
    }
  }
}

在 logstash 输出插件中提供以下详细信息以将数据发送到弹性搜索

elasticsearch {  
        hosts => "${elasticsearch_endpoint}"
        user => "${elasticsearch_user}"
        password => "${elasticsearch_password}"
        ilm_enabled => false 
        index => "products"
}

注意：索引名代表索引的别名。

【讨论】：