【问题标题】:Problem with Elasticsearch Index Lifecycle Policy that doesn't rollover不翻转的 Elasticsearch 索引生命周期策略问题
【发布时间】:2020-12-29 11:36:08
【问题描述】:

为了评估它对我们日常运营的帮助潜力,我部署了 Elastic Search 和 Kibana(7.7.1,带有 BASIC 许可证)并为 Ntopng(我们的监控平台)创建了一个索引模板。

由于索引不断增长,我想删除超过 20 天左右的 Ntopng 索引,因此我创建了一个名为 ntopng 的生命周期策略,其中时间戳索引应在 1 天后翻转(用于测试目的),然后将翻转 2 天后被删除:

接下来我选择了当天创建的时间戳索引并将生命周期策略应用到它:

在此之前,我必须为该索引创建一个别名,所以我手动完成了:

POST /_aliases
{
  "actions" : [
    { "add" : { "index" : "ntopng-2020.09.09", "alias" : "ntopng_Alias" } }
  ]
}

在那之后一切看起来都很好(我猜),因为没有显示错误或警报:

"indices" : {
    "ntopng-2020.09.09" : {
      "index" : "ntopng-2020.09.09",
      "managed" : true,
      "policy" : "ntopng",
      "lifecycle_date_millis" : 1599609600433,
      "age" : "20.14h",
      "phase" : "hot",
      "phase_time_millis" : 1599681721821,
      "action" : "rollover",
      "action_time_millis" : 1599680521920,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1599681721821,
      "is_auto_retryable_error" : true,
      "failed_step_retry_count" : 1,
      "phase_execution" : {
        "policy" : "ntopng",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_age" : "1d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "version" : 4,
        "modified_date_in_millis" : 1599509572867
      }
    }

我的预期是,在第二天,该策略将自动滚动到下一个索引 (ntopng-2020.10.10),以便最终在接下来的两天删除初始索引。

相反,我收到以下错误:

GET ntopng-*/_ilm/explain
{
  "indices" : {
    "ntopng-2020.09.09" : {
      "index" : "ntopng-2020.09.09",
      "managed" : true,
      "policy" : "ntopng",
      "lifecycle_date_millis" : 1599609600433,
      "age" : "1.94d",
      "phase" : "hot",
      "phase_time_millis" : 1599776521822,
      "action" : "rollover",
      "action_time_millis" : 1599680521920,
      "step" : "ERROR",
      "step_time_millis" : 1599777121822,
      "failed_step" : "check-rollover-ready",
      "is_auto_retryable_error" : true,
      "failed_step_retry_count" : 80,
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : """index name [ntopng-2020.09.09] does not match pattern '^.*-\d+$'""",
        "stack_trace" : """java.lang.IllegalArgumentException: index name [ntopng-2020.09.09] does not match pattern '^.*-\d+$'
    at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.generateRolloverIndexName(TransportRolloverAction.java:241)
    at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.masterOperation(TransportRolloverAction.java:133)
    at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.masterOperation(TransportRolloverAction.java:73)
    at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:170)
    at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
    at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:170)
    at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:133)
    at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:110)
    at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:59)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:153)
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:123)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:151)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:129)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64)
    at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83)
    at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:399)
    at org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:92)
    at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:155)
    at org.elasticsearch.xpack.ilm.LifecyclePolicySecurityClient.doExecute(LifecyclePolicySecurityClient.java:51)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:399)
    at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1234)
    at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.rolloverIndex(AbstractClient.java:1736)
    at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:127)
    at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:173)
    at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:329)
    at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:267)
    at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183)
    at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:211)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
    at java.base/java.lang.Thread.run(Thread.java:832)
"""
      },
      "phase_execution" : {
        "policy" : "ntopng",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_age" : "1d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "version" : 4,
        "modified_date_in_millis" : 1599509572867
      }
    }


  "ntopng-2020.09.10" : {
      "index" : "ntopng-2020.09.10",
      "managed" : true,
      "policy" : "ntopng",
      "lifecycle_date_millis" : 1599696000991,
      "age" : "22.57h",
      "phase" : "hot",
      "phase_time_millis" : 1599776521844,
      "action" : "rollover",
      "action_time_millis" : 1599696122033,
      "step" : "ERROR",
      "step_time_millis" : 1599777121839,
      "failed_step" : "check-rollover-ready",
      "is_auto_retryable_error" : true,
      "failed_step_retry_count" : 67,
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : "index.lifecycle.rollover_alias [ntopng_Alias] does not point to index [ntopng-2020.09.10]",
        "stack_trace" : """java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [ntopng_Alias] does not point to index [ntopng-2020.09.10]
    at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:104)
    at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:173)
    at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:329)
    at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:267)
    at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183)
    at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:211)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
    at java.base/java.lang.Thread.run(Thread.java:832)
"""
      },
      "phase_execution" : {
        "policy" : "ntopng",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_age" : "1d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        }

第一个索引错误显示“索引名称 [ntopng-2020.09.09] 不匹配模式 '^.*-\d+$”

虽然第二个显示:“”index.lifecycle.rollover_alias [ntopng_Alias] 不指向索引 [ntopng-2020.09.10]”

请注意,我正在学习 ES 索引管理的基础知识,因此我希望能提供任何有关问题所在的线索。

【问题讨论】:

    标签: elasticsearch kibana elastic-stack elk


    【解决方案1】:

    好的,我刚刚发现索引名称必须以 0001 之类的数字模式结尾,而不是 2020.09.09 所以我可能需要找到另一种方法来使它工作。

    【讨论】:

    • 每日索引并不理想,因为它们会给您留下大小不同且可能太小或太大的碎片。它总是取决于几个参数,但默认大小是每个分片 50GB 是有原因的——你的 2.2 到 2.9GB 可能并不理想,它们很有可能会随着时间而改变。
    • 我遇到了同样的错误,但我只想自动化翻转。到目前为止,我在这里和多个地方阅读到自动化所需的后缀是不可能的?
    【解决方案2】:

    根据 kibana 正则表达式,您也可以使用日期和时间模式,而不是 2020.01.01 使用 2020-01-01

    这应该也可以。你可以在这里查看正则表达式:https://regex101.com/r/VclptX/1

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 2020-12-01
      • 2021-05-14
      • 2020-11-17
      • 1970-01-01
      • 2023-01-02
      • 2021-06-29
      • 2018-12-24
      • 1970-01-01
      相关资源
      最近更新 更多