如何为 K8s 上的 Elasticsearch 部署应用配置？

【问题标题】：How to apply config for Elasticsearch deploy on K8s?如何为 K8s 上的 Elasticsearch 部署应用配置？
【发布时间】：2020-09-15 04:00:47
【问题描述】：

我在更改部署在 K8s 中的 Elasticsearch 配置时遇到了麻烦。 我想将此配置应用于我的 Elasticsearch 节点

# Force all memory to be locked, forcing the JVM to never swap
bootstrap.mlockall: true

## Threadpool Settings ##

# Search pool
threadpool.search.type: fixed
threadpool.search.size: 20
threadpool.search.queue_size: 100

# Bulk pool
threadpool.bulk.type: fixed
threadpool.bulk.size: 60
threadpool.bulk.queue_size: 300

# Index pool
threadpool.index.type: fixed
threadpool.index.size: 20
threadpool.index.queue_size: 100

 # Indices settings
indices.memory.index_buffer_size: 30%
indices.memory.min_shard_index_buffer_size: 12mb
indices.memory.min_index_buffer_size: 96mb

# Cache Sizes
indices.fielddata.cache.size: 15%
indices.fielddata.cache.expire: 6h
indices.cache.filter.size: 15%
indices.cache.filter.expire: 6h

# Indexing Settings for Writes
index.refresh_interval: 30s
index.translog.flush_threshold_ops: 50000

要在 K8s 中部署 Elasticsearch，我遵循两个步骤： 第 1 步：创建 .yml 文件，像这样

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: esc-lamtest
  namespace: lamtest
spec:
  version: 7.9.0
  nodeSets:
    - name: basic-1
    count: 3
    config:
        node.master: true
        node.data: true
        node.ingest: true
        # Search pool
        thread_pool.search.queue_size: 50
        thread_pool.search.size: 20
        thread_pool.search.min_queue_size: 10
        thread_pool.search.max_queue_size: 100  
volumeClaimTemplates:
- metadata:
    name: elasticsearch-data
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 150Gi
    storageClassName: vnptit-nfs
podTemplate:
  spec:
    containers:
    - name: elasticsearch
      env:
      - name: ES_JAVA_OPTS
        value: -Xms4g -Xmx4g
      resources:
        requests:
            memory: 0Gi
            cpu: 0Gi
        limits:
            memory: 0Gi
            cpu: 0Gi

http: tls： 自签名证书： 禁用：真

第 2 步：我运行命令行：

kubectl apply -f lamtest.yaml

但我只能为“搜索池”配置应用程序。 当我为 # Bulk pool # Index pool # Indices settings # Cache Sizes and # Indexing Settings for Writes 应用配置时，我的 Elasticsearch 失败，这是我的日志

"Suppressed: java.lang.IllegalArgumentException: unknown setting [thread_pool.bulk.queue_size] did   you mean any of [thread_pool.get.queue_size, thread_pool.write.queue_size, thread_pool.analyze.queue_size, thread_pool.search.queue_size, thread_pool.listener.queue_size]?", 
"\tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:431) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.node.Node.<init>(Node.java:385) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "Suppressed: java.lang.IllegalArgumentException: unknown setting [thread_pool.bulk.size] did you mean any of [thread_pool.get.size, thread_pool.write.size, thread_pool.analyze.size, thread_pool.search.size]?", 
 "\tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:431) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.node.Node.<init>(Node.java:385) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.9.0.jar:7.9.0]"] } 
 uncaught exception in thread [main] 
 java.lang.IllegalArgumentException: unknown setting [thread_pool.bulk.type] please check that any  required plugins are installed, or check the breaking changes documentation for removed settings 
 at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) 
 at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) 
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) 
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:431) 
at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149) 
at org.elasticsearch.node.Node.<init>(Node.java:385) 
at org.elasticsearch.node.Node.<init>(Node.java:277) 
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) 
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) 
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) 
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) 
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) 
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) 
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) 
at org.elasticsearch.cli.Command.main(Command.java:90) 
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) 
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)

有关完整的错误详细信息，请参阅 /usr/share/elasticsearch/logs/esc-lamtest.log 中的日志

【问题讨论】：

  • 感谢您接受这个问题，如果您也支持它，那就太好了:) 提前致谢

标签： elasticsearch


【解决方案1】：

错误信息非常清楚，您没有为其他线程池定义正确的配置，请仔细注意错误消息的第一行。

“抑制：java.lang.IllegalArgumentException：未知设置 [thread_pool.bulk.queue_size] 你的意思是 [thread_pool.get.queue_size，thread_pool.write.queue_size， thread_pool.analyze.queue_size，thread_pool.search.queue_size， thread_pool.listener.queue_size]?",

您已将批量队列大小定义为thread_pool.bulk.queue_size，这是不正确的，您可以阅读threadpools in ES，没有bulk 的线程池，它使用write 线程池作为@987654325 @ 请求，来自同一个文档，因此将其更改为 thread_pool.write.queue_size 将适用于此配置。

write 用于单文档索引/删除/更新和批量请求。 线程池类型是固定的，大小为分配的处理器数， queue_size 为 10000。此池的最大大小为 1 + # of 分配的处理器。

现在，为了解决 bulk、Index 和其他设置的问题，请确认您为其配置使用了正确的名称，threadpools 可以从任何运行 ES 实例，即可轻松构建相应的配置。

http://localhost:9200/_nodes/stats?pretty 给出了所有的threadpools，下面列出了一些。

"thread_pool": {
"analyze": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 1,
"completed": 1
},
"ccr": {
"threads": 0,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 0,
"completed": 0
},
}

【讨论】：

    猜你喜欢
    • 2022-01-23
    • 1970-01-01
    • 2019-04-03
    • 2020-01-07
    • 2020-07-07
    • 2023-03-06
    • 2021-04-23
    • 1970-01-01
    • 2019-06-22
    相关资源
    最近更新 更多
    热门标签
    Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode