Elasticsearch 多节点集群一个节点总是因 docker compose 而失败答案

【问题标题】：Elasticsearch multi-node cluster one node always fails with docker composeElasticsearch 多节点集群一个节点总是因 docker compose 而失败
【发布时间】：2020-09-12 08:38:03
【问题描述】：

我正在尝试按照官方文档在 docker compose 中设置一个基本的 dev 3 node 7.7.0 ES 集群 here 但无法让所有 3 个节点同时运行。在运行docker-compose up 后，至少有一个容器会立即退出，或者在不久之后退出，如下所示：

Starting es01 ... done
Starting es03 ... done
Starting es02 ... done
Attaching to es03, es02, es01
es01 exited with code 137

如果我尝试通过运行 docker-compose start es01（或任何一个退出，有时是随机的）来恢复它，它会导致这些错误在我杀死所有容器之前不会停止：

es03    | {"type": "server", "timestamp": "2020-05-25T15:52:37,214Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "es-docker-cluster", "node.name": "es03", "message": "master not discovered or elected yet, an election requires 2 nodes with ids [9RNLWSMbTmetFFh3Tm1q0g, 3CbHK5iBQAqt7poPRsMmaw], have discovered [{es03}{3CbHK5iBQAqt7poPRsMmaw}{AGMLafOfRyyMVeEuFuaTOA}{172.25.0.2}{172.25.0.2:9300}{dilmrt}{ml.machine_memory=2085416960, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, {es02}{9RNLWSMbTmetFFh3Tm1q0g}{bnuXTgAqQyCz04G7Cva1sA}{172.25.0.4}{172.25.0.4:9300}{dilmrt}{ml.machine_memory=2085416960, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}] which is a quorum; discovery will continue using [172.25.0.4:9300] from hosts providers and [{es03}{3CbHK5iBQAqt7poPRsMmaw}{AGMLafOfRyyMVeEuFuaTOA}{172.25.0.2}{172.25.0.2:9300}{dilmrt}{ml.machine_memory=2085416960, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
es02    | {"type": "server", "timestamp": "2020-05-25T15:52:37,936Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "es-docker-cluster", "node.name": "es02", "message": "failed to resolve host [es01]", "cluster.uuid": "e16AK3EnQ-28Xky2afhx4A", "node.id": "9RNLWSMbTmetFFh3Tm1q0g"

如果我不尝试恢复退出其他两个的容器似乎可以正常工作，如果我转到 http://localhost:9200/_cluster/health 我可以看到

{"cluster_name":"es-docker-cluster","status":"green","timed_out":false,"number_of_nodes":2,"number_of_data_nodes":2,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}

我将 Docker Desktop 版本 2.2.0.5 用于 MacOS Catalina 版本 10.15.4 和 Docker Compose 版本 1.25.4

这是我的 docker-compose.yml 文件供参考：

version: '2.2'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic
  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data02:/usr/share/elasticsearch/data
    networks:
      - elastic
  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data03:/usr/share/elasticsearch/data
    networks:
      - elastic

volumes:
  data01:
    driver: local
    name: data01
  data02:
    driver: local
    name: data02
  data03:
    driver: local
    name: data03

networks:
  elastic:
    driver: bridge
    name: elastic

【问题讨论】：

标签： docker elasticsearch docker-compose

【解决方案1】：

我的 docker 桌面内存设置为 2gb，一旦我将其设置为 4gb 以上，问题就停止了。

原来 docker 容器的退出代码 137 通常是由于内存分配问题！

【讨论】：

这解决了这个问题。谢谢！

【解决方案2】：

对我来说，退出代码 137 是由于向 Elasticsearch 提供了太多内存，以至于它试图为副本保留比我的系统可用内存更多的内存。 docker stats 中的主数据库占用了 6 GB，因此 Elasticsearch 为副本预留了另外 6 GB（但 never allocating it 因为它在同一个节点上）。

我在 docker-compose.yml 中限制了堆大小：

environment:
    - ES_JAVA_OPTS=-Xms750m -Xmx750m

这在 1 GB 内存中运行所有内容的速度与以前在 6 GB 内存中一样快，没有崩溃。

【讨论】：