docker stack/swarm 中的弹性

Question

我有两个节点群

[ra@speechanalytics-test ~]$ docker node ls
ID                            HOSTNAME                  STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
mlwwmkdlzbv0zlapqe1veq3uq     speechanalytics-preprod   Ready               Active                                  18.09.3
se717p88485s22s715rdir9x2 *   speechanalytics-test      Ready               Active              Leader              18.09.3

我正在尝试在堆栈中运行具有弹性的容器。这是我的docker-compose.yml 文件

version: '3.4'
services:
  elastic:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.7.0
    environment:
      - cluster.name=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata:/usr/share/elasticsearch/data
    deploy:
      placement:
        constraints:
          - node.hostname==speechanalytics-preprod

volumes:
  esdata:
    driver: local

使用 docker stack 启动后

​​>

docker stack deploy preprod -c docker-compose.yml

容器在 20 秒内崩溃

docker service logs preprod_elastic 
...
   | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   | OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=0
   | [2019-04-03T16:41:30,044][WARN ][o.e.b.JNANatives         ] [unknown] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
   | [2019-04-03T16:41:30,049][WARN ][o.e.b.JNANatives         ] [unknown] This can result in part of the JVM being swapped out.
   | [2019-04-03T16:41:30,049][WARN ][o.e.b.JNANatives         ] [unknown] Increase RLIMIT_MEMLOCK, soft limit: 16777216, hard limit: 16777216
   | [2019-04-03T16:41:30,050][WARN ][o.e.b.JNANatives         ] [unknown] These can be adjusted by modifying /etc/security/limits.conf, for example:
   | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   |     # allow user 'elasticsearch' mlockall
   | OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=0
   |     elasticsearch soft memlock unlimited
   | [2019-04-03T16:41:02,949][WARN ][o.e.b.JNANatives         ] [unknown] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
   |     elasticsearch hard memlock unlimited
   | [2019-04-03T16:41:02,954][WARN ][o.e.b.JNANatives         ] [unknown] This can result in part of the JVM being swapped out.
   | [2019-04-03T16:41:30,050][WARN ][o.e.b.JNANatives         ] [unknown] If you are logged in interactively, you will have to re-login for the new limits to take effect.
   | [2019-04-03T16:41:02,954][WARN ][o.e.b.JNANatives         ] [unknown] Increase RLIMIT_MEMLOCK, soft limit: 16777216, hard limit: 16777216
preprod

在我拥有的两个节点上

ra@speechanalytics-preprod:~$ sysctl vm.max_map_count
vm.max_map_count = 262144

任何想法如何解决？

Answer 1

您在 Elasticsearch 中看到的 memlock 错误是一个常见问题，并非使用 Docker 独有，而是在 Elasticsearch 被告知锁定其内存但无法这样做时发生。您可以通过从docker-compose.yml 文件中删除以下环境变量来规避错误：

- bootstrap.memory_lock=true

Memlock 可以与 Docker Swarm 模式一起使用，但有一些注意事项。

并非所有适用于docker-compose（Docker Compose）的选项都适用于docker stack deploy（Docker Swarm 模式），反之亦然，尽管它们共享 docker-compose YAML 语法。一个这样的选项是ulimits:，当与 docker stack deploy 一起使用时，将被忽略并显示警告消息，如下所示：

Ignoring unsupported options: ulimits

我的猜测是，对于您的 docker-compose.yml 文件，Elasticsearch 可以在 docker-compose up 上运行良好，但在 docker stack deploy 上运行良好。

使用 Docker Swarm 模式，默认情况下，您定义的 Elasticsearch 实例将无法使用 memlock。目前，官方支持 docker swarm 服务的 ulimits 设置为not yet。不过，有一些方法可以解决这个问题。

如果主机是 Ubuntu，则可以跨 docker 服务启用无限制的 memlock（请参阅 here 和 here）。这可以通过以下命令来实现：

echo -e "[Service]\nLimitMEMLOCK=infinity" | SYSTEMD_EDITOR=tee systemctl edit docker.service
systemctl daemon-reload
systemctl restart docker

但是，将 memlock 设置为无穷大并非没有缺点，正如 Elastic 自己所说的 here。

根据我的测试，该解决方案适用于 Docker 18.06，但不适用于 18.09。考虑到 Elasticsearch 无法启动的不一致和可能性，更好的选择是在 Swarm 上部署时不将 memlock 与 Elasticsearch 一起使用。相反，您可以选择Elasticsearch Docs 中提到的任何其他方法来获得类似的结果。