Elasticsearch kubernetes pod 因 crashloopbackoff 而失败 - 后退重新启动失败的容器

Question

我正在使用这些步骤https://vocon-it.com/2019/03/04/kubernetes-9-installing-elasticsearch-using-helm-charts/ 使用 helm 图表安装 elasticsearch。 我正在使用这些 elasticsearch helm 图表 - https://github.com/helm/charts/tree/master/stable/elasticsearch

ade-db2-6c56bc6dfd-jfnw4                                       0/1     ImagePullBackOff        0          30m
augmented-data-explorer-759b9bd96-jds24                        1/1     Running                 0          30m
augmented-data-explorer-elasticsearch-client-9f8c7984b-gz4kd   0/1     Init:CrashLoopBackOff   10         30m
augmented-data-explorer-elasticsearch-data-0                   0/1     Init:CrashLoopBackOff   10         30m
augmented-data-explorer-elasticsearch-master-0                 0/1     Init:CrashLoopBackOff   10         30m

 [root@dv-demo4-master-1 ~]# kubectl -n zen describe pod augmented-data-explorer-elasticsearch-data 
Name:               augmented-data-explorer-elasticsearch-data-0
Namespace:          zen
Priority:           0
PriorityClassName:  <none>
Node:               172.16.196.167/172.16.196.167
Start Time:         Sun, 16 Jun 2019 10:31:56 -0700
Labels:             app=elasticsearch
                    component=data
                    controller-revision-hash=augmented-data-explorer-elasticsearch-data-7fbd495c9f
                    release=augmented-data-explorer
                    role=data
                    statefulset.kubernetes.io/pod-name=augmented-data-explorer-elasticsearch-data-0
Annotations:        kubernetes.io/psp: augmented-data-explorer-elasticsearch
Status:             Pending
IP:                 10.1.213.210
Controlled By:      StatefulSet/augmented-data-explorer-elasticsearch-data
Init Containers:
  sysctl:
    Container ID:  docker://42be82c2aedb8971383b1d0ce9aa23f65b63294be1f4b1dd2addffa13f2173b3
    Image:         busybox:latest
    Image ID:      docker-pullable://busybox@sha256:7a4d4ed96e15d6a3fe8bfedb88e95b153b93e230a96906910d57fc4a13210160
    Port:          <none>
    Host Port:     <none>
    Command:
      sysctl
      -w
      vm.max_map_count=262144
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 16 Jun 2019 16:43:18 -0700
      Finished:     Sun, 16 Jun 2019 16:43:18 -0700
    Ready:          False
    Restart Count:  77
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from augmented-data-explorer-elasticsearch-data-token-mqtb5 (ro)
  chown:
    Container ID:  
    Image:         docker.elastic.co/elasticsearch/elasticsearch-oss:6.7.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/bash
      -c
      set -e; set -x; chown elasticsearch:elasticsearch /usr/share/elasticsearch/data; for datadir in $(find /usr/share/elasticsearch/data -mindepth 1 -maxdepth 1 -not -name ".snapshot"); do
        chown -R elasticsearch:elasticsearch $datadir;
      done; chown elasticsearch:elasticsearch /usr/share/elasticsearch/logs; for logfile in $(find /usr/share/elasticsearch/logs -mindepth 1 -maxdepth 1 -not -name ".snapshot"); do
        chown -R elasticsearch:elasticsearch $logfile;
      done

    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /usr/share/elasticsearch/data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from augmented-data-explorer-elasticsearch-data-token-mqtb5 (ro)
Containers:
  elasticsearch:
    Container ID:   
    Image:          docker.elastic.co/elasticsearch/elasticsearch-oss:6.7.0
    Image ID:       
    Port:           9300/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:  1
    Requests:
      cpu:      25m
      memory:   1536Mi
    Readiness:  http-get http://:9200/_cluster/health%3Flocal=true delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      DISCOVERY_SERVICE:           augmented-data-explorer-elasticsearch-discovery
      NODE_MASTER:                 false
      PROCESSORS:                  1 (limits.cpu)
      ES_JAVA_OPTS:                -Djava.net.preferIPv4Stack=true -Xms1536m -Xmx1536m  
      EXPECTED_MASTER_NODES:       1
      MINIMUM_MASTER_NODES:        1
      RECOVER_AFTER_MASTER_NODES:  1
      bootstrap.memory_lock:       true
    Mounts:
      /post-start-hook.sh from config (rw)
      /pre-stop-hook.sh from config (rw)
      /usr/share/elasticsearch/config/elasticsearch.yml from config (rw)
      /usr/share/elasticsearch/data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from augmented-data-explorer-elasticsearch-data-token-mqtb5 (ro)
Conditions:
  Type              Status
  Initialized       False 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-augmented-data-explorer-elasticsearch-data-0
    ReadOnly:   false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      augmented-data-explorer-elasticsearch
    Optional:  false
  augmented-data-explorer-elasticsearch-data-token-mqtb5:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  augmented-data-explorer-elasticsearch-data-token-mqtb5
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason   Age                      From                     Message
  ----     ------   ----                     ----                     -------
  Normal   Pulled   42m (x70 over 6h12m)     kubelet, 172.16.196.167  Successfully pulled image "busybox:latest"
  Normal   Pulling  37m (x71 over 6h12m)     kubelet, 172.16.196.167  pulling image "busybox:latest"
  Warning  BackOff  117s (x1713 over 6h12m)  kubelet, 172.16.196.167  Back-off restarting failed container

pvc 已绑定 - kubectl -n zen get pvc

data-augmented-data-explorer-elasticsearch-data-0     Bound    pvc-97fab8fa-905c-11e9-9850-00163e01c3eb   30Gi       RWO            oketi-gluster   176m
data-augmented-data-explorer-elasticsearch-master-0   Bound    pvc-97fbb612-905c-11e9-9850-00163e01c3eb   4Gi        RWO            oketi-gluster   176m

这是我用于弹性搜索数据的 PVC

kind: PersistentVolumeClaim
metadata:
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: {{ template "elasticsearch.name" . }}
    component: data
    release: {{ .Release.Name }}
  name: {{ .Values.adeElasticSearchDataPVC.name }}
  namespace: {{ toYaml .Values.namespace }}
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: oketi-gluster
  resources:
    requests:
      storage: 30Gi
  volumeMode: Filesystem

这是 elasticsearch-master 的 PVC

kind: PersistentVolumeClaim
metadata:
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: {{ template "elasticsearch.name" . }}
    component: master
    release: {{ .Release.Name }}
    role: master
  name: {{ .Values.adeElasticSearchMasterPVC.name }}
  namespace:  {{ toYaml .Values.namespace }}
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: oketi-gluster
  resources:
    requests:
      storage: 4Gi
  volumeMode: Filesystem

我在日志中没有发现任何有用的东西

kubectl -n zen log augmented-data-explorer-elasticsearch-client-9f8c7984b-gz4kd
Error from server (BadRequest): container "elasticsearch" in pod "augmented-data-explorer-elasticsearch-client-9f8c7984b-gz4kd" is waiting to start: PodInitializing

关于如何解决此问题或如何调试它的任何想法？

Answer 1

正如@Nidhi 在 cmets 中提到的，通过调整相关 ES 容器内 /etc/sysctl.conf 中的特定 vm.max_map_count 虚拟内存限制（源文档 here）并使用 mlockall，解决了引导 elasticserach 容器的问题通过授予 ES 用户锁定内存权限来防止内存分配被换出的功能。

这个可以通过修改/etc/security/limits.conf来调整，例如：

elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited