Prometheus - 由于 OOMKill 而导致 CrashLoopBackOff答案

【问题标题】：Prometheus - CrashLoopBackOff due to OOMKillPrometheus - 由于 OOMKill 而导致 CrashLoopBackOff
【发布时间】：2022-12-20 02:24:45
【问题描述】：

*问题是什么？

Prometheus pod 在 GKE 集群上失败OOMKill 退出代码 137.在日志中，最重复的错误是“msg="WAL 段已加载"“ 问题是我们没有大量的指标负载，但是 Promtheus 正在占用 30Gi Ram 并且总是渴望更多.. 图片显示charts of the CPU, Memory and disk(bleu: used / violet: limit / green: requested)

> kubectl describe pods  prometheus-prometheus -n monitoring       
Name:                 prometheus-prometheus-0
Namespace:            monitoring
Priority:             800000200
Priority Class Name:  infra-critical
Node:                 gke-primary-cluster-pool-d-nd2-n03-c0-23c0ac60-q5x3/*.*.*.*
Start Time:           Tue, 25 Oct 2022 11:04:37 +0100
Labels:               app=prometheus
                      app.kubernetes.io/component=prometheus-monitoring
                      app.kubernetes.io/name=prometheus
                      app.kubernetes.io/version=v2.15.2
                      controller-revision-hash=prometheus-prometheus-97cf4c6b6
                      prometheus=prometheus
                      statefulset.kubernetes.io/pod-name=prometheus-prometheus-0
Annotations:          <none>
Status:               Running
IP:                   172.*.*.*
IPs:
  IP:           172.*.*.*
Controlled By:  StatefulSet/prometheus-prometheus
Containers:
  prometheus:
    Container ID:  containerd://94f88524092b1299e8093e57328c56da341f903f29009e4399b23616939d1c4a
    Image:         quay.io/prometheus/prometheus:v2.15.2
    Image ID:      quay.io/prometheus/prometheus@sha256:914525123cf76a15a6aaeac069fcb445ce8fb125113d1bc5b15854bc1e8b6353
    Port:          9090/TCP
    Host Port:     0/TCP
    Args:
      --web.console.templates=/etc/prometheus/consoles
      --web.console.libraries=/etc/prometheus/console_libraries
      --storage.tsdb.retention.size=20GB
      --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      --storage.tsdb.path=/prometheus
      --storage.tsdb.retention.time=4d
      --web.enable-lifecycle
      --storage.tsdb.no-lockfile
      --web.external-url=https://monitoring.team/prometheus
      --web.route-prefix=/prometheus
      --storage.tsdb.wal-compression
      --storage.tsdb.max-block-duration=2h
    State:       Waiting
      Reason:    CrashLoopBackOff
    Last State:  Terminated
      Reason:    OOMKilled
      Message:   "WAL segment loaded" segment=273 maxSegment=326
level=info ts=2022-10-25T16:46:41.787Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=274 maxSegment=326
level=info ts=2022-10-25T16:46:42.084Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=275 maxSegment=326
level=info ts=2022-10-25T16:46:42.829Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=276 maxSegment=326
level=info ts=2022-10-25T16:46:43.266Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=277 maxSegment=326
level=info ts=2022-10-25T16:46:43.690Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=278 maxSegment=326
level=info ts=2022-10-25T16:46:44.434Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=279 maxSegment=326
level=info ts=2022-10-25T16:46:44.735Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=280 maxSegment=326
level=info ts=2022-10-25T16:46:44.967Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=281 maxSegment=326
level=info ts=2022-10-25T16:46:45.273Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=282 maxSegment=326
level=info ts=2022-10-25T16:46:46.042Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=283 maxSegment=326
level=info ts=2022-10-25T16:46:46.254Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=284 maxSegment=326
level=info ts=2022-10-25T16:46:47.057Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=285 maxSegment=326
level=info ts=2022-10-25T16:46:47.365Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=286 maxSegment=326
level=info ts=2022-10-25T16:46:48.183Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=287 maxSegment=326
level=info ts=2022-10-25T16:46:48.403Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=288 maxSegment=326
level=info ts=2022-10-25T16:46:49.473Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=289 maxSegment=326


      Exit Code:    137
      Started:      Tue, 25 Oct 2022 17:45:41 +0100
      Finished:     Tue, 25 Oct 2022 17:46:50 +0100
    Ready:          False
    Restart Count:  60
    Limits:
      memory:  15Gi
    Requests:
      memory:     400Mi
    Liveness:     http-get http://:web/prometheus/-/healthy delay=0s timeout=3s period=5s #success=1 #failure=6
    Readiness:    http-get http://:web/prometheus/-/ready delay=0s timeout=3s period=5s #success=1 #failure=120
    Environment:  <none>
    Mounts:
      /etc/prometheus/certs from tls-assets (ro)
      /etc/prometheus/config_out from config-out (ro)
      /etc/prometheus/rules/prometheus-prometheus-rulefiles-0 from prometheus-prometheus-rulefiles-0 (rw)
      /prometheus from prometheus-prometheus-db (rw,path="prometheus-db")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mp7tn (ro)
  prometheus-config-reloader:
    Container ID:  containerd://5bfec5130bdee2a3b730e89d9b9b0170ad6dd28a3b59e22900a43f3f5ac924ac
    Image:         quay.io/coreos/prometheus-config-reloader:v0.38.1
    Image ID:      quay.io/coreos/prometheus-config-reloader@sha256:d1cce64093d4a850d28726ec3e48403124808f76567b5bd7b26e4416300996a7
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/prometheus-config-reloader
    Args:
      --log-format=logfmt
      --reload-url=http://localhost:9090/prometheus/-/reload
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
    State:          Running
      Started:      Tue, 25 Oct 2022 11:04:43 +0100
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  25Mi
    Requests:
      cpu:     100m
      memory:  25Mi
    Environment:
      POD_NAME:  prometheus-prometheus-0 (v1:metadata.name)
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mp7tn (ro)
  rules-configmap-reloader:
    Container ID:  containerd://52af85d56ccae7560bf8f93517aa079da0f974d023f6ea945b72391dcd8f3226
    Image:         jimmidyson/configmap-reload:v0.3.0
    Image ID:      docker.io/jimmidyson/configmap-reload@sha256:d107c7a235c266273b1c3502a391fec374430e5625539403d0de797fa9c556a2
    Port:          <none>
    Host Port:     <none>
    Args:
      --webhook-url=http://localhost:9090/prometheus/-/reload
      --volume-dir=/etc/prometheus/rules/prometheus-prometheus-rulefiles-0
    State:          Running
      Started:      Tue, 25 Oct 2022 11:04:43 +0100
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  25Mi
    Requests:
      cpu:        100m
      memory:     25Mi
    Environment:  <none>
    Mounts:
      /etc/prometheus/rules/prometheus-prometheus-rulefiles-0 from prometheus-prometheus-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mp7tn (ro)
  thanos-sidecar:
    Container ID:  containerd://3e7e11f7fc1519d80c4d1dfaa0ef644b5e5b4982751214a85920dedc80538bbf
    Image:         quay.io/thanos/thanos:v0.12.1
    Image ID:      quay.io/thanos/thanos@sha256:5bc8f56bef8c85b97442236123b51dd9efa341bd9a5dff8a08db31d399e4a2ae
    Ports:         10902/TCP, 10901/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      sidecar
      --prometheus.url=http://localhost:9090/prometheus
      --tsdb.path=/prometheus
      --grpc-address=[$(POD_IP)]:10901
      --http-address=[$(POD_IP)]:10902
      --objstore.config=$(OBJSTORE_CONFIG)
    State:          Running
      Started:      Tue, 25 Oct 2022 11:04:43 +0100
    Ready:          True
    Restart Count:  0
    Environment:
      POD_IP:            (v1:status.podIP)
      OBJSTORE_CONFIG:  <set to the key 'thanos.yaml' in secret 'thanos-storage-config'>  Optional: false
    Mounts:
      /prometheus from prometheus-prometheus-db (rw,path="prometheus-db")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mp7tn (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  prometheus-prometheus-db:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  prometheus-prometheus-db-prometheus-prometheus-0
    ReadOnly:   false
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-prometheus
    Optional:    false
  tls-assets:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-prometheus-tls-assets
    Optional:    false
  config-out:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  prometheus-prometheus-rulefiles-0:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-prometheus-rulefiles-0
    Optional:  false
  kube-api-access-mp7tn:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
                             memory=32
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                    From     Message
  ----     ------     ----                   ----     -------
  Warning  Unhealthy  11m (x912 over 6h46m)  kubelet  Readiness probe failed: HTTP probe failed with statuscode: 503
  Warning  BackOff    74s (x1355 over 6h7m)  kubelet  Back-off restarting failed container
`

*我们如何尝试解决问题？

我们尝试了如下不同的解决方案，但似乎没有解决问题：

删除 WAL 文件夹。
设置亲和力以附加到最高内存节点。
添加副本。
设置资源内存限制。
还有其他建议可以尝试吗？

【问题讨论】：

我发现最新的普罗米修斯版本是2.40.0。不确定 v2.40.0 是否可以解决这个问题。或者你的集群很大并且有很多指标。

标签： kubernetes google-kubernetes-engine prometheus

【解决方案1】：

我没有答案，但同样的错误发生在我们的 k8s 本地集群上。我刚刚设置了 4 Go（以前是 3 Go）。同样的错误，由于 oomkilled 而导致 crashloopbackoff，退出代码 137。我看到 15 进入你的普罗米修斯配置。怎么会这么大？！

Kube 版本 1.24.4 牧场主2.6.7 Rancher监控prometheus 2.28.1 集群内部部署约。 40 个虚拟机

我无法弄清楚这背后的问题......仍在互联网上寻找有用的东西。

【讨论】：