【问题标题】:How to detect when a Docker container reaches its configured memory limit如何检测 Docker 容器何时达到其配置的内存限制
【发布时间】:2019-05-19 05:57:04
【问题描述】:

给定一个使用memory limit 启动的 Docker 容器,我想知道是否可以检测到何时强制执行该内存限制,因此容器即将崩溃,因为它的内存不足。是否有一些我可以捕获的信号?或者我可以轮询的一些日志文件?事后检测到这一点,即使延迟几分钟仍然有用

【问题讨论】:

  • 更简单:docker stats --no-stream | tr -d "%" | awk '{if($7>10)print "Warning: " $1 "/" $2" is consuming more than 80% of its availabile memory ("$7"%)"}'

标签: docker memory-limit


【解决方案1】:

如果 docker 由于内存限制而终止了您的容器,您将看到:

docker inspect --format '{{.State.OOMKilled}}' ${container_id}

设置为真。

请注意,如果操作系统在达到 docker 内存限制之前终止容器内运行的进程,那么您将看到的只是应用收到的 SIGKILL。


您可以主动监控docker stats 的容器,看看它是否接近此处其他答案所述的限制。


您可以近乎实时地监控 docker events 是否有任何因 OOM 而被杀死的容器:

docker events --filter type=container --filter event=oom

您可以调整上述事件命令以监控特定容器或受时间限制。有关可用标志,请参阅 documentation

【讨论】:

    【解决方案2】:

    您可以根据这些说明编写脚本:

    # for i in $(docker ps --no-trunc | awk '{print $1}' | grep -v CONTAINER); do \
    LIMIT=`cat /sys/fs/cgroup/memory/docker/${i}/memory.limit_in_bytes`; \
    USAGE=`systemd-cgtop -b --iterations 1 | grep ${i} | awk '{print $4}'`; \
    echo Container ${i} is using $USAGE and its limited at $LIMIT; done;
    
    Container e831b58193287ac61038da29f488423da894d44f810a9b2ff6df63b800d98217 is using 4.5G and its limited at 9223372036854771712
    Container 29b484032c895b84e7297091b0b0fc69513c34a03e10ac7906e428538a47e1a1 is using 19.4M and its limited at 9223372036854771712
    Container a6ff3e99790f7d3db1da14736e4f48623512fbd46b2683a158aed1736c17073d is using 14.7M and its limited at 9223372036854771712
    Container 36db60a74eab475397371bfa0d1ea432486841d036c7cc54e527df1f7b95bbe9 is using 15.3M and its limited at 9223372036854771712
    Container 9be4c12074f05eb00e4e5d17eafe733651ff7b7e962faf328e4dd8a10ca8fee7 is using 216.0K and its limited at 9223372036854771712
    Container e9da54900df72ab4b1c6233f51b011a68246315d714278e11b54e7b18367593c is using 2.6G and its limited at 9223372036854771712
    Container a670124330c56af011d9bc9b86cbd5571c45af9d2d2dbeffdc6d2aee0b04909f is using 350.1M and its limited at 9223372036854771712
    Container e2688ff5493c4517ae55e11281ba07c2743f2464da3c08f88fdce1f4bd0c2d8d is using 2.6G and its limited at 9223372036854771712
    Container b0b92b056f82f9b4bff5462b6b896d366245dc9af9847d40f85d20b0497c5601 is using 1.1G and its limited at 9223372036854771712
    Container 41e9c719c65e5bc568596f4046fab1acd0ac0e15c93c76563ef25ef5c483b357 is using 61.7M and its limited at 9223372036854771712
    Container 1ec5305130846edb67218b2010306090d1efed6e6bc12d04de68fae94db330bd is using 155.1M and its limited at 9223372036854771712
    Container 1d3cf6ed42eae789ecc01503e704ff29aa4bd9adb0764baf28f393a882bc0c42 is using 366.3M and its limited at 9223372036854771712
    Container 4e96919f2a0e5d969ee1907bffe45fed34b7a4e3dbb7a29ab18169579aed7994 is using 68.7M and its limited at 9223372036854771712
    Container 31cc3d3a04435c49d7afb8eda5bab0ef0ee309e1dda531fa8e90a012a5543e90 is using 19.0M and its limited at 9223372036854771712
    Container 64dd6daaebc1ef852dcb3a0d2294b52c52d92889ab3f6749a99e45cf89eaa7a6 is using 13.0M and its limited at 9223372036854771712
    Container 4f33db06c957306ea19a14c22d5248a161fe5d5344c9d4136eb34c13eb76664b is using 14.7M and its limited at 9223372036854771712
    Container 85d9a633adc138f6f506df77ec0828078f521fc5cdfa3cc85e20ab43f3ec0979 is using 40.7M and its limited at 9223372036854771712
    Container c102bc6691955d1c732a725c3e9b131c3a07c507996076fb64fce3586a46dbb0 is using 114.2M and its limited at 9223372036854771712
    

    解释:

    1. 您的容器 ID 很长
    2. 您将获得每个 docker 容器的 cgroup memory.limit_in_bytes
    3. 您使用 systemd-cgtop 实用程序获取当前使用情况
    4. 您必须进行一些单元大小计算(因为 systemd-cgtop 返回 G/M/K/B 并且 memory.limit_in_bytes 始终是字节)
    5. 将使用量除以限制,配置阈值(即:0.8)并捕获您正在使用的任何监控系统

    我不限制容器的内存,因此将 9223372036854771712 视为 LIMIT 是正常的。在您的情况下,您将以字节为单位。

    正如其他用户的回答,您还可以使用更容易的 docker 实用程序(不确定您正在运行哪个版本):

    docker stats --no-stream | tr -d "%" | awk '{if($7>10)print "Warning: " $1 "/" $2" is consuming more than 80% of its availabile memory ("$7"%)"}'
    

    【讨论】:

      【解决方案3】:

      一种监控方法是使用 docker stats 命令。您甚至可以在强制执行限制之前监控容器的使用情况。

      https://docs.docker.com/config/containers/runmetrics/#docker-stats

      docker stats container1 container2
      

      你也可以做一些 bash 脚本:

      how to find MAX memory from docker stats?

      另一种选择是查看 docker events

      https://www.systutorials.com/docs/linux/man/1-docker-events/

      Docker 容器将报告以下事件:

      attach, commit, copy, create, destroy, detach, die, exec_create, exec_detach, exec_start, export, kill, oom, pause, rename, resize, restart, start, stop, top, unpause, update
      

      【讨论】:

        猜你喜欢
        • 1970-01-01
        • 2017-09-13
        • 1970-01-01
        • 2020-03-23
        • 1970-01-01
        • 1970-01-01
        • 1970-01-01
        • 1970-01-01
        • 1970-01-01
        相关资源
        最近更新 更多