为什么在 docker 中运行 openstack 容器时会出现“Watchdog: BUG: Soft lockup CPU”错误。答案

【问题标题】：Why does “Watchdog: BUG: Soft lockup CPU” errors occur when running openstack containers in docker.为什么在 docker 中运行 openstack 容器时会出现“Watchdog: BUG: Soft lockup CPU”错误。
【发布时间】：2017-11-10 05:17:09
【问题描述】：

我刚刚使用 kolla 和 kolla-ansible stable/pike 版本部署了 openstack。我部署在一个控制节点和一个计算节点上。它有效，但是当我尝试使用地平线仪表板时，控制节点非常慢并且锁定并且终端输出以下内容：

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [runc:[2:INIT]:10527]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [fluentd:5005]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [keepalived:10664]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [java:5604]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [neutron-openvsw:3101]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [cinder-schedule:3193]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [java:6186]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [docker-containe:6601]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [keepalived:4295]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [keepalived:10666]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [gmain:939]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [node:3261]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [neutron-l3-agen:5071]

Message from syslogd@openstackcontroller at Nov  7 21:46:39 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [irqbalance:898]

每当我停止在控制节点上运行的所有 docker 容器时，它似乎不再有任何 CPU 锁定错误，但是一旦我再次启动所有在控制节点上运行 openstack 服务的 docker 容器，它就会启动再次运行很慢，锁定错误再次开始

【问题讨论】：

这更多是关于系统管理而不是开发软件——考虑 ServerFault 而不是 StackOverflow。

标签： docker cpu openstack openstack-nova openstack-neutron

【解决方案1】：

您可以使用更新的 Linux 内核 > 4.6.x 吗？

在软锁定时使内核恐慌并在恐慌时重新启动将在一定程度上帮助恢复系统。他们为我工作。

sysctl kernel.softlockup_panic=1  # this will panic on soft lockup
sysctl kernel.panic=60            # reset system on panic after 60 seconds

上述内核设置适用于每个https://github.com/moby/moby/issues/19758#issuecomment-272430028 的少数人。

【讨论】：