【发布时间】：2022-12-03 13:28:16
【问题描述】：
我遇到 K8S POD 被 OOM 杀死的问题，但有一些奇怪的情况和观察结果。
pod 是基于 golang 1.15.6 的 REST 服务，运行在 X86 64 位架构上。当 pod 在基于 VM 的集群上运行时，一切正常，服务运行正常。当服务在直接在硬件上配置的节点上运行时，它似乎会发生内存泄漏并最终导致 OOMed。
观察发现，当在有问题的配置上运行时，“kubectl top pod”将报告不断增加的内存利用率，直到达到定义的限制 (64MiB)，此时 OOM killer 被调用。
使用“top”从 pod 内部观察表明 pod 内部各种进程的内存使用情况稳定，使用大约 40MiB RSS。 top 报告的 VIRT、RES、SHR 的值随着时间的推移保持稳定，只有很小的波动。
我广泛分析了 golang 代码，包括随着时间的推移获取内存配置文件 (pprof)。实际的 golang 代码没有泄漏的迹象，这与基于 VM 的环境中的正确操作和来自顶部的观察结果相符。
下面的 OOM 消息还表明 pod 使用的 RSS 总量仅为 38.75MiB（总和/RSS = 9919 页 *4k = 38.75MiB）。
kernel: [651076.945552] xxxxxxxxxxxx invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=999
kernel: [651076.945556] CPU: 35 PID: 158127 Comm: xxxxxxxxxxxx Not tainted 5.4.0-73-generic #82~18.04.1
kernel: [651076.945558] Call Trace:
kernel: [651076.945567] dump_stack+0x6d/0x8b
kernel: [651076.945573] dump_header+0x4f/0x200
kernel: [651076.945575] oom_kill_process+0xe6/0x120
kernel: [651076.945577] out_of_memory+0x109/0x510
kernel: [651076.945582] mem_cgroup_out_of_memory+0xbb/0xd0
kernel: [651076.945584] try_charge+0x79a/0x7d0
kernel: [651076.945585] mem_cgroup_try_charge+0x75/0x190
kernel: [651076.945587] __add_to_page_cache_locked+0x1e1/0x340
kernel: [651076.945592] ? scan_shadow_nodes+0x30/0x30
kernel: [651076.945594] add_to_page_cache_lru+0x4f/0xd0
kernel: [651076.945595] pagecache_get_page+0xea/0x2c0
kernel: [651076.945596] filemap_fault+0x685/0xb80
kernel: [651076.945600] ? __switch_to_asm+0x40/0x70
kernel: [651076.945601] ? __switch_to_asm+0x34/0x70
kernel: [651076.945602] ? __switch_to_asm+0x40/0x70
kernel: [651076.945603] ? __switch_to_asm+0x34/0x70
kernel: [651076.945604] ? __switch_to_asm+0x40/0x70
kernel: [651076.945605] ? __switch_to_asm+0x34/0x70
kernel: [651076.945606] ? __switch_to_asm+0x40/0x70
kernel: [651076.945608] ? filemap_map_pages+0x181/0x3b0
kernel: [651076.945611] ext4_filemap_fault+0x31/0x50
kernel: [651076.945614] __do_fault+0x57/0x110
kernel: [651076.945615] __handle_mm_fault+0xdde/0x1270
kernel: [651076.945617] handle_mm_fault+0xcb/0x210
kernel: [651076.945621] __do_page_fault+0x2a1/0x4d0
kernel: [651076.945625] ? __audit_syscall_exit+0x1e8/0x2a0
kernel: [651076.945627] do_page_fault+0x2c/0xe0
kernel: [651076.945628] page_fault+0x34/0x40
kernel: [651076.945630] RIP: 0033:0x5606e773349b
kernel: [651076.945634] Code: Bad RIP value.
kernel: [651076.945635] RSP: 002b:00007fbdf9088df0 EFLAGS: 00010206
kernel: [651076.945637] RAX: 0000000000000000 RBX: 0000000000004e20 RCX: 00005606e775ce7d
kernel: [651076.945637] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00007fbdf9088dd0
kernel: [651076.945638] RBP: 00007fbdf9088e48 R08: 0000000000006c50 R09: 00007fbdf9088dc0
kernel: [651076.945638] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fbdf9088dd0
kernel: [651076.945639] R13: 0000000000000000 R14: 00005606e7c6140c R15: 0000000000000000
kernel: [651076.945640] memory: usage 65536kB, limit 65536kB, failcnt 26279526
kernel: [651076.945641] memory+swap: usage 65536kB, limit 9007199254740988kB, failcnt 0
kernel: [651076.945642] kmem: usage 37468kB, limit 9007199254740988kB, failcnt 0
kernel: [651076.945642] Memory cgroup stats for /kubepods/burstable/pod34ffde14-8e80-4b3a-99ac-910137a04dfe:
kernel: [651076.945652] anon 25112576
kernel: [651076.945652] file 0
kernel: [651076.945652] kernel_stack 221184
kernel: [651076.945652] slab 41406464
kernel: [651076.945652] sock 0
kernel: [651076.945652] shmem 0
kernel: [651076.945652] file_mapped 2838528
kernel: [651076.945652] file_dirty 0
kernel: [651076.945652] file_writeback 0
kernel: [651076.945652] anon_thp 0
kernel: [651076.945652] inactive_anon 0
kernel: [651076.945652] active_anon 25411584
kernel: [651076.945652] inactive_file 0
kernel: [651076.945652] active_file 536576
kernel: [651076.945652] unevictable 0
kernel: [651076.945652] slab_reclaimable 16769024
kernel: [651076.945652] slab_unreclaimable 24637440
kernel: [651076.945652] pgfault 7211542
kernel: [651076.945652] pgmajfault 2895749
kernel: [651076.945652] workingset_refault 71200645
kernel: [651076.945652] workingset_activate 5871824
kernel: [651076.945652] workingset_nodereclaim 330
kernel: [651076.945652] pgrefill 39987763
kernel: [651076.945652] pgscan 144468270
kernel: [651076.945652] pgsteal 71255273
kernel: [651076.945652] pgactivate 27649178
kernel: [651076.945652] pgdeactivate 33525031
kernel: [651076.945653] Tasks state (memory values in pages):
kernel: [651076.945653] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
kernel: [651076.945656] [ 151091] 0 151091 255 1 36864 0 -998 pause
kernel: [651076.945675] [ 157986] 0 157986 58 4 32768 0 999 dumb-init
kernel: [651076.945676] [ 158060] 0 158060 13792 869 151552 0 999 su
kernel: [651076.945678] [ 158061] 1234 158061 18476 6452 192512 0 999 yyyyyy
kernel: [651076.945679] [ 158124] 1234 158124 1161 224 53248 0 999 sh
kernel: [651076.945681] [ 158125] 1234 158125 348755 2369 233472 0 999 xxxxxxxxxxxx
kernel: [651076.945682] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=a0027a4fe415aa7a6ad54aa3fbf553b9af27c61043d08101931e985efeee0ed7,mems_allowed=0-3,oom_memcg=/kubepods/burstable/pod34ffde14-8e80-4b3a-99ac-910137a04dfe,task_memcg=/kubepods/burstable/pod34ffde14-8e80-4b3a-99ac-910137a04dfe/a0027a4fe415aa7a6ad54aa3fbf553b9af27c61043d08101931e985efeee0ed7,task=yyyyyy,pid=158061,uid=1234
kernel: [651076.945695] Memory cgroup out of memory: Killed process 158061 (yyyyyy) total-vm:73904kB, anon-rss:17008kB, file-rss:8800kB, shmem-rss:0kB, UID:1234 pgtables:188kB oom_score_adj:999
kernel: [651076.947429] oom_reaper: reaped process 158061 (yyyyyy), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
OOM 消息清楚地指出 usage = 65536kB，limit = 65536kB，但我并没有立即了解 RSS 下未考虑的大约 25MiB 内存已经消失的地方。
我看到 slab_unreclaimable = 24637440, (24MiB)，这大约是似乎下落不明的内存量，但不确定这是否有任何意义。
寻找有关内存使用位置的任何建议。欢迎任何意见。
【问题讨论】：
-
我也是K8S的新手。只是尝试一下。有时，由于错误的配置也会发生内存泄漏。也许 livenessProbe 没有正确配置？ K8S 有一个系统，如果某些地方没有按预期配置，它会自动丢弃 pod 作为内存泄漏。
标签： linux go kubernetes memory-leaks out-of-memory