如何从文件系统中删除 kubernetes pods 相关文件？答案

【问题标题】：How to remove kubernetes pods related files from file system?如何从文件系统中删除 kubernetes pods 相关文件？
【发布时间】：2022-01-24 05:30:33
【问题描述】：

我是 Kubernetes 新手。所以，我创建了几个 pod。然后我删除了所有 pod 使用

kubectl 删除 pod --all

但是df -h 的输出仍然显示 kubernetes 消耗了磁盘空间。

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       194G   19G  175G  10% /
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           1.6G  2.2M  1.6G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/loop0       34M   34M     0 100% /snap/amazon-ssm-agent/3552
/dev/loop2       56M   56M     0 100% /snap/core18/2246
/dev/loop1       25M   25M     0 100% /snap/amazon-ssm-agent/4046
/dev/loop3       56M   56M     0 100% /snap/core18/2253
/dev/loop4       68M   68M     0 100% /snap/lxd/21835
/dev/loop5       44M   44M     0 100% /snap/snapd/14295
/dev/loop6       62M   62M     0 100% /snap/core20/1242
/dev/loop7       43M   43M     0 100% /snap/snapd/14066
/dev/loop8       68M   68M     0 100% /snap/lxd/21803
/dev/loop9       62M   62M     0 100% /snap/core20/1270
tmpfs           1.6G   20K  1.6G   1% /run/user/123
tmpfs           7.8G   12K  7.8G   1% /var/lib/kubelet/pods/a2054657-e24d-434f-8ba5-b93813a405fc/volumes/kubernetes.io~secret/local-path-provisioner-service-account-token-4hkj6
tmpfs           7.8G   12K  7.8G   1% /var/lib/kubelet/pods/fa06c678-814f-4f98-8d2d-806e85923830/volumes/kubernetes.io~secret/metrics-server-token-pjbwh
shm              64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/daceb65d912a45e87d29955b499aff1d7fbc40584eade7903a75a2c5a317325a/shm
overlay         194G   19G  175G  10% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/daceb65d912a45e87d29955b499aff1d7fbc40584eade7903a75a2c5a317325a/rootfs
overlay         194G   19G  175G  10% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/956d3b341a87e4232792ebf1ad0925f07c180d6d86de149a6ec801f74c0b47f8/rootfs
shm              64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/374537a007565bba5b00824576d35e2f2ee8835c354205748117b6622dc68a6d/shm
overlay         194G   19G  175G  10% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/374537a007565bba5b00824576d35e2f2ee8835c354205748117b6622dc68a6d/rootfs
overlay         194G   19G  175G  10% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/babfe080e5ec18297a219e65f99d6156fbd8b8651950a63052606ffebd7a618a/rootfs
tmpfs           7.8G   12K  7.8G   1% /var/lib/kubelet/pods/4e3b15c1-f051-42eb-a3d1-9b3de38dae12/volumes/kubernetes.io~secret/default-token-lnpwv
tmpfs           7.8G   12K  7.8G   1% /var/lib/kubelet/pods/df53096e-f89b-4fc7-ab8a-672d841ac44f/volumes/kubernetes.io~secret/coredns-token-sxtjn
tmpfs           7.8G  8.0K  7.8G   1% /var/lib/kubelet/pods/415a1140-5813-48cf-bd88-17b647bd955c/volumes/kubernetes.io~secret/ssl
tmpfs           7.8G   12K  7.8G   1% /var/lib/kubelet/pods/415a1140-5813-48cf-bd88-17b647bd955c/volumes/kubernetes.io~secret/traefik-token-46qmp
shm              64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/d29d1a4a1ac25c92618ff9294e9045a1e2333899f64c3935c5e9955b7d1b3e61/shm
overlay         194G   19G  175G  10% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/d29d1a4a1ac25c92618ff9294e9045a1e2333899f64c3935c5e9955b7d1b3e61/rootfs
shm              64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/2ad63b79faa95666c75dfa397524c4ed5464acfebf577c388e19ae5fc349c0c8/shm
overlay         194G   19G  175G  10% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/2ad63b79faa95666c75dfa397524c4ed5464acfebf577c388e19ae5fc349c0c8/rootfs
overlay         194G   19G  175G  10% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/39b88e479947c9240a7c5233555c7a19b29f3ccc7bd1da117251c8e8959aca3c/rootfs
shm              64M     0   64M   0%

df -h 中显示的这些空格是什么？如何释放这些空间？

编辑：

我注意到在我删除它们后 pod 正在重新启动。

$ kubectl get pods
NAME                                               READY   STATUS    RESTARTS   AGE
mylab-airflow-redis-0                      1/1     Running   0          33m
mylab-airflow-postgresql-0                 1/1     Running   0          34m
mylab-postgresql-0                         1/1     Running   0          34m
mylab-keyclo-0                             1/1     Running   0          34m
mylab-keycloak-postgres-0                  1/1     Running   0          34m
mylab-airflow-scheduler-788f7f4dd6-ppg6v   2/2     Running   0          34m
mylab-airflow-worker-0                     2/2     Running   0          34m
mylab-airflow-flower-6d8585794d-s2jzd      1/1     Running   0          34m
mylab-airflow-webserver-859766684b-w9zcm   1/1     Running   0          34m
mylab-5f7d84fcbc-59mkf                     1/1     Running   0          34m

已编辑

所以我删除了部署。

kubectl delete deployment --all

现在，没有部署。

$ kubectl get deployment
No resources found in default namespace.

然后，我停止了集群。

systemctl stop k3s

磁盘空间仍未释放。

最新磁盘使用情况的输出。

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       194G   35G  160G  18% /
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           1.6G  2.5M  1.6G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/loop0       34M   34M     0 100% /snap/amazon-ssm-agent/3552
/dev/loop2       56M   56M     0 100% /snap/core18/2246
/dev/loop1       25M   25M     0 100% /snap/amazon-ssm-agent/4046
/dev/loop3       56M   56M     0 100% /snap/core18/2253
/dev/loop4       68M   68M     0 100% /snap/lxd/21835
/dev/loop5       44M   44M     0 100% /snap/snapd/14295
/dev/loop6       62M   62M     0 100% /snap/core20/1242
/dev/loop7       43M   43M     0 100% /snap/snapd/14066
/dev/loop8       68M   68M     0 100% /snap/lxd/21803
/dev/loop9       62M   62M     0 100% /snap/core20/1270
tmpfs           1.6G   20K  1.6G   1% /run/user/123
tmpfs           7.8G   12K  7.8G   1% /var/lib/kubelet/pods/a2054657-e24d-434f-8ba5-b93813a405fc/volumes/kubernetes.io~secret/local-path-provisioner-service-account-token-4hkj6
tmpfs           7.8G   12K  7.8G   1% /var/lib/kubelet/pods/fa06c678-814f-4f98-8d2d-806e85923830/volumes/kubernetes.io~secret/metrics-server-token-pjbwh
shm              64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/daceb65d912a45e87d29955b499aff1d7fbc40584eade7903a75a2c5a317325a/shm
overlay         194G   35G  160G  18% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/daceb65d912a45e87d29955b499aff1d7fbc40584eade7903a75a2c5a317325a/rootfs
overlay         194G   35G  160G  18% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/956d3b341a87e4232792ebf1ad0925f07c180d6d86de149a6ec801f74c0b47f8/rootfs
shm              64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/374537a007565bba5b00824576d35e2f2ee8835c354205748117b6622dc68a6d/shm
overlay         194G   35G  160G  18% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/374537a007565bba5b00824576d35e2f2ee8835c354205748117b6622dc68a6d/rootfs
tmpfs           7.8G   12K  7.8G   1% /var/lib/kubelet/pods/4e3b15c1-f051-42eb-a3d1-9b3de38dae12/volumes/kubernetes.io~secret/default-token-lnpwv
tmpfs           7.8G   12K  7.8G   1% /var/lib/kubelet/pods/df53096e-f89b-4fc7-ab8a-672d841ac44f/volumes/kubernetes.io~secret/coredns-token-sxtjn
tmpfs           7.8G  8.0K  7.8G   1% /var/lib/kubelet/pods/415a1140-5813-48cf-bd88-17b647bd955c/volumes/kubernetes.io~secret/ssl
tmpfs           7.8G   12K  7.8G   1% /var/lib/kubelet/pods/415a1140-5813-48cf-bd88-17b647bd955c/volumes/kubernetes.io~secret/traefik-token-46qmp
shm              64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/d29d1a4a1ac25c92618ff9294e9045a1e2333899f64c3935c5e9955b7d1b3e61/shm
overlay         194G   35G  160G  18% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/d29d1a4a1ac25c92618ff9294e9045a1e2333899f64c3935c5e9955b7d1b3e61/rootfs
shm              64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/2ad63b79faa95666c75dfa397524c4ed5464acfebf577c388e19ae5fc349c0c8/shm
overlay         194G   35G  160G  18% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/2ad63b79faa95666c75dfa397524c4ed5464acfebf577c388e19ae5fc349c0c8/rootfs
overlay         194G   35G  160G  18% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/39b88e479947c9240a7c5233555c7a19b29f3ccc7bd1da117251c8e8959aca3c/rootfs
shm              64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/6eddeab3511cf326a530dd042f5348978c6ba98bf8d595c2936cb6f56e30f754/shm
overlay         194G   35G  160G  18% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/6eddeab3511cf326a530dd042f5348978c6ba98bf8d595c2936cb6f56e30f754/rootfs
overlay         194G   35G  160G  18% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/78568d4850964c9c7b8ca5df11bf532a477492119813094631641132aadd23a0/rootfs
overlay         194G   35G  160G  18% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/14d87054e0c7a2a86ae64be70a79f94e2d193bc4739d97e261e85041c160f3bc/rootfs
overlay         194G   35G  160G  18% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/0971fe44fc6f0f5c9e0b8c1a0e3279c20b3bc574e03d12607644e1e7d427ff65/rootfs
tmpfs           1.6G  4.0K  1.6G   1% /run/user/1000

ctr containers ls的输出

# ctr container list
CONTAINER    IMAGE    RUNTIME

【问题讨论】：

标签： kubernetes

【解决方案1】：

集群运行时需要维护一些强制性数据（例如，默认服务令牌）。当您关闭（例如 systemctl stop k3s）集群（不仅仅是删除 pod）时，这些将被释放。

【讨论】：

systemctl stop and_what_here?
知道了：Failed to stop kubelet.service: Unit kubelet.service not loaded.
我看到我删除的 pod 又重新启动了。如何永久阻止他们？
运行systemctl stop k3s。要永久停止它，请执行systemctl disable k3s
systemd 会解决这个问题，您应该不理会它们。您的 k3s 已被禁用，下次重新启动机器时它不应启动。保留或删除 k3s 现在不会更改这些条目。你可以尝试重启你的机器。