Kubectl 在运行时不断重启容器答案

【问题标题】：Kubectl keeps restarting container when ranKubectl 在运行时不断重启容器
【发布时间】：2018-04-22 19:28:45
【问题描述】：

我目前正在尝试将我们公司的 squid 服务器迁移到 dockerized 版本，我正在努力让它与 Kubernetes 一起使用。

我已经构建了一个 Docker 镜像，当使用“docker run”运行时它运行得非常好。完整的 Docker Run 命令是：

sudo docker run -d -i -t --privileged --volume=/proc/sys/net/ipv4/ip_nonlocal_bind:/var/proc/sys/net/ipv4/ip_nonlocal_bind --net=host --cap-add=SYS_MODULE --cap-add=NET_ADMIN --cap-add=NET_RAW -v /dev:/dev -v /lib/modules:/lib/modules -p80:80 -p8080:8080 -p53:53/udp -p5353:5353/udp -p5666:5666/udp -p4500:4500/udp -p500:500/udp -p3306:3306 --name=edge crossense/edge:latest /bin/bash

当我尝试使用 Kubernetes 运行映像时，类似：

kubectl run --image=crossense/edge:latest --port=80 --port=8080 --port=53 --port=5353 --port=5666 --port=4500 --port=500 --port=3306 edge

似乎 Kubernetes 试图让容器启动并运行，但没有任何成功...

$kubectl get po
NAME         READY     REASON    RESTARTS   AGE
edge-sz7wp   0/1       Running   10         15m

$kubectl describe pod edge 命令给了我很多这些：

Thu, 09 Nov 2017 17:13:05 +0000 Thu, 09 Nov 2017 17:13:05 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           started     Started with docker id abcc2ff25a624a998871e02bcb62d42d6f39e9db0a39f601efa4d357dd8334aa
Thu, 09 Nov 2017 17:13:15 +0000 Thu, 09 Nov 2017 17:13:15 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           created     Created with docker id 878778836bd3cc25bdf1e3b9cc2f2f6fa22b75b938a481172f08a6ec50571582
Thu, 09 Nov 2017 17:13:15 +0000 Thu, 09 Nov 2017 17:13:15 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           started     Started with docker id 878778836bd3cc25bdf1e3b9cc2f2f6fa22b75b938a481172f08a6ec50571582
Thu, 09 Nov 2017 17:13:25 +0000 Thu, 09 Nov 2017 17:13:25 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           created     Created with docker id aa51e94536216b905ff9ba07951fedbc0007476b55dfdb2e5106418fb6aee05c
Thu, 09 Nov 2017 17:13:25 +0000 Thu, 09 Nov 2017 17:13:25 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           started     Started with docker id aa51e94536216b905ff9ba07951fedbc0007476b55dfdb2e5106418fb6aee05c
Thu, 09 Nov 2017 17:13:35 +0000 Thu, 09 Nov 2017 17:13:35 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           created     Created with docker id f4661e5ea33471cd1ba30816b40c8ba2d204fa22509b973da4af6eedb64c592e
Thu, 09 Nov 2017 17:13:35 +0000 Thu, 09 Nov 2017 17:13:35 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           started     Started with docker id f4661e5ea33471cd1ba30816b40c8ba2d204fa22509b973da4af6eedb64c592e
Thu, 09 Nov 2017 17:13:45 +0000 Thu, 09 Nov 2017 17:13:45 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           created     Created with docker id 75f83dcb9b4f8af5134d6fd2edcd9342ecf56111e132a45f4e9787e83466e28b
Thu, 09 Nov 2017 17:13:45 +0000 Thu, 09 Nov 2017 17:13:45 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           started     Started with docker id 75f83dcb9b4f8af5134d6fd2edcd9342ecf56111e132a45f4e9787e83466e28b
Thu, 09 Nov 2017 17:13:55 +0000 Thu, 09 Nov 2017 17:13:55 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           created     Created with docker id c9d0535b3962ec9da29c068dbb0a6b64426a5ac3e52f72e79bcbaf03c9f3d403
Thu, 09 Nov 2017 17:13:55 +0000 Thu, 09 Nov 2017 17:13:55 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           started     Started with docker id c9d0535b3962ec9da29c068dbb0a6b64426a5ac3e52f72e79bcbaf03c9f3d403
Thu, 09 Nov 2017 17:14:05 +0000 Thu, 09 Nov 2017 17:14:05 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           created     Created with docker id 579f4428e9804404bd746cceee88bb6c73066a33263202bb5f1eb15f6ff26d7b
Thu, 09 Nov 2017 17:14:05 +0000 Thu, 09 Nov 2017 17:14:05 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           started     Started with docker id 579f4428e9804404bd746cceee88bb6c73066a33263202bb5f1eb15f6ff26d7b
Thu, 09 Nov 2017 17:14:15 +0000 Thu, 09 Nov 2017 17:14:15 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           started     Started with docker id d36b2c9ddf0b1a05d86b43d2a92eb3c00ae92d00e155d5a1be1da8e2682f901b
Thu, 09 Nov 2017 17:14:15 +0000 Thu, 09 Nov 2017 17:14:15 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           created     Created with docker id d36b2c9ddf0b1a05d86b43d2a92eb3c00ae92d00e155d5a1be1da8e2682f901b
Thu, 09 Nov 2017 17:14:25 +0000 Thu, 09 Nov 2017 17:14:25 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           created     Created with docker id 2d7b24537414f5e6f2981bf5f01596b19ea1abdb0eb4b81508fc7f44e8c34609
Thu, 09 Nov 2017 17:14:25 +0000 Thu, 09 Nov 2017 17:14:25 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           started     Started with docker id 2d7b24537414f5e6f2981bf5f01596b19ea1abdb0eb4b81508fc7f44e8c34609
Thu, 09 Nov 2017 17:14:35 +0000 Thu, 09 Nov 2017 17:14:35 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           started     Started with docker id fdae44c599b77d44839e4897b750203c183001a6053c926432ef5a3c7f4deb38
Thu, 09 Nov 2017 17:14:35 +0000 Thu, 09 Nov 2017 17:14:35 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           created     Created with docker id fdae44c599b77d44839e4897b750203c183001a6053c926432ef5a3c7f4deb38
Thu, 09 Nov 2017 17:14:45 +0000 Thu, 09 Nov 2017 17:14:45 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           created     Created with docker id 544351dda838d698e3bc125840edb6ad71cd0165a970cce46825df03b826eb38
Thu, 09 Nov 2017 17:14:45 +0000 Thu, 09 Nov 2017 17:14:45 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           started     Started with docker id 544351dda838d698e3bc125840edb6ad71cd0165a970cce46825df03b826eb38
Thu, 09 Nov 2017 17:14:55 +0000 Thu, 09 Nov 2017 17:14:55 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           created     Created with docker id 00fe4c286c1cc9b905c9c0927f82b39d45d41295a9dd0852131bba087bb19610
Thu, 09 Nov 2017 17:14:55 +0000 Thu, 09 Nov 2017 17:14:55 +0000 1   {kubelet 127.0.0.1} spec.containers{edge}           started     Started with docker id 00fe4c286c1cc9b905c9c0927f82b39d45d41295a9dd0852131bba087bb19610

任何帮助将不胜感激！

【问题讨论】：

能否分享kubectl logs -f edge的输出？
感谢您的回复！好像kubectl logs -f edge 没有返回任何日志... :(

标签： docker kubernetes containers kubectl

【解决方案1】：

虽然如果没有重新生成和缺少日志的能力，我无法得出结论，但可以很容易注意到的差异之一是您在 docker 命令中提供的权限，例如 NET_ADMIN 或 NET_RAW Kubernetes 运行命令中缺少的等。

Kubernetes 还提供了将此类权限分配给 pod 声明中 securityContext 内的 capabilities 的能力。

我不确定你是否可以使用 Kubectl 来做到这一点，但如果你对 pod 使用 YAML 声明，规格大致如下：

apiVersion: v1 kind: Pod metadata: name: mypod spec: containers: - name: myshell image: "ubuntu:14.04" command: - /bin/sleep - "300" securityContext: capabilities: add: - NET_ADMIN

如需更多参考，我建议快速浏览一下：

【讨论】：

我尝试将其转换为 yaml 文件，结果如下：apiVersion: v1 kind: Pod metadata: name: edge spec: containers: - name: edge image: "crossense/edge:production" command: - /bin/bash securityContext: capabilities: add: - NET_ADMIN - SYS_MODULE - NET_RAW volumeMounts: - name: ip-nonlocal-bind mountPath: /host/proc/sys/net/ipv4 - name: dev mountPath: /host/dev - name: modules mountPath: /host/lib/modules
` 端口：-容器端口：80 -容器端口：8080 -容器端口：53 -容器端口：5353 -容器端口：5666 -容器端口：4500 -容器端口：500 -容器端口：3306卷：-名称：ip- nonlocal-bind hostPath: path: /proc/sys/net/ipv4 - name: dev hostPath: path: /dev type: Directory - name: modules hostPath: path: /lib/modules type: Directory`
结果仍然是 - 没有任何日志并且 pod 一直在重启

【解决方案2】：

对于那些无法找到答案的可怜的灵魂， Pod一直重启的原因是它执行的命令已经退出，代码为0（表示成功）。

在我的例子中，我运行 /bin/bash 作为入口点命令，在我的 pod 配置 .yaml 文件中指定：

apiVersion: v1 
kind: Pod 
metadata: 
 name: edge 
spec: 
 containers: 
   - name: edge 
     image: "crossense/edge:production" 
     command: 
       - /bin/bash 
     securityContext: 
       capabilities: 
         add: 
           - NET_ADMIN 
           - SYS_MODULE 
           - NET_RAW 
     volumeMounts: 
       - name: ip-nonlocal-bind 
         mountPath: /host/proc/sys/net/ipv4 
       - name: dev 
         mountPath: /host/dev 
       - name: modules 
         mountPath: /host/lib/modules

....

解决方案只是在中添加一个非退出命令 入口点。这可以是在前台运行的任何进程，也可以只是一个 /bin/sleep

为了示例和将来的学习，我最终的 pod 配置文件如下所示：

apiVersion: v1
kind: Pod
metadata:
 name: edge
spec:
 hostNetwork: true
 containers:
   - name: edge
     image: "crossense/edge:production"
     command: ["/bin/bash", "-c"]
     args: ["service rsyslog restart; service proxysql start; service mongodb start; service pdns-recursor start; service supervisor start; service danted start; touch /var/run/squid.pid; chown proxy /var/run/squid.pid; service squid restart; service ipsec start; /sbin/iptables-restore < /etc/iptables/rules.v4; sleep infinity"]
     securityContext:
       privileged: true
       capabilities:
         add:
           - NET_ADMIN
           - SYS_MODULE
           - NET_RAW
     volumeMounts:
       - mountPath: /dev/shm
         name: dshm
       - name: ip-nonlocal-bind
         mountPath: /host/proc/sys/net/ipv4
       - name: dev
         mountPath: /dev
       - name: modules
         mountPath: /lib/modules
     ports:
       - containerPort: 80
       - containerPort: 8080
       - containerPort: 53
         protocol: UDP
       - containerPort: 5353
         protocol: UDP
       - containerPort: 5666
       - containerPort: 4500
       - containerPort: 500
       - containerPort: 3306
 volumes:
   - name: dshm
     emptyDir:
       medium: Memory
   - name: ip-nonlocal-bind
     hostPath:
       path: /proc/sys/net/ipv4
   - name: dev
     hostPath:
       path: /dev
       type: Directory
   - name: modules
     hostPath:
       path: /lib/modules
       type: Directory

如有任何问题，请随时对此主题发表评论，或通过 max.vlashchuk@gmail.com 向我提问 :)

【讨论】：