运行容器问题答案

【问题标题】：Running containers issue运行容器问题
【发布时间】：2020-10-23 07:29:32
【问题描述】：

我在 azure kubernetes 集群中部署了一点 windows .exe。当我运行kubectl get pods我得到以下结果，

NAME                                    READY   STATUS             RESTARTS   AGE
sample-deploy-548d6b9c6b-8v2nb          0/1     CrashLoopBackOff   5          6m12s
sample-deploy-548d6b9c6b-fpmz9          0/1     CrashLoopBackOff   5          6m12s
sample-deploy-548d6b9c6b-hgsj7          0/1     CrashLoopBackOff   5          6m12s

当我运行 kubectl describe pod sample-deploy-548d6b9c6b-8v2nb 时，我得到以下详细信息

Name:           sample-deploy-548d6b9c6b-8v2nb
Namespace:      default
Priority:       0
Node:           akswin000000/10.240.0.35
Start Time:     Thu, 02 Jul 2020 16:59:02 +0100
Labels:         app=sampleservice
                pod-template-hash=548d6b9c6b
Annotations:    <none>
Status:         Running
IP:             10.240.0.45
IPs:            <none>
Controlled By:  ReplicaSet/sample-deploy-548d6b9c6b
Containers:
  sampleservice:
    Container ID:  docker://3d22a9e647d4652227a9986f6940c6806e477f0b790a74f5795840131cc861ca
    Image:         samplekube.azurecr.io/sample:v1
    Image ID:      docker-pullable://samplekube.azurecr.io/sample@sha256:a814e92d5af97b8cfbd6cd0789e164858848f82f0316a771670382ce0bbcba92
    Port:          <none>
    Host Port:     <none>
    State:         Waiting
      Reason:      CrashLoopBackOff
    Last State:    Terminated
      Reason:      ContainerCannotRun
      Message:     hcsshim::CreateComputeSystem 3d22a9e647d4652227a9986f6940c6806e477f0b790a74f5795840131cc861ca: The container operating system does not match the host operating system.
(extra info: {"SystemType":"Container","Name":"3d22a9e647d4652227a9986f6940c6806e477f0b790a74f5795840131cc861ca","Owner":"docker","VolumePath":"\\\\?\\Volume{58649455-b9a5-4d00-b151-485ec8ab6006}","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\\ProgramData\\docker\
\windowsfilter\\3d22a9e647d4652227a9986f6940c6806e477f0b790a74f5795840131cc861ca","Layers":[{"ID":"7d7579eb-d8f7-5314-b6a0-399937aee9ca","Path":"C:\\ProgramData\\docker\\windowsfilter\\e0357f9d6b48e4b580a09cefedec8aac329894b57a49a30f9dc27795a1626aca"},{"ID":"f9bd195c-3ff
c-5c98-9713-1a7658666667","Path":"C:\\ProgramData\\docker\\windowsfilter\\019404385f250e8807ea3b693e35813b3328b3a14e83da51e8119401f0d20f9f"},{"ID":"0d763990-3499-5a19-b5e9-5e0788397f83","Path":"C:\\ProgramData\\docker\\windowsfilter\\3be0598c3fa3671a1436c670b6964c0a30ddc
2bd2e4011f347e6ef503888826a"},{"ID":"88fb7b4f-d24a-5ddf-9b67-861041ffef72","Path":"C:\\ProgramData\\docker\\windowsfilter\\978600b419ddd768b0b03c09e198d7b8d411cc6ca63b5ba15b6cc5343bb8b2a7"}],"ProcessorWeight":5000,"HostName":"sample-deploy-548d6b9c6b-8v2nb","MappedDirect
ories":[{"HostPath":"c:\\var\\lib\\kubelet\\pods\\8257607b-9506-42af-9068-a3965bb46648\\volumes\\kubernetes.io~secret\\default-token-9wzn2","ContainerPath":"c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount","ReadOnly":true,"BandwidthMaximum":0,"IOPSMaximum":0,"Create
InUtilityVM":false}],"HvPartition":false,"NetworkSharedContainerName":"fbd7d679302c57485ca7d4842528077fbb09e43ad691f47dd4cc84cbd8d3e3db"})
      Exit Code:    128
      Started:      Thu, 02 Jul 2020 16:59:37 +0100
      Finished:     Thu, 02 Jul 2020 16:59:37 +0100
    Ready:          False
    Restart Count:  2
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-9wzn2 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-9wzn2:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-9wzn2
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  kubernetes.io/os=windows
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                From                   Message
  ----     ------     ----               ----                   -------
  Normal   Scheduled  67s                default-scheduler      Successfully assigned default/sample-deploy-548d6b9c6b-8v2nb to akswin000000
  Warning  BackOff    18s (x2 over 45s)  kubelet, akswin000000  Back-off restarting failed container
  Normal   Pulling    7s (x4 over 60s)   kubelet, akswin000000  Pulling image "samplekube.azurecr.io/sample:v1"
  Normal   Pulled     6s (x4 over 57s)   kubelet, akswin000000  Successfully pulled image "samplekube.azurecr.io/sample:v1"
  Normal   Created    5s (x4 over 57s)   kubelet, akswin000000  Created container sampleservice
  Warning  Failed     5s (x4 over 56s)   kubelet, akswin000000  Error: failed to start container "sampleservice": Error response from daemon: hcsshim::CreateComputeSystem sampleservice: The container operating system does not match the host operating system.
(extra info: {"SystemType":"Container","Name":"sampleservice","Owner":"docker","VolumePath":"\\\\?\\Volume{58649455-b9a5-4d00-b151-485ec8ab6006}","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\\ProgramData\\docker\\windowsfilter\\sampleservice","Layers":[{"ID":"7d7
579eb-d8f7-5314-b6a0-399937aee9ca","Path":"C:\\ProgramData\\docker\\windowsfilter\\e0357f9d6b48e4b580a09cefedec8aac329894b57a49a30f9dc27795a1626aca"},{"ID":"f9bd195c-3ffc-5c98-9713-1a7658666667","Path":"C:\\ProgramData\\docker\\windowsfilter\\019404385f250e8807ea3b693e35
813b3328b3a14e83da51e8119401f0d20f9f"},{"ID":"0d763990-3499-5a19-b5e9-5e0788397f83","Path":"C:\\ProgramData\\docker\\windowsfilter\\3be0598c3fa3671a1436c670b6964c0a30ddc2bd2e4011f347e6ef503888826a"},{"ID":"88fb7b4f-d24a-5ddf-9b67-861041ffef72","Path":"C:\\ProgramData\\do
cker\\windowsfilter\\978600b419ddd768b0b03c09e198d7b8d411cc6ca63b5ba15b6cc5343bb8b2a7"}],"ProcessorWeight":5000,"HostName":"sample-deploy-548d6b9c6b-8v2nb","MappedDirectories":[{"HostPath":"c:\\var\\lib\\kubelet\\pods\\8257607b-9506-42af-9068-a3965bb46648\\volumes\\kuber
netes.io~secret\\default-token-9wzn2","ContainerPath":"c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount","ReadOnly":true,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false}],"HvPartition":false,"NetworkSharedContainerName":"fbd7d679302c57485ca7d4842528077
fbb09e43ad691f47dd4cc84cbd8d3e3db"})

在我看来是尝试在 linux 而不是 windows 下运行容器，我的集群中有 windows 和 linux 节点。我该如何解决这个问题。谢谢

kubectl get nodes -o wide --show-labels下方

NAME                                STATUS   ROLES   AGE     VERSION    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION      CONTAINER-RUNTIME       LABELS
aks-agentpool-38156504-vmss000000   Ready    agent   5h17m   v1.15.11   10.240.0.4    <none>        Ubuntu 16.04.6 LTS               4.15.0-1089-azure   docker://3.0.10+azure   agentpool=agentpool,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=Standard_DS2_v2,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=westeurope,failure-domain.beta.kubernetes.io/zone=0,kubernetes.azure.com/cluster=MC_testmass_anthonycluster_westeurope,kubernetes.azure.com/mode=system,kubernetes.azure.com/node-image-version=AKSUbuntu-1604-2020.06.18,kubernetes.azure.com/role=agent,kubernetes.io/arch=amd64,kubernetes.io/hostname=aks-agentpool-38156504-vmss000000,kubernetes.io/os=linux,kubernetes.io/role=agent,node-role.kubernetes.io/agent=,storageprofile=managed,storagetier=Premium_LRS
akswin000000                        Ready    agent   5h14m   v1.15.11   10.240.0.35   <none>        Windows Server 2019 Datacenter   10.0.17763.1282     docker://19.3.5         agentpool=win,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=Standard_DS2_v2,beta.kubernetes.io/os=windows,failure-domain.beta.kubernetes.io/region=westeurope,failure-domain.beta.kubernetes.io/zone=0,kubernetes.azure.com/cluster=MC_testmass_anthonycluster_westeurope,kubernetes.azure.com/node-image-version=AKSWindows-2019-17763.1282.200610,kubernetes.azure.com/role=agent,kubernetes.io/arch=amd64,kubernetes.io/hostname=akswin000000,kubernetes.io/os=windows,kubernetes.io/role=agent,node-role.kubernetes.io/agent=,storageprofile=managed,storagetier=Premium_LRS

这是yaml文件

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-deploy
  labels:
    app: sampleservice
spec:
  replicas: 3
  template:
    metadata:
      name: sampleservice
      labels:
        app: sampleservice
    spec:
      nodeSelector:
        "kubernetes.io/os": windows
      containers:
      - name: sampleservice
        image: samplekube.azurecr.io/sample:v1
        imagePullPolicy: Always
      restartPolicy: Always    
  selector:
    matchLabels:
      app: sampleservice

---

apiVersion: v1
kind: Service
metadata:
  name: sample-service
spec:
  selector:
    app: sampleservice
  ports:
    - port: 80
  type: LoadBalancer

这就是我现在描述一个 pod 时所得到的，

Events:
  Type     Reason     Age                From                     Message
  ----     ------     ----               ----                     -------
  Normal   Scheduled  94s                default-scheduler        Successfully assigned default/sample-deploy-6d4b86bf46-djtvf to aksnpwin000000
  Normal   Pulled     31s (x4 over 82s)  kubelet, aksnpwin000000  Container image "masskube.azurecr.io/sample2:v1" already present on machine
  Normal   Created    31s (x4 over 82s)  kubelet, aksnpwin000000  Created container sampleservice
  Normal   Started    29s (x4 over 79s)  kubelet, aksnpwin000000  Started container sampleservice
  Warning  BackOff    3s (x5 over 59s)   kubelet, aksnpwin000000  Back-off restarting failed container

这就是我在运行 kubectl logs podname 后得到的结果

'Sample.exe' is not recognized as an internal or external command,
operable program or batch file.

【问题讨论】：

分享 kubectl get nodes -o wide --show-labels 的输出
共享见上文
已添加 yaml 文件
你能发布 samplekube.azurecr.io/sample 的 dockerfile 吗？我想知道您的图像是否有问题。如果你看这里，它看起来像是在 Windows 上运行它：Successfully assigned default/sample-deploy-548d6b9c6b-8v2nb to akswin000000
我同意@BrianPursley。也许您只是使用了一些 linux 映像作为容器的基础映像。请提供 Dockerfile。

标签： azure kubernetes containers azure-aks

【解决方案1】：

我已经设法通过如下修改 docker 文件解决了这个问题，

FROM mcr.microsoft.com/dotnet/framework/runtime:4.8-windowsservercore-ltsc2019
WORKDIR /app
EXPOSE 80
COPY /bin/Release .
ENTRYPOINT ["Sample.exe"]

感谢大家的帮助

【讨论】：

【解决方案2】：

您需要为部署模板设置node selector，如下所示：

   nodeSelector:
      kubernetes.io/os: windows

【讨论】：

你的 yaml 看起来不一样，它有 "kubernetes.io/os": windows。你试过不带引号吗？

【解决方案3】：

部署中有nodeSelector kubernetes.io/os: windows，但windows节点有标签beta.kubernetes.io/os=windows。 nodeSelector 和 label 需要完全匹配。

您需要在 pod 规范中具有如下所示的 nodeSelector 才能在 windows 节点上调度 pod。

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-deploy
  labels:
    app: sampleservice
spec:
  replicas: 3
  template:
    metadata:
      name: sampleservice
      labels:
        app: sampleservice
    spec:
      nodeSelector:
        "beta.kubernetes.io/os": windows
      containers:
      - name: sampleservice
        image: samplekube.azurecr.io/sample:v1
        imagePullPolicy: Always
      restartPolicy: Always    
  selector:
    matchLabels:
      app: sampleservice

https://docs.microsoft.com/en-us/azure/aks/windows-container-cli

【讨论】：