【问题标题】:Kubernetes rolling update not workingKubernetes 滚动更新不起作用
【发布时间】:2018-06-13 17:12:11
【问题描述】:

我为不同的项目安装了 2 个 kubernetes,据我所知,它们在重要的领域具有相同的配置,但 2 个执行滚动更新的方式不同。

两者都是使用 kops 安装在 AWS 上的。

系统 1 (k8s v1.7.0) - 使用 k8s web gui 在部署中杀死 pod,首先创建新 pod,然后一旦运行将终止旧 pod。没有停机时间。

System 2 (k8s v1.8.4) - 使用 k8s web gui 在部署中杀死 pod,旧的 pod 立即被杀死,然后创建新的 pod。导致短暂停机。

关于它们为何表现不同以及如何让系统 2 在终止旧 pod 之前创建新 pod 有任何建议或想法吗?

系统 1 部署

{
  "kind": "Deployment",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "name": "proxy-deployment",
    "namespace": "namespace",
    "selfLink": "/apis/extensions/v1beta1/namespaces/namespace/deployments/proxy-deployment",
    "uid": "d12778ba-8950-11e7-9e69-12f38e55b21a",
    "resourceVersion": "31538492",
    "generation": 7,
    "creationTimestamp": "2017-08-25T04:49:45Z",
    "labels": {
      "app": "proxy"
    },
    "annotations": {
      "deployment.kubernetes.io/revision": "6",
      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"apps/v1beta1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{},\"name\":\"proxy-deployment\",\"namespace\":\"namespace\"},\"spec\":{\"replicas\":2,\"template\":{\"metadata\":{\"labels\":{\"app\":\"proxy\"}},\"spec\":{\"containers\":[{\"image\":\"xxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/nginx-proxy-xxxxxx:latest\",\"name\":\"proxy-ctr\",\"ports\":[{\"containerPort\":80},{\"containerPort\":8080}]}]}}}}\n"
    }
  },
  "spec": {
    "replicas": 1,
    "selector": {
      "matchLabels": {
        "app": "proxy"
      }
    },
    "template": {
      "metadata": {
        "creationTimestamp": null,
        "labels": {
          "app": "proxy",
          "date": "1522386390"
        }
      },
      "spec": {
        "containers": [
          {
            "name": "proxy-ctr",
            "image": "xxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/nginx-proxy-xxxxxx:latest",
            "ports": [
              {
                "containerPort": 80,
                "protocol": "TCP"
              },
              {
                "containerPort": 8080,
                "protocol": "TCP"
              }
            ],
            "resources": {},
            "terminationMessagePath": "/dev/termination-log",
            "terminationMessagePolicy": "File",
            "imagePullPolicy": "Always"
          }
        ],
        "restartPolicy": "Always",
        "terminationGracePeriodSeconds": 30,
        "dnsPolicy": "ClusterFirst",
        "securityContext": {},
        "schedulerName": "default-scheduler"
      }
    },
    "strategy": {
      "type": "RollingUpdate",
      "rollingUpdate": {
        "maxUnavailable": "25%",
        "maxSurge": "25%"
      }
    },
    "revisionHistoryLimit": 2,
    "progressDeadlineSeconds": 600
  },
  "status": {
    "observedGeneration": 7,
    "replicas": 1,
    "updatedReplicas": 1,
    "readyReplicas": 1,
    "availableReplicas": 1,
    "conditions": [
      {
        "type": "Progressing",
        "status": "True",
        "lastUpdateTime": "2018-03-30T05:03:01Z",
        "lastTransitionTime": "2017-08-25T04:49:45Z",
        "reason": "NewReplicaSetAvailable",
        "message": "ReplicaSet \"proxy-deployment-1457650622\" has successfully progressed."
      },
      {
        "type": "Available",
        "status": "True",
        "lastUpdateTime": "2018-06-01T06:55:12Z",
        "lastTransitionTime": "2018-06-01T06:55:12Z",
        "reason": "MinimumReplicasAvailable",
        "message": "Deployment has minimum availability."
      }
    ]
  }
}

系统 2 部署

{
  "kind": "Deployment",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "name": "prodefault-deployment",
    "namespace": "namespace",
    "selfLink": "/apis/extensions/v1beta1/namespaces/namespace/deployments/prodefault-deployment",
    "uid": "a80528c8-eb79-11e7-9364-068125440f70",
    "resourceVersion": "25203392",
    "generation": 10,
    "creationTimestamp": "2017-12-28T02:49:00Z",
    "labels": {
      "app": "prodefault"
    },
    "annotations": {
      "deployment.kubernetes.io/revision": "7",
      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"apps/v1beta1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{},\"name\":\"prodefault-deployment\",\"namespace\":\"namespace\"},\"spec\":{\"replicas\":1,\"strategy\":{\"rollingUpdate\":{\"maxSurge\":\"25%\",\"maxUnavailable\":\"25%\"},\"type\":\"RollingUpdate\"},\"template\":{\"metadata\":{\"labels\":{\"app\":\"prodefault\"}},\"spec\":{\"containers\":[{\"image\":\"xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/xxxxxxxxxxx-pro-default:latest\",\"livenessProbe\":{\"httpGet\":{\"path\":\"/healthchk\",\"port\":80},\"initialDelaySeconds\":120,\"periodSeconds\":15,\"timeoutSeconds\":1},\"name\":\"prodefault-ctr\",\"ports\":[{\"containerPort\":80}],\"readinessProbe\":{\"httpGet\":{\"path\":\"/healthchk\",\"port\":80},\"initialDelaySeconds\":5,\"periodSeconds\":2,\"timeoutSeconds\":3},\"resources\":{\"limits\":{\"cpu\":\"1\",\"memory\":\"1024Mi\"},\"requests\":{\"cpu\":\"150m\",\"memory\":\"256Mi\"}},\"volumeMounts\":[{\"mountPath\":\"/var/www/html/homes\",\"name\":\"efs-pvc\"},{\"mountPath\":\"/var/xero\",\"name\":\"xero-key\",\"readOnly\":true},{\"mountPath\":\"/var/gcal\",\"name\":\"gcal-json\",\"readOnly\":true}]}],\"volumes\":[{\"name\":\"efs-pvc\",\"persistentVolumeClaim\":{\"claimName\":\"tio-pv-claim-homes\"}},{\"name\":\"xero-key\",\"secret\":{\"secretName\":\"xero-key\"}},{\"name\":\"gcal-json\",\"secret\":{\"secretName\":\"gcaljson\"}}]}}}}\n"
    }
  },
  "spec": {
    "replicas": 1,
    "selector": {
      "matchLabels": {
        "app": "prodefault"
      }
    },
    "template": {
      "metadata": {
        "creationTimestamp": null,
        "labels": {
          "app": "prodefault"
        }
      },
      "spec": {
        "volumes": [
          {
            "name": "efs-pvc",
            "persistentVolumeClaim": {
              "claimName": "tio-pv-claim-homes"
            }
          },
          {
            "name": "xero-key",
            "secret": {
              "secretName": "xero-key",
              "defaultMode": 420
            }
          },
          {
            "name": "gcal-json",
            "secret": {
              "secretName": "gcaljson",
              "defaultMode": 420
            }
          }
        ],
        "containers": [
          {
            "name": "prodefault-ctr",
            "image": "xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/xxxxxxxxxxx-pro-default:latest",
            "ports": [
              {
                "containerPort": 80,
                "protocol": "TCP"
              }
            ],
            "resources": {
              "limits": {
                "cpu": "1",
                "memory": "1Gi"
              },
              "requests": {
                "cpu": "150m",
                "memory": "256Mi"
              }
            },
            "volumeMounts": [
              {
                "name": "efs-pvc",
                "mountPath": "/var/www/html/homes"
              },
              {
                "name": "xero-key",
                "readOnly": true,
                "mountPath": "/var/xero"
              },
              {
                "name": "gcal-json",
                "readOnly": true,
                "mountPath": "/var/gcal"
              }
            ],
            "livenessProbe": {
              "httpGet": {
                "path": "/healthchk",
                "port": 80,
                "scheme": "HTTP"
              },
              "initialDelaySeconds": 120,
              "timeoutSeconds": 1,
              "periodSeconds": 15,
              "successThreshold": 1,
              "failureThreshold": 3
            },
            "readinessProbe": {
              "httpGet": {
                "path": "/healthchk",
                "port": 80,
                "scheme": "HTTP"
              },
              "initialDelaySeconds": 5,
              "timeoutSeconds": 3,
              "periodSeconds": 2,
              "successThreshold": 1,
              "failureThreshold": 3
            },
            "terminationMessagePath": "/dev/termination-log",
            "terminationMessagePolicy": "File",
            "imagePullPolicy": "Always"
          }
        ],
        "restartPolicy": "Always",
        "terminationGracePeriodSeconds": 30,
        "dnsPolicy": "ClusterFirst",
        "securityContext": {},
        "schedulerName": "default-scheduler"
      }
    },
    "strategy": {
      "type": "RollingUpdate",
      "rollingUpdate": {
        "maxUnavailable": "25%",
        "maxSurge": "25%"
      }
    },
    "revisionHistoryLimit": 2,
    "progressDeadlineSeconds": 600
  },
  "status": {
    "observedGeneration": 10,
    "replicas": 1,
    "updatedReplicas": 1,
    "readyReplicas": 1,
    "availableReplicas": 1,
    "conditions": [
      {
        "type": "Progressing",
        "status": "True",
        "lastUpdateTime": "2018-01-15T06:07:52Z",
        "lastTransitionTime": "2017-12-28T03:00:16Z",
        "reason": "NewReplicaSetAvailable",
        "message": "ReplicaSet \"prodefault-deployment-9685f46d4\" has successfully progressed."
      },
      {
        "type": "Available",
        "status": "True",
        "lastUpdateTime": "2018-06-13T07:12:41Z",
        "lastTransitionTime": "2018-06-13T07:12:41Z",
        "reason": "MinimumReplicasAvailable",
        "message": "Deployment has minimum availability."
      }
    ]
  }
}

【问题讨论】:

  • 所以你试图通过杀死 pod 来执行滚动更新?通常的方法是“kubectl set image”或“kubectl apply”
  • apply 或 set image 都没有任何效果,因为 :latest 在 kubectl 所知的情况下没有改变。在仪表板中杀死 pod 在 1.7 安装中始终有效,然后从存储库中获取真正的最新版本
  • 是的,您必须更改图像 - 如果您只有“最新”标签,您应该使用图像的摘要。见kubernetes.io/docs/concepts/configuration/overview/…

标签: kubernetes kops


【解决方案1】:

我注意到两个 pod 都定义了以下滚动更新策略:

"strategy": {
  "type": "RollingUpdate",
  "rollingUpdate": {
    "maxUnavailable": "25%",
    "maxSurge": "25%"
  }
},

这样,它应该在通过“set image”或“kubectl apply”在正常滚动更新中创建新pod之后终止旧pod。

因此,两个系统之间的不同行为可能来自仪表板。我猜你在两个系统中运行不同版本的dashboard,因为根据dashboard的兼容性指标,kubernetes v1.7需要dashboard 1.7支持,而kubernetes v1.8需要dashboard 1.8支持。也许版本不同的仪表板将“kill pod”视为不同的操作,我不知道。

或者,如果您在 v1.8 系统中运行仪表板 1.7,则首先尝试升级您的仪表板。

最后,不要使用 'kill pod' 进行滚动更新。

【讨论】:

  • 是的仪表板版本似乎分别是 1.7 和 1.8。不愿升级 1.7,因为目前正在工作。了解 kill pod 可能不是理想的选择,只是之前完美无缺,而且比查找/键入 id 更容易,所以习惯了它。由于 yaml 和图像链接保持不变,因此应用和设置图像从未起作用。
猜你喜欢
  • 2019-06-16
  • 1970-01-01
  • 2020-08-27
  • 1970-01-01
  • 2021-01-17
  • 1970-01-01
  • 2018-02-08
  • 2018-03-04
相关资源
最近更新 更多