Kubernetes Python 客户端错误 create_namespaced_binding: (409) 原因：冲突

Question

我正在使用 k8 v1.7 和 Python Client v2.0。我的自定义调度程序检测到一个挂起的 pod 并成功调度它。但是，在将 pod 分配给节点后，它会抱怨该 pod 已分配给节点，尽管它只是由调度程序本身分配的。这有什么要担心的吗？或者我该如何解决这个问题？

错误信息

create_namespaced_binding: (409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'Date': 'Tue, 19 Jun 2018 16:14:57 GMT', 'Content-Length': '289', 'Content-Type': 'application/json'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Operation cannot be fulfilled on pods/binding \"ps0-16-r935x\": pod ps0-16-r935x is already assigned to node \"blipp65\"","reason":"Conflict","details":{"name":"ps0-16-r935x","kind":"pods/binding"},"code":409}

scheduler.py

from kubernetes import client, config, watch
from kubernetes.client.rest import ApiException
config.load_kube_config()
v1 = client.CoreV1Api()

scheduler_name = 'my-custom-scheduler-v1'

def nodes_available():
    ready_nodes = []
    for n in v1.list_node().items:
        for status in n.status.conditions:
            if status.status == 'True' and status.type == 'Ready':
                ready_nodes.append(n.metadata.name)
    return ready_nodes


def scheduler(name, node, namespace='default'):
    body = client.V1Binding()

    target = client.V1ObjectReference()
    target.kind = 'Node'
    target.apiVersion = 'v1'
    target.name = node

    meta = client.V1ObjectMeta()
    meta.name = name

    body.target = target
    body.metadata = meta

    return v1.create_namespaced_binding_binding(name, namespace, body)


def main():
    w = watch.Watch()
    for event in w.stream(v1.list_namespaced_pod, 'default'):
        if event['object'].status.phase == 'Pending' and event['object'].spec.scheduler_name == scheduler_name:
            print "Pending Found"
            try:
                res = scheduler(event['object'].metadata.name,random.choice(nodes_available()))
                print "success"
            except Exception as a:
                print ("Exception when calling CoreV1Api->create_namespaced_binding: %s\n" % a)

POD YML 文件

apiVersion: v1
kind: Pod
metadata:
  name: shoeb-pod
spec:
  schedulerName: my-custom-scheduler-v1
  containers:
  - name: redis
    image: redis

于 2019 年 6 月 3 日更新

我刚刚添加了更新的 main 方法（根据@VAS 的回答，谢谢），以找到尚未安排的正确PENDING pod。请看我的回答。

Answer 1

创建 pod 时，调度程序会收到三个“待处理”事件：

1. Pod 尚未安排 ('node_name': None, 'status': {'conditions': None,...})
2. Pod 已安排好 ('node_name': 'some_node_name','status': {'conditions': [...,'status': True, 'type':'PodScheduled'],...})
3. Pod 已初始化但尚未准备好 ('node_name': 'minikube','status': {'conditions': [...,'status': True, 'type':'Initialized'], ... ,'status': False, 'type':'Ready']})

因此，您的自定义调度器应该在第一个事件时将 pod 绑定到节点，检查 pod 的状态并确保在第二个事件出现时调度它，然后在第三个事件出现时检查 pod 是否已初始化.

如果出现问题，调度器可能需要考虑之前的错误，并可能尝试将 pod 调度到不同的节点。

在您的情况下，您的调度程序会像第一个一样威胁所有三个事件，并尝试一次又一次地调度 pod。这就是您看到“pod xxx is already assigned to node yyy”错​​误的原因。

Answer 2

这里是更新后的 main 方法（根据@VAS 的回答，谢谢）找到尚未安排好的PENDING pod。

def main():
    w = watch.Watch()
    for event in w.stream(v1.list_namespaced_pod, 'default'): # default == namespace name
        # All pending pods have 3 states (not scheduled, scheduled, initialized but not ready yet)
        # We look for NOT SCHEDULED pod and conditions==None
        if event['object'].status.phase == 'Pending' and event['object'].status.conditions == None and event['object'].spec.scheduler_name == CUSTOM_SCHEDULER_NAME:
            print "Pending and Not Scheduled POD Found "+event['object'].metadata.name
            try:
                res = scheduler(event['object'].metadata.name,random.choice(nodes_available())) # nodes_available() returns all available nodes
                print "success"
            except Exception as a:
                print ("Exception when calling CoreV1Api->create_namespaced_binding: %s\n" % a)