【问题标题】:Two kubernetes deployments in the same namespace are not able to communicate同一命名空间中的两个 Kubernetes 部署无法通信
【发布时间】:2019-09-27 02:34:00
【问题描述】:

我正在将 ELK 堆栈 (oss) 部署到 kubernetes 集群。 Elasticsearch 部署和服务正确启动,API 可访问。 Kibana 部署启动但无法访问 elasticsearch:

来自 Kibana 容器日志:

{"type":"log","@timestamp":"2019-05-08T22:49:26Z","tags":["error","elasticsearch","admin"],"pid":1,"message":"Request error, retrying\nHEAD http://elasticsearch:9200/ => getaddrinfo ENOTFOUND elasticsearch elasticsearch:9200"}
{"type":"log","@timestamp":"2019-05-08T22:50:44Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2019-05-08T22:50:44Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}

两个部署都在同一个命名空间“可观察性”中。我还尝试将 elasticsearch 容器引用为 elasticsearch.observability.svc.cluster.local,但它也不起作用。

我做错了什么?如何从kibana容器中引用elasticsearch容器?

更多信息:

kubectl --context=19team-observability-admin-context -n observability 获取 pod

NAME                            READY     STATUS    RESTARTS   AGE
elasticsearch-9d495b84f-j2297   1/1       Running   0          15s
kibana-65bc7f9c4-s9cv4          1/1       Running   0          15s

kubectl --context=19team-observability-admin-context -n observability 获取服务

NAME            TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
elasticsearch   NodePort   10.104.250.175   <none>        9200:30083/TCP,9300:30059/TCP   1m
kibana          NodePort   10.102.124.171   <none>        5601:30124/TCP                  1m

我用命令启动我的容器

kubectl --context=19team-observability-admin-context -n observability apply -f .\elasticsearch.yaml -f .\kibana.yaml

elasticsearch.yaml

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: observability
spec:
  type: NodePort
  ports:
  - name: "9200"
    port: 9200
    targetPort: 9200
  - name: "9300"
    port: 9300
    targetPort: 9300
  selector:
    app: elasticsearch
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: observability
spec:
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      initContainers:
      - name: set-vm-max-map-count
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ['sysctl', '-w', 'vm.max_map_count=262144']
        securityContext:
          privileged: true
        resources:
          requests:
            memory: "512Mi"
            cpu: "1"
          limits:
            memory: "724Mi"
            cpu: "1"
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.7.1
        ports:
        - containerPort: 9200
        - containerPort: 9300
        resources:
          requests:
            memory: "3Gi"
            cpu: "1"
          limits:
            memory: "3Gi"
            cpu: "1"

kibana.yaml

apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: observability
spec:
  type: NodePort
  ports:
  - name: "5601"
    port: 5601
    targetPort: 5601
  selector:
    app: observability_platform_kibana
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: observability_platform_kibana
  name: kibana
  namespace: observability
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: observability_platform_kibana
    spec:
      containers:
      - env:
        # THIS IS WHERE WE SET CONNECTION BETWEEN KIBANA AND ELASTIC
        - name: ELASTICSEARCH_HOSTS
          value: http://elasticsearch:9200
        - name: SERVER_NAME
          value: kibana
        image: docker.elastic.co/kibana/kibana-oss:6.7.1
        name: kibana
        ports:
        - containerPort: 5601
        resources:
          requests:
            memory: "512Mi"
            cpu: "1"
          limits:
            memory: "724Mi"
            cpu: "1"
      restartPolicy: Always

更新 1

正如 gonzalesraul 建议的那样,我已经为具有 ClusterIP 类型的弹性创建了第二个服务:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: elasticsearch
  name: elasticsearch-local
  namespace: observability
spec:
  type: ClusterIP
  ports:
  - port: 9200
    protocol: TCP
    targetPort: 9200
  selector:
    app: elasticsearch

服务已创建:

kubectl --context=19team-observability-admin-context -n observability 获取服务

NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
elasticsearch         NodePort    10.106.5.94     <none>        9200:31598/TCP,9300:32018/TCP   26s
elasticsearch-local   ClusterIP   10.101.178.13   <none>        9200/TCP                        26s
kibana                NodePort    10.99.73.118    <none>        5601:30004/TCP                  26s

并将弹性引用为“http://elasticsearch-local:9200”

不幸的是,它在 kibana 容器中不起作用:

{"type":"log","@timestamp":"2019-05-09T10:13:54Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch-local:9200/"}

【问题讨论】:

  • 输出是什么?有什么错误吗?您是否检查过日志并确认两个服务都在运行?
  • 输出什么?获取 pod、获取服务的输出存在问题,并且它们处于“运行”状态。 Elasticserach API 是可访问的。我检查了 Kibana 的日志,似乎“elasticsearch”主机未知(请参阅有问题的日志)
  • 我的意思是 Elasticserach API 可以通过主节点和 30083 端口访问,但在 Kibana 容器中不能作为 elasticsearch:9200 访问
  • 好的。你如何配置kibana?所有配置都在设置变量 ELASTICSEARCH_HOSTS 和 SERVER_NAME?
  • 是的,就是这样。 ELASTICSEARCH_HOSTS 应该引用 elasticsearch API 地址(参见elastic.co/guide/en/kibana/current/docker.html)。当我使用 docker-compose (gist.github.com/bocharovf/e19fa80f7b5f6b65db17249c91e79416)

标签: kubernetes


【解决方案1】:

不要使用NodePort 服务,而是使用ClusterIP。如果您需要将服务公开为 Nodeport,请另外创建第二个服务,例如:

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: elasticsearch
  name: elasticsearch-local
  namespace: observability
spec:
  type: ClusterIP
  ports:
  - port: 9200
    protocol: TCP
    targetPort: 9200
  selector:
    app: elasticsearch

然后更新 kibana 清单以指向 ClusterIP 服务:

# ...
# THIS IS WHERE WE SET CONNECTION BETWEEN KIBANA AND ELASTIC
- name: ELASTICSEARCH_HOSTS
  value: http://elasticsearch-local:9200
# ...

nodePort 服务不会在 kubernetes 上创建“dns 条目”(例如 elasticsearch.observability.svc.cluster.local)

【讨论】:

【解决方案2】:

编辑kibana.yaml 中的服务器名称值并将其设置为kibana:5601。

我认为如果你不这样做,默认情况下它会尝试去端口 80。

这就是现在的样子kibana.yaml:

...
spec:
  containers:
  - env:
    - name: ELASTICSEARCH_HOSTS
      value: http://elasticsearch:9200
    - name: SERVER_NAME
      value: kibana:5601
    image: docker.elastic.co/kibana/kibana-oss:6.7.1
    imagePullPolicy: IfNotPresent
    name: kibana
 ...

这是现在的输出:

{"type":"log","@timestamp":"2019-05-09T10:37:16Z","tags":["status","plugin:console@6.7.1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-05-09T10:37:16Z","tags":["status","plugin:interpreter@6.7.1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-05-09T10:37:16Z","tags":["status","plugin:metrics@6.7.1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-05-09T10:37:16Z","tags":["status","plugin:tile_map@6.7.1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-05-09T10:37:16Z","tags":["status","plugin:timelion@6.7.1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-05-09T10:37:16Z","tags":["status","plugin:elasticsearch@6.7.1","info"],"pid":1,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-05-09T10:37:17Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0:5601"}

更新

我刚刚在裸机集群(通过 kubeadm 引导)上对其进行了测试,然后再次工作。

这是输出:

{"type":"log","@timestamp":"2019-05-09T11:09:59Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
{"type":"log","@timestamp":"2019-05-09T11:10:01Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
{"type":"log","@timestamp":"2019-05-09T11:10:01Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
{"type":"log","@timestamp":"2019-05-09T11:10:04Z","tags":["status","plugin:elasticsearch@6.7.1","info"],"pid":1,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}
{"type":"log","@timestamp":"2019-05-09T11:10:04Z","tags":["info","migrations"],"pid":1,"message":"Creating index .kibana_1."}
{"type":"log","@timestamp":"2019-05-09T11:10:06Z","tags":["info","migrations"],"pid":1,"message":"Pointing alias .kibana to .kibana_1."}
{"type":"log","@timestamp":"2019-05-09T11:10:06Z","tags":["info","migrations"],"pid":1,"message":"Finished in 2417ms."}
{"type":"log","@timestamp":"2019-05-09T11:10:06Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0:5601"}

请注意,它从“No Living Connections”传递到“Running”。我在 GCP 上运行节点。我必须打开防火墙才能让它工作。你的环境是什么?

【讨论】:

  • 不幸的是错误是一样的。我添加了“值:kibana:5601”,但它没有任何改变。我还尝试将它与“elasticsearch:9200”和“elasticsearch-local:9200”结合使用——没有运气。如果它适用于您的环境而不适用于我的环境 - 可能是集群 DNS 问题?
  • 您是否将服务类型恢复为ClusterIP?
  • 我目前有两个服务。一个名为“elasticsearch”的 NodePort 类型和一个名为“elasticsearch-local”的 ClusterIP
  • “你的环境是什么?”。我不是管理员所以我不知道。我必须与支持/管理员交谈(需要时间)或以某种方式自行检查。如果有帮助 - kubectl 版本说:服务器版本:version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.7+1.1.2.el7", GitCommit:"02dd7eef0970a31fb7bd6e3392a3d0e411870c59", GitTreeState:"存档”,BuildDate:“2019-04-05T17:53:36Z”,GoVersion:“go1.10.8”,编译器:“gc”,平台:“linux/amd64”}
  • 没有。我的意思是你的节点在哪里运行?如果是云提供商,那么您可能需要打开正确的端口。如果在本地,我会说相同但不同的方式。我的观点是,您可能对配置没有任何问题,但使用下面的基础架构。
猜你喜欢
  • 2021-07-08
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2021-03-05
  • 2018-07-31
  • 2019-03-15
  • 2021-11-10
  • 2018-03-09
相关资源
最近更新 更多