使用 prometheus 为 HTTP 和 HTTPS 端口抓取 kubernetes 容器的指标

Question

我们希望我们的 Prometheus 安装能够抓取 pod 中两个容器的指标。 一个容器通过 HTTPS 在端口 443 公开指标，而另一个容器通过 HTTP 在端口 8080 公开指标。两个容器在同一路径上提供指标，即/metrics。

如果我们将 prometheus.io/scheme 声明为 http 或 https，则只会抓取一个容器。对于另一个我们总是收到：server returned HTTP status 400 Bad Request 如果我们根本不定义 prometheus.io/scheme，也会发生同样的情况。然后，Prometheus 将对两个端口都使用 http，而在 443 端口公开指标的容器会失败，因为它只需要 HTTPS 请求。

有没有办法告诉 prometheus 它应该如何准确地抓取我们部署中的各个容器？获取两个容器的指标的可行解决方法是什么？

版本

Kubernetes：1.10.2

普罗米修斯：2.2.1

部署摘录

apiVersion: apps/v1
kind: Deployment
metadata:
  name: xxx
  namespace: xxx
spec:
  selector:
    matchLabels:
      app: xxx
  template:
    metadata:
      labels:
        app: xxx
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - name: container-1
        image: xxx
        ports:
        - containerPort: 443
      - name: container-2
        image: xxx
        ports:
        - containerPort: 8080

Prometheus 配置：

- job_name: kubernetes-pods
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: pod
    namespaces:
      names: []
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    separator: ;
    regex: "true"
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    separator: ;
    regex: (.+)
    target_label: __metrics_path__
    replacement: $1
    action: replace
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    separator: ;
    regex: ([^:]+)(?::\d+)?;(\d+)
    target_label: __address__
    replacement: $1:$2
    action: replace
  - separator: ;
    regex: __meta_kubernetes_pod_label_(.+)
    replacement: $1
    action: labelmap
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: kubernetes_namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: kubernetes_pod_name
    replacement: $1
    action: replace

Answer 1

我发现了一个 GIST sn-p，如果它被命名为“metrics”，它会直接从容器中获取端口，而不是依赖于每个 pod 的注释。它还包含一个 cmets，使其成为任何以“metrics”开头的端口的正则表达式。

也许您可以扩展它以从端口名称中提取架构，例如“metrics-http”和“metrics-https”。

https://gist.github.com/bakins/5bf7d4e719f36c1c555d81134d8887eb

# Example scrape config for pods
#
# The relabeling allows the actual pod scrape endpoint to be configured via the
# following annotations:
#
# * `prometheus.io/scrape`: Only scrape pods that have a value of `true`
# * `prometheus.io/path`: If the metrics path is not `/metrics` override this. This
#    will be the same for every container in the pod that is scraped.
# * this will scrape every container in a pod with `prometheus.io/scrape` set to true and the
    port is name `metrics` in the container
# * note `prometheus.io/port` is no longer honored. You must name the port(s) to scrape `metrics`
#   Also, in some of the issues I read, there was mention of a container role, but I couldn't get 
#   that to work - or find any more info on it.
- job_name: 'kubernetes-pods'

  kubernetes_sd_configs:
  - role: pod

  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_container_port_name]
    action: keep
    regex: metrics
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [ __address__, __meta_kubernetes_pod_container_port_number]
    action: replace
    regex: (.+):(?:\d+);(\d+)
    replacement: ${1}:${2}
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name

Answer 2

您不仅限于在部署清单中添加 prometheus 注释。事实上，我将它们添加到 kubernetes 服务清单中。

这意味着您可以添加 2 个服务，一个用于容器 1，另一个用于容器 2。比特会被普罗米修斯刮掉