从 external.api 获取查询时的 Prometheus Adapter Goroutine 问题答案

【问题标题】：Prometheus Adapter Goroutine issue when fetch the query from external.api从 external.api 获取查询时的 Prometheus Adapter Goroutine 问题
【发布时间】：2019-10-05 01:24:43
【问题描述】：

我已经在 k8s 环境中实现并注册了外部 api，但是在获取指标时遇到了一些问题。

这是外部 api 指标的规则：

  - seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
    resources:
      overrides:
        kubernetes_namespace: {resource: "namespace"}
        kubernetes_pod_name: {resource: "pod"}
    name:
      matches: "^(.*)_total"
      as: "${1}"

在检查 kubernetes 外部 api 指标时，资源值不为空，看起来不错，但是，

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "external.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "http_requests",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

在所有这些事情之后，当我想检查 http_requests metric kubernetes 的值时返回此错误；

Error from server (InternalError): Internal error occurred: unable to fetch metrics

我已经从prometheus-adapter pod 检查了这个问题，这些是与这个查询相关的错误日志；

E1005 01:21:26.387384       1 provider.go:47] unable to generate a query for the metric: empty query produced by metrics query template
I1005 01:21:26.387594       1 wrap.go:42] GET /apis/external.metrics.k8s.io/v1beta1/namespaces/default/http_requests: (504.956µs) 500
goroutine 1909 [running]:
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc420374930, 0x1f4)
   /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xd2
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc420374930, 0x1f4)
   /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc42105b2a0, 0x1f4)
   /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:192 +0xac
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/metrics.(*ResponseWriterDelegator).WriteHeader(0xc42107f920, 0x1f4)
   /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:307 +0x45
github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x1791e20, 0x10, 0x7f340078e6c8, 0xc4204ec580, 0x190fd20, 0xc420e8db88, 0xc421081600, 0x1f4, 0x18f9920, 0xc420039320)
   /go/src/github.com/directxman12/k8s-prometheus-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:95 +0x8d

....
....
....

但是我已经解析了这个错误信息：unable to generate a query for the metric: empty query produced by metrics query template

这个问题的原因是什么？

请帮忙

【问题讨论】：

标签： kubernetes monitoring grafana prometheus

【解决方案1】：

我认为您收到此错误：

无法为指标生成查询：空查询由指标查询模板

因为您的 api 指标不包含 metricsQuery 值。

第二件事是我不确定你为什么在http_request 中使用external.metrics。在这种情况下，您可以使用custom.metrics。提到的指标是最受欢迎的指标之一，因此您可以在网上找到许多教程，逐步了解如何实现这一点。例如this 或这个Github tutorial。

【讨论】：