prometheus alertmanager 规则的自动标签答案

【问题标题】：automatic labels to prometheus alertmanager rulesprometheus alertmanager 规则的自动标签
【发布时间】：2021-11-23 23:34:17
【问题描述】：

我正在使用prometheus-community/prometheus 图表

我想将以下标签自动添加到任何警报管理器规则触发

env=产品
cluster=project-prod-eks

这样我就不会手动将这些标签添加到每个警报规则中。

 - alert: NGINXTooMany400s
   expr: 100 * ( sum( nginx_ingress_controller_requests{status=~"4.+"} ) / sum(nginx_ingress_controller_requests) ) > 5
   for: 1m
   labels:
     severity: warning
     env: prod
     cluster: project-prod-eks              <---------------HOW to inject them?
   annotations:
     description: Too many 4XXs
     summary: More than 5% of all requests returned 4XX, this requires your attention

这样我就可以做类似的事情

 - alert: NGINXTooMany400s
   expr: 100 * ( sum( nginx_ingress_controller_requests{status=~"4.+"} ) / sum(nginx_ingress_controller_requests) ) > 5
   for: 1m
   labels:
     severity: warning
   annotations:
     description: Too many 4XXs on {{ $labels.env }} / {{ $labels.cluster }}  <----- THIS
     summary: More than 5% of all requests returned 4XX, this requires your attention

有什么想法吗？

【问题讨论】：

标签： kubernetes prometheus prometheus-alertmanager

【解决方案1】：

您可以将external_labels 添加到您的prometheus.yml：

global:
  # The labels to add to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    env: prod
    cluster: project-prod-eks

社区图表在values.yml：

serverFiles:
  prometheus.yml:
    global:
      external_labels:
        foo: bar
    ...

【讨论】：

【解决方案2】：

所以我的做法与修改“serverFiles”略有不同，见下文

server:
  nodeSelector: 
    prometheus: "true"
  baseURL: "https://prometheus.project.io"
  enabled: true
  retention: "30d"
  strategy:
    type: RollingUpdate
  global:
    scrape_interval: 30s
    external_labels:
      env: prod
      client: client-name
      project: project-name
      cluster: project-prod-eks

【讨论】：