从 Prometheus alertmanager 向 Google 聊天室发送通知答案

【问题标题】：send notification from Prometheus alertmanager to Google Chat room从 Prometheus alertmanager 向 Google 聊天室发送通知
【发布时间】：2021-02-23 13:16:10
【问题描述】：

我们最近开始在我们的组织中使用 Google Chat，因此我们需要将警报从 Prometheus-Alertmanager 发送到 Google Chat 房间。下面是我用于 Google 聊天的 alertmanager 配置

receiver:
    - name: 'gchat-receiver'
      webhook_configs:
        - url: "https://chat.googleapis.com/v1/spaces/AAAArcv5Snw/messages?key=KEY&token=TOKEN"
          send_resolved: false

当我们实施上述配置时，我们的警报管理器日志中出现以下错误：

level=error ts=2021-02-23T06:23:43.931Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="gchat-receiver/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: https://chat.googleapis.com/v1/spaces/AAAArcv5Snw/messages?key=KEY&token=TOKEN"
level=error ts=2021-02-23T06:23:43.971Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="gchat-receiver/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: https://chat.googleapis.com/v1/spaces/AAAArcv5Snw/messages?key=KEY&token=TOKEN"
level=error ts=2021-02-23T06:23:43.976Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=21 err="gchat-receiver/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: https://chat.googleapis.com/v1/spaces/AAAArcv5Snw/messages?key=KEY&token=TOKEN"
level=error ts=2021-02-23T06:23:43.994Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="gchat-receiver/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: https://chat.googleapis.com/v1/spaces/AAAArcv5Snw/messages?key=KEY&token=TOKEN"

对此的任何帮助或指导将不胜感激

【问题讨论】：

标签： kubernetes prometheus prometheus-alertmanager google-chat

【解决方案1】：

这是因为 Google Chat 期望将特定的 JSON 负载架构发布到该 webhook 地址，而默认的 Alertmanager payload 不匹配，因此您会收到 HTTP 400 response: bad request。

您可能需要在两者之间放置一个自定义应用程序，以将 JSON 转换为 Google Chat 可接受的格式。 Github 中已经有一些符合该一般标准的存储库。（例如calert）

【讨论】：