我们如何在 filebeat kubernetes 中过滤命名空间？

Question

我正在设置管道以将 kubernetes pod 日志发送到弹性集群。我已在集群中将 filebeat 安装为 deamonset（流：stdout），并将输出连接到 logstash。 Beats 与 logstash 连接没有问题，现在我想要来自应用程序命名空间的日志，而不是来自集群中所有命名空间的日志。谁能指导我如何在 beat 中过滤这个以及如何在 es 中查看来自 json 的源消息？

这是我的配置：

data:
  kubernetes.yml: |-
    - type: docker
      containers:
        path: "/var/lib/docker/containers"
        stream: "stdout"
        ids: "*"
        multiline.pattern: '^\s'
        multiline.match: after
      fields:
         logtype: container
      multiline:
         pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
         negate: true
         match: after
      ignore_older: 1h
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
        - decode_json_fields:
            fields: ["log"]
            overwrite_keys: true
            target: ""

kibana 中的输出：

{
  "_index": "filebeat-6.8.4-2020.03.06",
  "_type": "doc",
  "_id": "vHkzsHABJ57Tsdxxxxx",
  "_version": 1,
  "_score": null,
  "_source": {
    "log": {
      "file": {
        "path": "/var/lib/docker/containers/aa54562be9448183d69d8d2e1953e74560309176f044aed23484ac9e3260982c/sdnksdsdlsdnfsdlfslfnsdslfnsnlnflksdnflkdsfnsdflsdfndslffndslf-json.log"
      }
    },
    "tags": [
      "beats_input_codec_plain_applied",
      "_grokparsefailure"
    ],
    "input": {
      "type": "docker"
    },
    "@version": "1",
    "prospector": {
      "type": "docker"
    },
    "beat": {
      "version": "6.8.4",
      "name": "filebeat-vtp2f",
      "hostname": "filebeat-vtp2f"
    },
    "host": {
      "name": "filebeat-vtp2f"
    },
    "offset": 5798785,
    "stream": "stdout",
    "fields": {
      "logtype": "container"
    },
    "kubernetes": {
      "node": {
        "name": "k8-test-22313607-0"
      },
      "labels": {
        "version": "v1",
        "kubernetes": {
          "io/cluster-service": "true"
        },
        "controller-revision-hash": "6b56cfcb69",
        "pod-template-generation": "1",
        "k8s-app": "fluent"
      },
      "container": {
        "name": "fluentd"
      },
      "pod": {
        "uid": "72c50b54-5ef0-11ea-83e1-26018882335d",
        "name": "fluent-4lft2"
      },
      "namespace": "fluentd"
    },
    "source": "/var/lib/docker/containers/aa54562be9448183d69d8d2e1953e74560309176f044aed23484ac9e3260982c/aa54562be9448183d69d8d2e1953e74560309176f044aed23484ac9e3260982c-json.log",
    "@timestamp": "2020-03-06T14:15:18.561Z"
  },
  "fields": {
    "@timestamp": [
      "2020-03-06T14:15:18.561Z"
    ]
  },
  "highlight": {
    "prospector.type": [
      "@kibana-highlighted-field@docker@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1583504118561
  ]
}

Answer 1

我不知道如何过滤 filebeat（或者即使可能），但您可以使用 conditionals 过滤您的 logstash 配置的输出部分中的字段：

output {
    if [kubernetes][namespace] == "fluentd" {
        ...
        Send to Elasticsearch
        ...
    } else {
        ...
    }
}

通过这种方式，您可以根据 kubernetes.namespace 字段的值选择对每条消息采取不同的操作。

Answer 2

如果您希望 Filebeat 仅从某些命名空间中获取日志，请使用条件：

filebeat.yml:

    logging.level: error
    logging.json: true
    filebeat.config:
      inputs:
        # Mounted `filebeat-inputs` configmap:
        path: ${path.config}/inputs.d/*.yml
        # Reload inputs configs as they change:
        reload.enabled: false
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: false
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          templates:
          - condition:
              equals:
                kubernetes.namespace: stage
            config:
              - type: container
                paths:
                 - /var/log/containers/*${data.kubernetes.container.id}.log
                multiline.pattern: '^[[:space:]]'
                multiline.negate: false
                multiline.match: after
                include_lines: ['^{']

注意这部分：

          templates:
          - condition:
              equals:
                kubernetes.namespace: stage

我确实在每个命名空间中将 Filebeat 作为守护程序集运行。这有点额外的开销，但 Filebeat 可能很挑剔，所以这确实有助于我们首先解决其他逻辑环境中的问题。

Answer 3

如何删除一些命名空间，我记录在这里：https://ezyforanykey.blogspot.com/2020/11/filebeat-exclude-kubernetes-namespace.html

示例如下：

- type: container
      paths:
        - /var/log/containers/*.log
      exclude_files:
        - /var/log/containers/java.*
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
        - drop_event.when:
            or:
            - equals:
                kubernetes.namespace: "kube-system"
            - equals:
                kubernetes.namespace: "calico-system"