【发布时间】：2023-01-05 23:43:17
【问题描述】：
我们在 ${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT} 有 elasticsearch 集群
和导出其他 pod 日志的 k8s 集群中的 filebeat pod
有filebeat.yml：
filebeat.autodiscover:
providers:
- type: kubernetes
templates:
- condition:
equals:
kubernetes.namespace: develop
config:
- type: container
paths:
- /var/log/containers/*-${data.kubernetes.container.id}.log
exclude_lines: ["^\\s+[\\-`('.|_]"]
hints.enabled: true
hints.default_config:
type: container
multiline.type: pattern
multiline.pattern: '^[[:space:]]'
multiline.negate: false
multiline.match: after
http:
enabled: true
host: localhost
port: 5066
output.elasticsearch:
hosts: '${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}'
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
indices:
- index: "develop"
when:
equals:
kubernetes.namespace: "develop"
- index: "kubernetes-dev"
when:
not:
and:
- equals:
kubernetes.namespace: "develop"
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
- decode_json_fields:
fields: ["message"]
add_error_key: true
process_array: true
overwrite_keys: false
max_depth: 10
target: json_message
我检查过：filebeat 可以访问 kuber 上的 /var/log/containers/，但弹性集群仍然没有获得任何 develop 或 kubernetes-dev 索引。 （集群有这个索引的相对索引模板）
http://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_cluster/health?pretty:
{
"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 14,
"active_shards" : 28,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
文件拍日志：
{
"log.level": "info",
"@timestamp": "2022-11-25T08:35:18.084Z",
"log.logger": "monitoring",
"log.origin": {
"file.name": "log/log.go",
"file.line": 184
},
"message": "Non-zero metrics in the last 30s",
"service.name": "filebeat",
"monitoring": {
"metrics": {
"beat": {
"cgroup": {
"cpu": {
"stats": {
"periods": 38
}
},
"cpuacct": {
"total": {
"ns": 1576170001
}
},
"memory": {
"mem": {
"usage": {
"bytes": 4096
}
}
}
},
"cpu": {
"system": {
"ticks": 13570,
"time": {
"ms": 56
}
},
"total": {
"ticks": 23320,
"time": {
"ms": 90
},
"value": 23320
},
"user": {
"ticks": 9750,
"time": {
"ms": 34
}
}
},
"handles": {
"limit": {
"hard": 1048576,
"soft": 1048576
},
"open": 11
},
"info": {
"ephemeral_id": "a88f461e-1fd3-48d3-a1c7-3bfea1124ab9",
"uptime": {
"ms": 7530198
},
"version": "8.1.0"
},
"memstats": {
"gc_next": 17767824,
"memory_alloc": 16385608,
"memory_total": 709250600,
"rss": 104206336
},
"runtime": {
"goroutines": 23
}
},
"filebeat": {
"harvester": {
"open_files": 0,
"running": 0
}
},
"libbeat": {
"config": {
"module": {
"running": 0
}
},
"output": {
"events": {
"active": 0
}
},
"pipeline": {
"clients": 0,
"events": {
"active": 0
}
}
},
"registrar": {
"states": {
"current": 0
}
},
"system": {
"load": {
"1": 3.99,
"15": 3.91,
"5": 4.08,
"norm": {
"1": 0.4988,
"15": 0.4888,
"5": 0.51
}
}
}
},
"ecs.version": "1.6.0"
}
}
哪里会出问题？我对麋鹿绝对陌生，所以不是从一开始就使用它有点困难
【问题讨论】：
-
我可能是错的，但在你
filebeat.yml中输出kubernetes-dev我觉得你有一个奇怪的条件not+and，不应该只是not吗？ -
@保罗是对的。您不需要添加
and，它应该只是not。
标签： elasticsearch kubernetes filebeat