EFK（Elasticsearch+Fluentd-(td-agent)+Kibana）：Kibana 没有显示正确的日志

Question

我已经在 redhat linux 系统（7.6 版）上安装了 EFK。 fluentd的稳定分布，即用td-agent代替fluentd。日志文件的路径（/mnt/Log/Startup.log）在 td-agent.conf 文件中配置。但在 kibana 仪表板上，它显示来自 td-agent.log 的内容，而不是日志文件（Startup.log）。

td-agent.log：

2020-09-04 16:02:16 +0530 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-elasticsearch' version '4.0.9'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-kafka' version '0.13.0'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-prometheus' version '1.8.0'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.0.2'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.3.0'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-s3' version '1.3.2'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-systemd' version '1.0.2'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-td' version '1.1.0'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.4'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-webhdfs' version '1.2.5'
2020-09-04 16:02:16 +0530 [info]: gem 'fluentd' version '1.11.1'
2020-09-04 16:02:16 +0530 [info]: 'flush_interval' is configured at out side of <buffer>. 'flush_mode' is set to 'interval' to keep existing behaviour
2020-09-04 16:02:16 +0530 [debug]: 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2020-09-04 16:02:16 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: Need substitution: false
2020-09-04 16:02:16 +0530 [debug]: 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2020-09-04 16:02:16 +0530 [info]: using configuration file: <ROOT>
  <system>
    log_level debug
  </system>
  <source>
    @type tail
    path "/mnt/Log/Startup.log"
    pos_file "/mnt/Log/Startup.log.pos"
    format multiline
    format_firstline /\d{4}-\d{1,2}-\d{1,2}/
    format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
    tag "log"
    <parse>
      format_firstline /\d{4}-\d{1,2}-\d{1,2}/
      @type multiline
      format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
      unmatched_lines
    </parse>
  </source>
  <match *.**>
    @type elasticsearch
    host "localhost"
    port 9200
    include_tag_key true
    tag_key "@log_name"
    logstash_format true
    flush_interval 10s
    <buffer>
      flush_interval 10s
    </buffer>
  </match>
</ROOT>
2020-09-04 16:02:16 +0530 [info]: starting fluentd-1.11.1 pid=67918 ruby="2.4.10"
2020-09-04 16:02:16 +0530 [info]: spawn command to main:  cmdline=["/opt/td-agent/embedded/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/embedded/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2020-09-04 16:02:17 +0530 [info]: adding match pattern="*.**" type="elasticsearch"
2020-09-04 16:02:17 +0530 [info]: #0 'flush_interval' is configured at out side of <buffer>. 'flush_mode' is set to 'interval' to keep existing behaviour
2020-09-04 16:02:17 +0530 [debug]: #0 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2020-09-04 16:02:17 +0530 [debug]: #0 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 Need substitution: false
2020-09-04 16:02:17 +0530 [debug]: #0 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
2020-09-04 16:02:17 +0530 [info]: adding source type="tail"
2020-09-04 16:02:17 +0530 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2020-09-04 16:02:17 +0530 [info]: #0 starting fluentd worker pid=67935 ppid=67930 worker=0
2020-09-04 16:02:17 +0530 [debug]: #0 buffer started instance=70139276565080 stage_size=0 queue_size=0
2020-09-04 16:02:17 +0530 [debug]: #0 enqueue_thread actually running
2020-09-04 16:02:17 +0530 [debug]: #0 tailing paths: target = /mnt/Log/Startup.log | existing =
2020-09-04 16:02:17 +0530 [info]: #0 following tail of /mnt/Log/Startup.log
2020-09-04 16:02:17 +0530 [info]: #0 fluentd worker is now running worker=0
2020-09-04 16:02:17 +0530 [debug]: #0 flush_thread actually running

虽然上面的日志显示它是在拖尾 Startup.log。它仍然在 kibana 仪表板上显示来自 td-agent.log 的内容，而不是来自配置的日志文件的内容。我还可以在 kibana 上找到默认索引

td-agent.conf：

<system>
log_level debug
</system>

<source>
@type tail
path /mnt/Log/Startup.log
pos_file /mnt/Log/Startup.log.pos
format multiline
format_firstline /\d{4}-\d{1,2}-\d{1,2}/
format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
tag log
</source>

<match *.**>
   @type elasticsearch
    host localhost
    port 9200
    include_tag_key true
    tag_key @log_name
    logstash_format true
    flush_interval 10s
</match>

版本详情：

Elasticsearch 版本 7.8.0 Fluentd (td-agent) 版本 3.8.0 Kibana 版本 7.8.0

我已经使用 rpms 安装了以上所有工具：

elasticsearch-7.8.1-x86_64.rpm td-agent-3.8.0-0.el7.x86_64.rpm kibana-7.8.1-x86_64.rpm

curl -X GET "localhost:9200/?pretty" 给出：

{
  "name" : "ncnsidapp2",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "MnfFXTyZT0ahp9u4eLSW2A",
  "version" : {
    "number" : "7.8.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "b5ca9c58fb664ca8bf9e4057fc229b3396bf3a89",
    "build_date" : "2020-07-21T16:40:44.668009Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

我可以在 kibana 上找到默认索引。

这可能是什么问题？请帮忙。

Answer 1

尝试在您的部分设置logstash_prefix。当您将 logstash_format 设置为 true 时，它​​会覆盖您设置为索引名称的内容。如果你设置了logstash_prefix，它会使索引名默认为{logstash_prefix}-%y.%m.%d的格式。