Logstash 未与 filebeat 一起运行

Question

我正在按照数字海洋指南安装 ELK 堆栈。它正在运行，但现在我遇到了logstash的问题。

当我运行调试时，我得到了这个::

tail -f /var/log/logstash/logstash.log 

{:timestamp=>"2017-02-10T10:38:05.169000-0500", :message=>"Error: Expected one of #, input, filter, output at line 1, column 1 (byte 1) after "}
{:timestamp=>"2017-02-10T10:38:05.201000-0500", :message=>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you choose\nto restart a running system."}
{:timestamp=>"2017-02-14T10:51:46.921000-0500", :message=>"fetched an invalid config", :config=>"input {\n  lumberjack {\n    port => 5044\n    type => syslog\n    ssl_certificate => \"/etc/pki/tls/certs/logstash-forwarder.crt\"\n    ssl_key => \"/etc/pki/tls/private/logstash-forwarder.key\"\n  }\n}\n\ninput {\n  beats {\n    port => 5044\n    ssl => true\n    ssl_certificate => \"/etc/pki/tls/certs/logstash-forwarder.crt\"\n    ssl_key => \"/etc/pki/tls/private/logstash-forwarder.key\"\n  }\n}\n\nfilter {\n  if [type] == \"syslog\" {\n    grok {\n      match => { \"message\" => \"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}\" }\n      add_field => [ \"received_at\", \"%{@timestamp}\" ]\n      add_field => [ \"received_from\", \"%{host}\" ]\n    }\n    syslog_pri { }\n    date {\n      match => [ \"syslog_timestamp\", \"MMM  d HH:mm:ss\", \"MMM dd HH:mm:ss\" ]\n    }\n  }\n}\n\ninput{\n  beats {\n    port => 5044\n  }\n}\n\noutput {\n  elasticsearch {\n    hosts => \"10.84.234.224:9200\"\n    manage_template => false\n    index => \"%{[@metadata][beat]}-%{+YYYY.MM.dd}\"\n    document_type => \"%{[metadata][type]}\"\n  }\n}\n\noutput {\n  elasticsearch {\n    hosts => [\"localhost:9200\"]\n    sniffing => true\n    manage_template => false\n    index => \"%{[@metadata][beat]}-%{+YYYY.MM.dd}\"\n    document_type => \"%{[@metadata][type]}\"\n  }\n\u007Fstdout { codec => rubydebug }\n}\n\noutput {\n  elasticsearch { host => localhost }\n  stdout { codec => rubydebug }\n}\n\ninput {\n  tcp {\n    port => 5400\n    type => syslog\n  }\n  udp {\n    port => 5400\n    type => syslog\n  }\n}\n\nfilter {\n  if [type] == \"syslog\" {\n    grok {\n      match => { \"message\" => \"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}\" }\n      add_field => [ \"received_at\", \"%{@timestamp}\" ]\n      add_field => [ \"received_from\", \"%{host}\" ]\n    }\n    date {\n      match => [ \"syslog_timestamp\", \"MMM  d HH:mm:ss\", \"MMM dd HH:mm:ss\" ]\n    }\n  }\n}\n\noutput {\n  elasticsearch { hosts => [\"localhost:9200\"] }\n  stdout { codec => rubydebug }\n}\n\n\n", :reason=>"Expected one of #, if, \", ', } at line 56, column 1 (byte 1278) after output {\n  elasticsearch {\n    hosts => [\"localhost:9200\"]\n    sniffing => true\n    manage_template => false\n    index => \"%{[@metadata][beat]}-%{+YYYY.MM.dd}\"\n    document_type => \"%{[@metadata][type]}\"\n  }\n", :level=>:error}

但是，我尝试删除我的伐木工人配置文件，但在我的任何配置中都找不到问题。有谁知道是否有办法重新安装logstash。我想我弄乱了太多的 confs、logstash、logstash-forwarder 等。

配置文件：

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch { 
    host => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

更新 当我运行 configtest 时，我收到此错误：

sudo service logstash configtest

/etc/init.d/logstash: 156: /etc/init.d/logstash: /opt/logstash/bin/logstash: not found

Answer 1

请注意，logstash 作为一项服务正在读取来自/etc/logstash/conf.d 的所有 文件，因此如果您只编辑一个文件，这并不意味着它会重新加载并修复其他文件。

我不确定您正在编辑什么文件，但您有许多文件被 logstash 附加在一起。

例如，一个文件有这个

input {
  tcp {
    port => 5400
    type => syslog
  }
  udp {
    port => 5400
    type => syslog
  }
}

另一个有这个

input{
  beats {
    port => 5044
  }
}

等等...所以，如错误所述，使用logstash --configtest 分别检查/etc/logstash/conf.d 中的所有文件是否存在错误。

---

如果您查看错误消息的末尾...

output {
  elasticsearch {
    hosts => [\"localhost:9200\"]
    sniffing => true
    manage_template => false
    index => \"%{[@metadata][beat]}-%{+YYYY.MM.dd}\"
    document_type => \"%{[@metadata][type]}\"
  }

它认为，您似乎缺少 output {} 的右括号，或者第 56 行的某些内容不好/缺少。

---

我不确定configtest 有什么问题，但您不需要为此提供服务。命令logstash --configtest 可以测试您的文件本身。