多行解析模式

Question

我想在 ELK stack 6.3.2 版本中解析一个标准的 JAVA 异常，它看起来像：

2018-09-04 05:29:03.955 [default task-38] ERROR c.r.e.u.util.MongoConnectionUtil.createMongoUser - Exception occured while creating mongo userCommand failed with error 11000: 'User "asdf" already exists' on server 192.168.1.33:27017. The full response is { "ok" : 0.0, "errmsg" : "User \"asdf\" already exists", "code" : 11000, "codeName" : "DuplicateKey" }
com.mongodb.MongoCommandException: Command failed with error 11000: 'User "qwer" already exists' on server 192.168.1.33:27017. The full response is { "ok" : 0.0, "errmsg" : "User \"asdf\" already exists", "code" : 11000, "codeName" : "DuplicateKey" }
    at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:115)
    at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:114) ...

我的 filebeat.yml 有配置：

filebeat.inputs:

- type: log
  enabled: true
  paths:
    - C:\logs\test.log
  multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
  multiline.negate: false
  multiline.match: after

我的 logstash.conf 输入看起来像：

input {

beats {
    port=>5044
        codec => multiline {
               pattern => "^\s"
              what => "previous"
}
}

但是 logstash 说无法解析模式，实际上它崩溃了异常。如果我只删除codec 配置，则异常的第一行将被解析。我也在https://discuss.elastic.co/t/multiline-parsing-patterns/147171 提出了同样的问题，但没有回应。

Answer 1

您需要更改： multiline.negate 到 true。 而且我不确定您要使用该模式实现什么目标，但似乎您应该使用：

multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'

此外，您无需在 logstash 中使用多行 - 只需简单：

input {
    beats {
        port => 5044            
    }
}

为了获取所有日志，我会将您的 Filebeat 配置更改为：

- type: log
  enabled: true
  paths:
    - C:\logs\test.log
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

一点解释： 当我们选择negate:true 和match: after 时，我们告诉FileBeat：

不匹配模式的连续行被追加到 匹配的上一行。

换句话说 - 它告诉 FileBeat 收集以给定模式开头的每一行，并在该模式再次出现在新行的开头时停止。
对于这种模式^[0-9]{4}-[0-9]{2}-[0-9]{2}，如果你得到这两个例外：

2018-09-04 05:29:03.955 [default task-38] ERROR c.r.e.u.util.MongoConnectionUtil.createMongoUser - Exception occured while creating mongo userCommand failed with error 11000: 'User "asdf" already exists' on server 192.168.1.33:27017. The full response is { "ok" : 0.0, "errmsg" : "User \"asdf\" already exists", "code" : 11000, "codeName" : "DuplicateKey" }
com.mongodb.MongoCommandException: Command failed with error 11000: 'User "qwer" already exists' on server 192.168.1.33:27017. The full response is { "ok" : 0.0, "errmsg" : "User \"asdf\" already exists", "code" : 11000, "codeName" : "DuplicateKey" }
    at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:115)
    at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:114) ...

2018-09-04 05:30:00.000 [default task-38] ERROR c.r.e.u.util.MongoConnectionUtil.createMongoUser - Exception occured while creating mongo userCommand failed with error 11000: 'User "asdf" already exists' on server 192.168.1.33:27017. The full response is { "ok" : 0.0, "errmsg" : "User \"asdf\" already exists", "code" : 11000, "codeName" : "DuplicateKey" }
com.mongodb.MongoCommandException: Command failed with error 11000: 'User "qwer" already exists' on server 192.168.1.33:27017. The full response is { "ok" : 0.0, "errmsg" : "User \"asdf\" already exists", "code" : 11000, "codeName" : "DuplicateKey" }
    at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:115)
    at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:114) ...

它将每个异常捕获为不同的条目日志。如果您记录更多内容并且您希望 filebeat 只收集错误，那是另一回事。在我们的程序中，我们收集所有信息并按严重性进行查询（即错误、信息、警告等）