Logstash 不能添加字段？

Question

我一直在使用 logstash 来读取一些数据库恢复日志。这是几行示例记录。

07/08/2016  6:33:22.50: START restore database                      
SQL2540W  Restore is successful, however a warning "2539" was encountered 
during Database Restore while processing in No Interrupt mode.
07/08/2016  6:33:28.93: END restore database                        
SQL4406W  The DB2 Administration Server was started successfully.
07/08/2016  6:35:35.29: END restart server                          
connect reset
DB20000I  The SQL command completed successfully.
07/08/2016  6:35:38.48: END p:\s6\source\system\CMD\res_uw.cmd      

这是我的 conf 文件的过滤器部分。

if ([message] =~ /Backup successful/){
    grok{
        match => {"message" => ['%{GREEDYDATA:Message}'] }
    }
    mutate {
        add_tag => "send_to_es"
        add_field => {"Timestamp" => "%{GREEDYDATA:DATETIME}"}
    }
}
if ([message] =~ /warning "2539"/){
    grok{
        match => {"message" => ['%{GREEDYDATA:Message}'] }
    }
    mutate {
        add_tag => "send_to_es"
        add_field => {"Timestamp" => "%{GREEDYDATA:DATETIME}"}
    }
}
if ([message] =~ /(END p:|END P:)/){
    grok{
        match => {"message" => ['%{GREEDYDATA:DATETIME}:%{SPACE}END%{SPACE}%{GREEDYDATA:Mis}'] }
        remove_field => "%{GREEDYDATA:Mis}"
    }
    mutate {
        add_tag => "send_to_es"
    }
}   

我想将从我的记录的最后一行提取的数据“DATETIME”添加到消息中以同时索引其他消息。但是，它无法成功添加该字段。输出会变成

      "message": "SQL2540W  Restore is successful, however a warning \"2539\" was encountered \r\r",
      "@version": "1",
      "@timestamp": "2016-07-12T02:28:52.337Z",
      "path": "C:/CIGNA/hkiapp67_db_restore/res_uw.log",
      "host": "SIMSPad",
      "type": "txt",
      "Message": "SQL2540W  Restore is successful, however a warning \"2539\" was encountered \r\r",
      "Timestamp": "%{GREEDYDATA:DATETIME}",
      "tags": [
        "send_to_es"
      ]

我该如何解决这个问题？

Answer 1

Logstash 在收到一条线路时，不知道任何其他线路。您必须使用多行编解码器/过滤器将您需要的所有行重新组合为带有日期的行。然后使用 grok 过滤器提取日期并将其添加到文档中。

多行编解码器/过滤器的配置如下所示：

multiline {
  pattern => "%{DATE}"
  negate => "true"
  what => "next"
}

这样，所有不以模式DATE 开头的行都将与下一行连接。