Logstash grok 过滤器不解析消息答案

【问题标题】：Logstash grok filter does not parse messageLogstash grok 过滤器不解析消息
【发布时间】：2017-09-04 13:20:52
【问题描述】：

在 grok 调试器中一切正常，但在我启动 logstash 时无法解析。

日志行：

# Time: 170904 10:16:01\n# User@Host: mmcite[mmcite] @ localhost []\n# Thread_id: 18712  Schema: mmcite  QC_hit: No\n# Query_time: 0.502068  Lock_time: 0.000030  Rows_sent: 0  Rows_examined: 1\n# Rows_affected: 1\nuse mmcite;\nSET timestamp=1504512961;\nUPDATE `ajedem456456` SET `id`='cotjo4mim2j7fp3ui2kit7gns6' WHERE id='pvueh0rm6l2meiguootdfqsan7';

过滤器：

grok {

    match => { "message" => "#%{SPACE}Time:%{SPACE}%{NUMBER}%{SPACE}%{TIME}(.|\n)*%{HOSTNAME}\[%{HOSTNAME:mysql_host}\]%{SPACE}@%{SPACE}localhost \[\](.|\n)*#%{SPACE}Thread_id:%{SPACE}%{NUMBER}%{SPACE}Schema:%{SPACE}%{WORD}%{SPACE}%{WORD}:%{SPACE}%{WORD}(.|\n)*#%{SPACE}Query_time:%{SPACE}%{BASE16FLOAT:mysql_query_time}%{SPACE}Lock_time:%{SPACE}%{BASE16FLOAT:mysql_lock_time}%{SPACE}Rows_sent:%{SPACE}%{NUMBER:mysql_rows_sent}%{SPACE}Rows_examined:%{SPACE}%{NUMBER:mysql_rows_examined}(.|\n)*%{SPACE}Rows_affected:%{SPACE}%{NUMBER:mysql_rows_affected}(.|\n)*%{WORD}%{SPACE}%{WORD};(.|\n)*SET%{SPACE}timestamp=%{NUMBER:timestamp};\\n%{GREEDYDATA:mysql_query}" }

}

输出：

{
    "@timestamp" => 2017-09-04T13:08:06.260Z,
        "offset" => 3441,
      "@version" => "1",
    "input_type" => "log",
          "beat" => {
        "hostname" => "server.jerewan.cz",
            "name" => "server.jerewan.cz",
         "version" => "5.1.1"
    },
          "host" => "server.jerewan.cz",
        "source" => "/usr/home/admin/filebeat/mysql.slow.log",
       "message" => "# Time: 170904 10:16:01\n# User@Host: mmcite[mmcite] @ localhost []\n# Thread_id: 18712  Schema: mmcite  QC_hit: No\n# Query_time: 0.502068  Lock_time: 0.000030  Rows_sent: 0  Rows_examined: 1\n# Rows_affected: 1\nuse mmcite;\nSET timestamp=1504512961;\nUPDATE `PAJKA` SET `id`='cotjo4mim2j7fp3ui2kit7gns6' WHERE id='pvueh0rm6l2meiguootdfqsan7';",
          "type" => "mysql_slow_log",
          "tags" => [
        [0] "beats_input_codec_plain_applied",
        [1] "_grokparsefailure"
    ]
}

非常感谢您的帮助。

【问题讨论】：

标签： logstash logstash-grok logstash-configuration

【解决方案1】：

我不知道如何，但它有效。

grok {    
    match => { "message" => "#%{SPACE}Time:%{SPACE}%{NUMBER}%{SPACE}%{TIME}(.|\n)*%{HOSTNAME}\[%{HOSTNAME:mysql_host}\]%{SPACE}@%{SPACE}localhost \[\](.|\n)*#%{SPACE}Thread_id:%{SPACE}%{NUMBER}%{SPACE}Schema:%{SPACE}%{WORD}%{SPACE}%{WORD}:%{SPACE}%{WORD}(.|\n)*#%{SPACE}Query_time:%{SPACE}%{BASE16FLOAT:mysql_query_time}%{SPACE}Lock_time:%{SPACE}%{BASE16FLOAT:mysql_lock_time}%{SPACE}Rows_sent:%{SPACE}%{NUMBER:mysql_rows_sent}%{SPACE}Rows_examined:%{SPACE}%{NUMBER:mysql_rows_examined}(.|\n)*%{SPACE}Rows_affected:%{SPACE}%{NUMBER:mysql_rows_affected}(.|\n)*%{WORD}%{SPACE}%{WORD};(.|\n)*SET%{SPACE}timestamp=%{NUMBER:timestamp};(.|\n)%{GREEDYDATA:mysql_query}" }    
}

我已将%{GREEDYDATA:mysql_query} 之前的\\n 替换为(.|\n)。

【讨论】：