Elasticsearch 从过滤器剖析更改字段类型

Question

我使用logstash-logback-encoder 将java 日志文件发送到logstash，然后发送到elasticsearch。为了解析 java 日志中的message，我使用以下过滤器来剖析消息

input {
  file {
    path => "/Users/MacBook-201965/Work/java/logs/oauth-logstash.log"
    start_position => "beginning"
    codec => "json"
  }
}

filter {
  if "EXECUTION_TIME" in [tags] {
    dissect {
      mapping => {
        "message" => "%{endpoint} timeMillis:[%{execution_time_millis}] data:%{additional_data}"
      }
    }
    mutate {
      convert => { "execution_time_millis" => "integer" }
    }
  }
}

output {
  elasticsearch { 
     hosts => "localhost:9200"
     index => "elk-%{+YYYY}"
     document_type => "log"
  }

  stdout {
    codec => json
  }
}

它剖析了消息，因此我可以获得execution_time_millis 的值。但是数据类型是字符串。我使用 Kibana 索引模式创建了索引。如何将execution_time_millis的数据类型改为long？

这是来自 logback 的示例 json 消息

{  
   "message":"/tests/{id} timeMillis:[142] data:2282||0:0:0:0:0:0:0:1",
   "logger_name":"com.timpamungkas.oauth.client.controller.ElkController",
   "level_value":20000,
   "endpoint":"/tests/{id}",
   "execution_time_millis":"142",
   "@version":1,
   "host":"macbook201965s-MacBook-Air.local",
   "thread_name":"http-nio-8080-exec-7",
   "path":"/Users/MacBook-201965/Work/java/logs/oauth-logstash.log",
   "@timestamp":"2018-01-04T11:20:20.100Z",
   "level":"INFO",
   "tags":[  
      "EXECUTION_TIME"
   ],
   "additional_data":"2282||0:0:0:0:0:0:0:1"
}{  
   "message":"/tests/{id} timeMillis:[110] data:2280||0:0:0:0:0:0:0:1",
   "logger_name":"com.timpamungkas.oauth.client.controller.ElkController",
   "level_value":20000,
   "endpoint":"/tests/{id}",
   "execution_time_millis":"110",
   "@version":1,
   "host":"macbook201965s-MacBook-Air.local",
   "thread_name":"http-nio-8080-exec-5",
   "path":"/Users/MacBook-201965/Work/java/logs/oauth-logstash.log",
   "@timestamp":"2018-01-04T11:20:19.780Z",
   "level":"INFO",
   "tags":[  
      "EXECUTION_TIME"
   ],
   "additional_data":"2280||0:0:0:0:0:0:0:1"
}

谢谢

Answer 1

如果您已经对文档进行了索引，则必须在更改任何字段的数据类型后重新索引数据。

但是，您可以使用类似的方法将millis 的类型从字符串更改为整数。 （这里不支持long）

https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-convert

另外，如果要添加多个索引名称具有某种正则表达式模式的索引，请尝试在创建索引之前定义弹性搜索模板。否则，您也可以预先定义索引格式，然后开始索引。