如何使用 Json 文件中的 logstash 获取时间戳？ JSON中有多个日期字段答案

【问题标题】：How to get TimeStamp using logstash from Json file? There multiple date fields in the JSON如何使用 Json 文件中的 logstash 获取时间戳？ JSON中有多个日期字段
【发布时间】：2017-10-11 09:39:10
【问题描述】：

我的 JSON 输入如下，其中包含日期字段，需要从 Json 中提取日期时间字段，

{
  "Properties": {
         "Client Name": "Chubb",
         "Portfolio": "Chubb-Transfer"
  },
"Capture": [
         {
            "CaptureGUID": "caa1f5ba-1e93-4926-b3ac-e30d0d9d4cbb",
            "HTMLPath": "Captures\\C:\\",
            "ScreenName": "Amdocs CRM - ClearCallCenter - [Console]",
            "TimeStamp": "20170926110036"
          },
         {
            "CaptureGUID": "0faf6b54-999f-4bfd-b8d0-e81a589f9185",
            "HTMLPath": "Captures\\C:\\",
            "ScreenName": "Microsoft Excel - 1.0.1 1.0.6 1.0.8 Match 3.0.6 Hit NAIC Optimized.xlsx",
            "TimeStamp": "20170926105418"
          }
     ]
}

我的 Logstash 配置如下，如何将字符串日期（“TimeStamp”：“20170926105418”）转换为日期格式。已更新完整的 Logstash 文件

input {
    file { 
        type => "json"
        path => "C:/ELK/data/Recordings/*.json"
        start_position => beginning
        codec => multiline {
           pattern => "^{"
           negate => "true"
           what => "previous"
           multiline_tag => "multi_tagged"
           max_lines => 30000
       }
    }
}
filter{
    date {
        match => ["Capture.TimeStamp", "yyyyMMddHHmmss"]
        target => "TimeStamp"
    }

    mutate { 
    replace => { "message" => "%{message}}" }
    gsub => [ 'message','\n','']
    }

    json { 
        source => "message" 
        remove_field => ["message"]
    }


}

output {
    elasticsearch {
    index => "test10"
    }
    stdout { codec => rubydebug }
}

【问题讨论】：

标签： json elasticsearch logstash

【解决方案1】：

从 logstash 配置文件中删除日期过滤器。在映射索引时处理日期解析。以下是您的用例的映射。

PUT json
{
  "mappings": {
    "json": {
      "properties": {
        "Capture": {
          "type": "nested",
          "properties": {
            "CaptureGUID": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "HTMLPath": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "ScreenName": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "TimeStamp": {
              "type": "date",
              "format": "yyyyMMddHHmmss"
            }
          }
        },
        "Properties": {
          "properties": {
            "Client Name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "Portfolio": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        }
      }
    }
  }
}

【讨论】：

不，我试过了，它甚至没有考虑到，输出如下，{“Properties”=> {“Client Name”=>“Chubb”，“Portfolio”=>“Chubb- Transfer" }, "Capture" => [ { "CaptureGUID"=> "caa1f5ba-1e93-4926-b3ac-e30d0d9d4cbb", "HTMLPath"=> "Captures\\C:\\", "ScreenName"=> docs CRM - ClearCallCenter - [Console]", "TimeStamp"=> 20170926110036"}]
是的已经粘贴了
嗨，谢谢，把它放在我的 Logstash 配置文件的什么位置。
没有。这是索引映射，就 sql 而言，它就像“创建表等等……”。在 kibana 中运行它，但请确保首先删除现有映射。
是的，我已经在 Kibana 中直接对其进行了测试，它可以正常工作。但是windows中有没有办法自动读取一个文件夹来检测json并开始索引，就像logstash一样。

【解决方案2】：

已通过以下方式解决，

input {
    file { 
        type => "json"
        path => "C:/ELK/data/Recordings/*.json"
        start_position => beginning
        codec => multiline {
           pattern => "^{"
           negate => "true"
           what => "previous"
           max_lines => 30000
       }
    }
}
filter{

    mutate { 
    replace => { "message" => "%{message}}" }
    gsub => [ 'message','\n','']
    }

    json { 
    source => "message" 
    remove_field => ["message"]
    }

    date {
    match => ["[Capture][0][TimeStamp]", "yyyyMMddHHmmss"]
    target=> "[Capture][0]StartTime"
    timezone => "Africa/Lome"
    locale => "en" 
    }


}

output {
    elasticsearch {
    index => "test15"
    }
    stdout { codec => rubydebug } 
}

【讨论】：