Logstash 中的 Json 转换

Question

我想将 json 文档从一种结构转换为另一种结构。 我已经尝试使用多种解决方法来解决它，但不知何故我没有得到预期的输出。

我有一个json日志文件如下：

 {
    "consumer": {
        "created_at": 1566912618154,
        "username": "dummyapp",
        "id": "07912445-ca35-464b-8596-b2ace5e60481"
    },
    "service": {
        "created_at": 1567173649,
        "connect_timeout": 60000,
        "protocol": "https",
        "read_timeout": 60000,
        "port": 9090,
        "updated_at": 1567173649,
        "retries": 1,
        "write_timeout": 60000
    },
    "request": {
        "querystring": {},
        "size": "361",
        "headers": {
            "cache-control": "no-cache",
            "content-length": "0",
            "postman-token": "fb9e10e4-2f66-4126-beec-d5c1f7c06bf7",
            "user-agent": "PostmanRuntime/7.15.0",
            "accept": "/",
            "accept-encoding": "gzip, deflate",
            "connection": "keep-alive"
        },
        "method": "POST"
    }
}

我想把它改成这样：

{
    "Title":"Sample"
    "consumer": {
        "created_at": 1566912618154,
        "username": "dummyapp"
    },
    "service": {
        "created_at": 1567173649,
        "connect_timeout": 60000
    },
    "request": {
        "querystring": {},
        "headers": {
            "user-agent": "PostmanRuntime\/7.15.0"
        }
        "method": "POST"
    }
}

这是我的管道配置：

input {
    file {
        path => "Path_to_Log\test.log"
        start_position => "beginning"
        type => "json"
        codec => "json"
    }
}

filter {
    mutate {
        add_field => {
            "Title" => "Sample"
        }
        add_field => {
            "consumer" => "%{[consumer]}"
        }
        ...further changes
    }
}

output {
    file {
        path => "Output_Path\Output.log"
    }
}

这是我得到的回复：

    { 
   "@timestamp":"2019-09-10T09:20:38.569Z",
   "Title":"Sample",
   "@version":"1",
   "consumer":[ 
      [ 
         "created_at",
         1566912618154
      ],
      [ 
         "username",
         "dummyapp"
      ],
      "{\"created_at\":1566912618154,\"username\":\"dummyappapp\"}"
   ],
   "type":"json"
   ..some data
}

• 为什么我会得到类似上面的输出？
• 如何转换 json 文档以满足上述要求？
• 如何设置从源到目标子元素的标签？

Answer 1

那你为什么不直接删除你不感兴趣的字段呢？

filter {
    mutate {
        add_field => {
            "Title" => "Sample"
        }

        remove_field => [
          "[consumer][id]",
          "[service][protocol]",
          # and so on for the service element
          "[request][size]",
          "[request][headers][cache-control]",
          # and so on for the request.headers element
        ]

        # THIS IS OBSOLETE NOW!
        #add_field => {
        #    "consumer" => "%{[consumer]}"
        #}
        ...further changes
    }
}

有关如何访问事件数据和字段（尤其是嵌套的）的更​​多信息，请参阅this guide。