Logstash XML 解析失败

Question

我正在 deviantony/docker-elk 映像上运行最新的 ELK 堆栈 6.6。我有以下 XML 文件，我尝试将其解析为 ES JSON 对象：

<?xml version="1.0" encoding="UTF-8"?>
<root>
    <ChainId>7290027600007</ChainId>
    <SubChainId>001</SubChainId>
    <StoreId>001</StoreId>
    <BikoretNo>9</BikoretNo>
    <DllVerNo>8.0.1.3</DllVerNo>
</root>

我的配置文件是：

input {
  file {
    path => "/usr/share/logstash/logs/example1.xml"
    type => "xml"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline {
      pattern => "<?xml version"
      negate => true
      what => "previous"
    }
  }
}

filter {
    xml {
        source => "message"
        store_xml => false
        xpath => [ "/root/ChainId/text()", "ChainId" ]
    }
}

output {
  elasticsearch {
    hosts => "elasticsearch:9200"
    index => "xml_index"
    manage_template => false
  }
}

我的 Logstash 输出是：

{ logstash_1 | "@timestamp" => 2019-03-26T06:45:27.941Z, logstash_1 | “标签” => [ logstash_1 | [0]“多行” logstash_1 | ], logstash_1 | "主机" => "751b3a8bf341", logstash_1 | "链ID" => [], logstash_1 | "message" => "\r\n\r\n 7290027600007\r\n 001\r\n 001\r\n 9\r\n 8.0.1.3\r\n \r", logstash_1 | "路径" => "/usr/share/logstash/logs/example1.xml", logstash_1 | "@version" => "1", logstash_1 | “类型” => “xml” logstash_1 | }

消息下的 XML 正文显示为带有转义和 \r\n 的字符串。 XPathChainId 字段返回空数组。我也尝试使用其他 XML 文件，结果相同。

更新： 在尝试删除 \r\n 后仍然没有得到 XPath 解析的字段。我的输出是：

logstash_1 | "message" => "7290027600007001001 98.0.1.3",
logstash_1 | "StoreId" => [],
logstash_1 | "BikoretNo" => [],
logstash_1 | "链ID" => [],
logstash_1 | “类型” => “xml”，
logstash_1 | “标签” => [
logstash_1 | [0]“多行”
logstash_1 | ],
logstash_1 | "@timestamp" => 2019-03-27T20:51:09.575Z,
logstash_1 | "DllVerNo" => [],
logstash_1 | "路径" => "/usr/share/logstash/logs/example1.xml",
logstash_1 | "主机" => "751b3a8bf341",
logstash_1 | "子链 ID" => [],
logstash_1 | “@版本”=>“1”
logstash_1 | }

Answer 1

请使用gsub mutate 过滤器从消息中删除特殊字符。

mutate { 
        gsub => [ "message", "[\r\n]", "" ] 
    }

将目标设置添加到用于放置数据的 xml 过滤器。

filter {

    xml{
        source => "message"
        store_xml => false
        target => "root"

    }

}

这是完整的工作 logstash conf 文件。

input
{
    file
        {
            path => "C:\Users\KZAPAGOL\Desktop\CSV\XMLFile.xml"
            start_position => "beginning"
            sincedb_path => "/dev/null"
            exclude => "*.gz"
            type => "xml"
            codec => multiline {
                    pattern => "<?xml " 
                    negate => "true"
                    what => "previous"
                }
        }
}

filter {

    xml{
        source => "message"
        store_xml => false
        target => "root"
        xpath => [
            "/root/ChainId/text()", "ChainId",
            "/root/SubChainId/text()", "SubChainId",
            "/root/StoreId/text()", "StoreId",
            "/root/BikoretNo/text()", "BikoretNo",
            "/root/DllVerNo/text()", "DllVerNo"
        ]
    }

    mutate { 
        gsub => [ "message", "[\r\n]", "" ] 
    }
}

output{

elasticsearch{
        hosts => ["http://localhost:9200/"]
        index => "parse_xml"
    }

    stdout
    {
        codec => rubydebug
    }
}

输出

{
  "_index": "parse_xml",
  "_type": "doc",
  "_id": "vNj4v2kBZ2Q_C9FO94eF",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-03-27T16:25:58.379Z",
    "path": "filePath",
    "tags": [
      "multiline"
    ],
    "ChainId": [
      "7290027600007"
    ],
    "BikoretNo": [
      "9"
    ],
    "DllVerNo": [
      "8.0.1.3"
    ],
    "host": "xxxx",
    "@version": "1",
    "SubChainId": [
      "001"
    ],
    "message": "<?xml version=\"1.0\" encoding=\"UTF-8\"?><root>    <ChainId>7290027600007</ChainId>    <SubChainId>001</SubChainId>    <StoreId>001</StoreId>    <BikoretNo>9</BikoretNo>    <DllVerNo>8.0.1.3</DllVerNo></root>",
    "type": "xml",
    "StoreId": [
      "001"
    ]
  },
  "fields": {
    "@timestamp": [
      "2019-03-27T16:25:58.379Z"
    ]
  },
  "sort": [
    1553703958379
  ]
}

Answer 2

我试过你的配置，它在windows环境下工作，它曾经和我一起发生过，我改变了xpath表达式

尝试将 xpath 表达式更改为以下之一

xpath => [ "//*[local-name() = 'ChainId']/text()", "ChainId" ]

或

xpath => [ "//ChainId/text()", "ChainId" ]

Answer 3

我的 XML 文件被编码为 UTF-8 BOM 而不是 UTF-8。问题解决了！