如何配置 FileBeat 和 Logstash 以在 Elasticsearch 中添加 XML 文件？

Question

我是这里的初学者。我自己的问题是配置 FileBeat 和 Logstash 以在 CentOS 7 上的 Elasticsearch 中添加 XML 文件。 我已经安装了最新版本的 filebeat、logstash、elasticsearch 和 Kibana，并在独立版中安装了插件“elasticsearch-head”以查看 elasticsearch 内部。为了测试我的安装，我已经成功地从 CentOS 系统（/var/log/messages）添加了简单的日志文件，并在 elasticsearch-head 插件中看到它（6 个索引和 26 个分片）： This is a viex of my elasticsearch-head plug-in

现在，下一步是从 XML 文件添加日志。阅读文档后，我配置了 filebeat 和 logstash。所有服务都在运行，我尝试使用命令“touch /mes/AddOf.xml”来尝试激活 filebeat 事件，并将日志转发到 logstash（AddOf.xml 是我的日志文件）。

对于一个日志事件，我的 XML 数据结构是这样的：

<log4j:event logger="ServiceLogger" timestamp="1494973209812" level="INFO" thread="QueueWorker_1_38a0fec5-7c7f-46f5-a87a-9134fff1b493">
    <log4j:message>Traitement du fichier \\ifs-app-01\Interfaces_MES\AddOf\ITF_MES_01_01_d2bef200-3a85-11e7-1ab5-9a50967946c3.xml</log4j:message>
    <log4j:properties>
        <log4j:data name="log4net:HostName" value="MES-01" />
        <log4j:data name="log4jmachinename" value="MES-01" />
        <log4j:data name="log4net:Identity" value="" />
        <log4j:data name="log4net:UserName" value="SOFRADIR\svc_mes_sf" />
        <log4j:data name="LogName" value="UpdateOperationOf" />
        <log4j:data name="log4japp" value="MES_SynchroService.exe" />
    </log4j:properties>
    <log4j:locationInfo class="MES_SynchroService.Core.FileManager" method="TraiteFichier" file="C:\src\MES_PROD\MES_SynchroService\Core\FileManager.cs" line="47" />
</log4j:event>

我的filebeat配置是这样的（/etc/filebeat/filebeat.yml）：

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.
- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /mes/*.xml

  document_type: message

  ### Multiline options

  # Mutiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  multiline.pattern: ^<log4j:event

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  multiline.negate: true

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  multiline.match: after

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

我的输入logstash配置（/etc/logstash/conf.d/01-beats-input.conf）：

input {
 beats {
 port => 5044
 ssl => true
 ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
 ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
 }
}

我的过滤器logstash配置（/etc/logstash/conf.d/01-beats-filter.conf）：

filter 
{
    xml 
    {
        source => "message"
        xpath => 
        [
            "/log4j:event/log4j:message/text()", "messageMES"
        ]
        store_xml => true
        target => "doc"
    }
}

我的输出logstash配置（/etc/logstash/conf.d/01-beats-output.conf）：

output {
 elasticsearch {
 hosts => ["localhost:9200"]
 sniffing => true
 manage_template => false
 index => "mes_log"
 document_type => "%{[@metadata][type]}"
 }
}

但是当我尝试命令“touch /mes/AddOf.xml”，或者在 AddOf.xml 中手动添加事件日志时，我在 elasticsearch 中看不到 XML 文件中的事件日志的新索引。

我已经看到了用于 logstash (here) 的 XML 插件的文档，但是如果我需要安装一些东西，我现在不需要？或者也许我没有为 filebeat 将日志发送到 logstash 做正确的事情？

我非常热衷于学习 ELK 堆栈。提前感谢您的专业知识和帮助。我会很感激 ！ :)

Answer 1

在您的 filebeat 配置中，multiline.pattern 的正则表达式可能应该是单引号：

multiline.pattern: '^<log4j:event'