配置 dockerized fluentd 以从远程源接收日志行

Question

我正在尝试让 dockerized fluentd 通过 tcp 接收一些日志行。 可以在 fluent.conf 中为每个日志行格式添加一个正则表达式 parser 块，以将它们转换为 fluentdy time-tag-jsonmsg 格式。

但是，我无法访问日志行本身的创建方式。它们可能看起来像这样：

10:26:30 WARNING [PyClass.method:135]: Some text.

这是我已经尝试过的：

Dockerfile 没什么特别的：

FROM fluent/fluentd:v1.11-1

# This line is important. Apparently the default user is "fluent". However, root is needed for apk.
# https://github.com/fluent/fluentd-docker-image/issues/21
USER root

RUN apk add --no-cache --update --virtual .build-deps \
        sudo build-base ruby-dev \
 && sudo gem install fluent-plugin-elasticsearch \
 && sudo gem sources --clear-all \
 && apk del .build-deps \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

COPY log_fluentd/config/fluent.conf /fluentd/etc/
# COPY log_fluentd/config/entrypoint.sh /bin/  # << Led to trouble but there already is such a file.

# Default port for Forward
EXPOSE 24224
# Default port for TCP
EXPOSE 5170

USER fluent

docker-compose up-ping 该图像（项目名为 paws）通过

  server_fluentd:
    image: paws_server_fluentd:latest
    container_name: server_fluentd
    environment:
      - "FLUENTD_CONF=fluent.conf"
    ports:
      - "24224:24224"
      - "24224:24224/udp"
      - "5170:5170"
    networks:
      - paws
    volumes:
      - "paws_log:/fluentd/log"

最后，这是我从文档中提取的配置文件的相关部分：

<system>
  workers 1
  @log_level info
  root_dir /fluentd/log
</system>

<source>
  @type tcp
  @label @mainstream
  @id pawc
  tag myapp.tcp # required
  port 5170 # defaults to 5170
  bind 0.0.0.0
  # https://docs.fluentd.org/parser/regexp
  <parse>
    @type regexp
    # Example from PAWC log line: "10:26:29 INFO [trifles.config:114]: Some text."
    expression /^(?<logtime>[^\s]+) (?<loglvl>[^\s]+) \[(?<file>[^\]:]+):(?<line>\d+)\]: (?<msg>.*)$/
    time_key logtime
    time_format %H:%M:%S
    types line:integer
  </parse>
</source>

<filter **>
  @type stdout
</filter>

<label @mainstream>
  <match myapp.tcp>
    @type file
    @id output_tcp
    path /fluentd/log/tcp.*.log
    symlink_path /fluentd/log/tcp.log
  </match>
</label>

这个（和隐藏的 http 东西）导致了这个容器日志：

[info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
[info]: gem 'fluent-plugin-elasticsearch' version '4.1.1'
[info]: gem 'fluentd' version '1.11.1'
[warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
[info]: using configuration file: <ROOT> [.. shortened ..]
[info]: starting fluentd-1.11.1 pid=6 ruby="2.5.8"
[info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
[info]: adding match in @mainstream pattern="docker.**" type="file"
[info]: adding match in @mainstream pattern="myapp.tcp" type="file"
[info]: adding match in @mainstream pattern="myapp.access" type="file"
[info]: adding filter pattern="**" type="stdout"
[info]: adding source type="tcp"
[info]: adding source type="http"
[warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
[info]: #0 starting fluentd worker pid=20 ppid=6 worker=0
[info]: #0 fluentd worker is now running worker=0
  fluent.info: {"pid":20,"ppid":6,"worker":0,"message":"starting fluentd worker pid=20 ppid=6 worker=0"}
[warn]: #0 no patterns matched tag="fluent.info"
  fluent.info: {"worker":0,"message":"fluentd worker is now running worker=0"}
[warn]: #0 no patterns matched tag="fluent.info"

然后在我的主机上启动我的 python 3.7 说

data_to_send = bytes("10:26:30 WARNING [ServiceBase.subservice:135]: Some text.", 'utf-8')
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('localhost', 5170))
s.sendall(data_to_send)
s.close()

毫无反应地惩罚我。容器中至少有一个（空）文件树：

/fluentd/log # tree
.
├── http.log -> /fluentd/log/worker0/output_http/buffer/buffer.b5ac49abfabe305660f691eb4d5e782c2.log
└── worker0
    [.. shortened ..]
    └── output_tcp
        └── buffer

13 directories, 3 files [none of them about tcp]

我怀疑这与标记有关。你看到我想念什么了吗？

Answer 1

实际上一切都很简单：正则表达式希望行尾 \n 与其$ 字符匹配。