如何在 fluentd tail/s3 插件中删除 unicode

Question

我有 fluentd 配置，源为尾部类型，目标为 aws s3。 我可以将日志存储在 S3 中。

我们已根据 winston 记录器中的日志级别在应用程序日志中启用着色，但在存储在 S3 中时，我得到了颜色的 unicode 值，例如 \u001b[34mdebug\u001b[ 39米。特殊字符也会发生同样的情况（\u003c for >）

Fluentd Config
--------------
    <source>
      @type tail
      path /var/log/containers/abc-*.log
      pos_file /var/log/abc.log.pos
      tag abc.**
      <parse>
        @type none
      </parse>
      read_from_head true
    </source>
    
    <match abc.**>
       @type s3
    
       aws_key_id "#{ENV['AWS_ACCESS_KEY']}"
       aws_sec_key "#{ENV['AWS_SECRET_ACCESS_KEY']}"
       s3_bucket "#{ENV['S3_LOGS_BUCKET_NAME']}"
       s3_region "#{ENV['S3_LOGS_BUCKET_REGION']}"
       path "#{ENV['S3_LOGS_BUCKET_PREFIX']}"
       s3_object_key_format %{path}/abc/%Y%m%d/%{index}.json
    
       buffer_chunk_limit 20m
       buffer_path /var/log/fluentd-buffer
       store_as json
       flush_interval 600s
       time_slice_format %Y/%m/%d
       utc
    
       <format>
          @type single_value
       </format>
    
       <instance_profile_credentials>
         ip_address 169.254.169.254
         port       80
       </instance_profile_credentials>
    </match>

存储在 S3 中的当前日志：

{"log":"2021-04-10T12:34:51.050Z - \u001b[34mdebug\u001b[39m: \u003e\u003e\u003e\u003e testlog1 from app \n","stream":"stdout","time":"2021-04-10T12:34:51.050571552Z"}
{"log":"2021-04-10T12:34:51.067Z - \u001b[34mdebug\u001b[39m: \u003c\u003c\u003c\u003c testlog2 from app\n","stream":"stdout","time":"2021-04-10T12:34:51.068105637Z"}

预期

{"log":"2021-04-10T12:34:51.050Z - debug: <<<< exec start from app \n","stream":"stdout","time":"2021-04-10T12:34:51.050571552Z"}
{"log":"2021-04-10T12:34:51.067Z - debug: <<<< exec end from app\n","stream":"stdout","time":"2021-04-10T12:34:51.068105637Z"}

需要有关如何打印原始值的帮助。

Answer 1

试试像这样的流利的record_transformer过滤插件：

<filter abc.**>
  @type record_transformer
  enable_ruby true
  <record>
    message ${ record["message"].gsub(/(\\u\d+b\[\d+m)|(\\u\d+e)/, '') }
  </record>
</filter>