Logback：以 64k 优雅地拆分消息

Question

我们有一个日志收集服务，可以自动拆分 64KB 的消息，但是拆分一点也不优雅。我们将单个日志消息打印为带有一些附加元数据的 json blob。有时这些包括我们想要完整保留的大型堆栈跟踪。

所以我正在考虑编写一个自定义记录器或附加程序包装器，它可以接收消息并将其拆分为更小的块并重新记录它，但这看起来并不简单。

如果消息的大小大于某个值，是否有一种简单的方法可以配置 logback 以将其消息拆分为多个单独的消息？

这里是附加器配置：

<!-- Sumo optimized rolling log file -->
<appender name="file" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <Append>true</Append>
    <file>${log.dir}/${service.name}-sumo.log</file>
    <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
        <providers>
            <timestamp>
                <fieldName>t</fieldName>
                <pattern>yyyy-MM-dd'T'HH:mm:ss.SSS'Z'</pattern>
                <timeZone>UTC</timeZone>
            </timestamp>
            <message/>
            <loggerName/>
            <threadName/>
            <logLevel/>
            <stackTrace>
                <if condition='isDefined("throwable.converter")'>
                    <then>
                        <throwableConverter class="${throwable.converter}"/>
                    </then>
                </if>
            </stackTrace>
            <mdc/>
            <tags/>
        </providers>
    </encoder>
    <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
        <maxIndex>1</maxIndex>
        <FileNamePattern>${log.dir}/${service.name}-sumo.log.%i</FileNamePattern>
    </rollingPolicy>
    <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
        <MaxFileSize>256MB</MaxFileSize>
    </triggeringPolicy>
</appender>

<appender name="sumo" class="ch.qos.logback.classic.AsyncAppender">
    <queueSize>500</queueSize>
    <discardingThreshold>0</discardingThreshold>
    <appender-ref ref="file" />
</appender>

Answer 1

我想出的解决方案只是将我的记录器包装在可以很好地拆分消息的东西中。请注意，我主要对使用 Throwables 拆分消息感兴趣，因为它们会导致长消息。

用 Java 8 的 lambdas 编写

另外请注意，此代码未经过全面测试，如果发现任何错误，我将对其进行更新。

public class MessageSplittingLogger extends MarkerIgnoringBase {
    //Target size is 64k for split.  UTF-8 nominally has 1 byte characters, but some characters will use > 1 byte so leave some wiggle room
    //Also leave room for additional messages
    private static final int MAX_CHARS_BEFORE_SPLIT = 56000;
    private static final String ENCODING = "UTF-8";
    private Logger LOGGER;

    public MessageSplittingLogger(Class<?> clazz) {
        this.LOGGER = LoggerFactory.getLogger(clazz);
    }

    private void splitMessageAndLog(String msg, Throwable t, Consumer<String> logLambda) {
        String combinedMsg = msg + (t != null ? "\nStack Trace:\n" + printStackTraceToString(t) : "");

        int totalMessages = combinedMsg.length() / MAX_CHARS_BEFORE_SPLIT;
        if(combinedMsg.length() % MAX_CHARS_BEFORE_SPLIT > 0){
            totalMessages++;
        }

        int index = 0;
        int msgNumber = 1;
        while (index < combinedMsg.length()) {
            String messageNumber = totalMessages > 1 ? "(" + msgNumber++ + " of " + totalMessages + ")\n" : "";
            logLambda.accept(messageNumber + combinedMsg.substring(index, Math.min(index + MAX_CHARS_BEFORE_SPLIT, combinedMsg.length())));
            index += MAX_CHARS_BEFORE_SPLIT;
        }
    }

    /**
     * Get the stack trace as a String
     */
    private String printStackTraceToString(Throwable t) {
        try {
            ByteArrayOutputStream baos = new ByteArrayOutputStream();
            PrintStream ps = new PrintStream(baos, true, ENCODING);
            t.printStackTrace(ps);

            return baos.toString(ENCODING);
        } catch (UnsupportedEncodingException e) {
            return "Exception printing stack trace: " + e.getMessage();
        }
    }

    @Override
    public String getName() {
        return LOGGER.getName();
    }

    @Override
    public boolean isTraceEnabled() {
        return LOGGER.isTraceEnabled();
    }

    @Override
    public void trace(String msg) {
        LOGGER.trace(msg);
    }

    @Override
    public void trace(String format, Object arg) {
        LOGGER.trace(format, arg);
    }

    @Override
    public void trace(String format, Object arg1, Object arg2) {
        LOGGER.trace(format, arg1, arg2);
    }

    @Override
    public void trace(String format, Object... arguments) {
        LOGGER.trace(format, arguments);
    }

    @Override
    public void trace(String msg, Throwable t) {
        splitMessageAndLog(msg, t, LOGGER::trace);
    }

    //... Similarly wrap calls to debug/info/error
}