使用 python 正则表达式拆分多行日志条目答案

【问题标题】：Split out multiline log entries with python regular expressions使用 python 正则表达式拆分多行日志条目
【发布时间】：2018-03-14 07:49:08
【问题描述】：

我需要在 python 中创建一个正则表达式，它可以获取以下示例并拆分每个日志条目。我使用日期来识别每个日志条目的开头，但它只能获取从日期开始到第一行结尾的单行。它完全错过了所有堆栈跟踪内容。我想要所有的日志条目，因为有很多重复的日志记录，我希望能够过滤掉重复并将其减少到少数独特的日志条目。一旦我确定了一个日志条目，我还希望能够删除关于字符串的任何独特内容，例如日期时间戳，以便比较函数可以将其标记为重复项。我尝试使用积极的前瞻和多行标志，但无济于事。有人知道我想做什么吗？

我尝试过的一些正则表达式

^\d{4}-\d{2}-\d{2}.*\(.*\)$ // it matches single line date to parenthesis
^(\d{4}-\d{2}-\d{2}|\s|).*\)$ // matches single line with tabs - not much better
^\d{4}-\d{2}-\d{2}.*(?=\d{4}-\d{2}-\d{2}) // positive lookahead but barely works

示例字符串：

2018-03-06 11:36:40:048 INFO:Starting.  (com.X.s.f.o.o)
2018-03-06 11:36:42:931 SEVERE: Error attempting to s: StatusRuntimeException   (com.Y.W.Z_H.ZHGC.sHToVe)
io.G.StatusRuntimeException: EXCEEDED
    at io.G.stub.CCalls.toStatusRuntimeException(CCalls.java:227)
    at io.G.stub.CCalls.getUnchecked(CCalls.java:208)
    at io.G.stub.CCalls.blockingUnaryCall(CCalls.java:141)
2018-03-06 11:36:46:159 SEVERE: Error attempt: StatusRuntimeException   (com.Y.W.Z_H.ZHGC.sHToVe)
io.G.StatusRuntimeException: EXCEEDED
    at io.G.stub.CCalls.toStatusRuntimeException(CCalls.java:227)
    at io.G.stub.CCalls.getUnchecked(CCalls.java:208)
    at io.G.stub.CCalls.blockingUnaryCall(CCalls.java:141)
2018-03-06 11:36:46:824 SEVERE: getConfigInteger(): eGSWindowsPortNumber    (com.Y.W.Y_Z_config_s.YZConfigs.getInteger)
2018-03-06 11:36:46:844 SEVERE: Failed to get (com.Y.W.Z_H.ZHGC.create)

期望的输出：

第 1 场比赛：

INFO:Starting.  (com.X.s.f.o.o)

第 2 场比赛：

SEVERE: Error attempting to s: StatusRuntimeException   (com.Y.W.Z_H.ZHGC.sHToVe)
io.G.StatusRuntimeException: EXCEEDED
    at io.G.stub.CCalls.toStatusRuntimeException(CCalls.java:227)
    at io.G.stub.CCalls.getUnchecked(CCalls.java:208)
    at io.G.stub.CCalls.blockingUnaryCall(CCalls.java:141)

第 3 场比赛：

SEVERE: Error attempt: StatusRuntimeException   (com.Y.W.Z_H.ZHGC.sHToVe)
io.G.StatusRuntimeException: EXCEEDED
    at io.G.stub.CCalls.toStatusRuntimeException(CCalls.java:227)
    at io.G.stub.CCalls.getUnchecked(CCalls.java:208)
    at io.G.stub.CCalls.blockingUnaryCall(CCalls.java:141)

第 4 场比赛：

SEVERE: getConfigInteger(): eGSWindowsPortNumber    (com.Y.W.Y_Z_config_s.YZConfigs.getInteger)

第 5 场比赛：

SEVERE: Failed to get (com.Y.W.Z_H.ZHGC.create)

【问题讨论】：

标签： python regex logging multiline

【解决方案1】：

无需尝试将整个字符串与正则表达式匹配，您只需匹配日期并使用它将字符串分隔到所需的日志中：

import re

sample="""2018-03-06 11:36:40:048 INFO:Starting.  (com.X.s.f.o.o)
2018-03-06 11:36:42:931 SEVERE: Error attempting to s: StatusRuntimeException   (com.Y.W.Z_H.ZHGC.sHToVe)
io.G.StatusRuntimeException: EXCEEDED
    at io.G.stub.CCalls.toStatusRuntimeException(CCalls.java:227)
    at io.G.stub.CCalls.getUnchecked(CCalls.java:208)
    at io.G.stub.CCalls.blockingUnaryCall(CCalls.java:141)
2018-03-06 11:36:46:159 SEVERE: Error attempt: StatusRuntimeException   (com.Y.W.Z_H.ZHGC.sHToVe)
io.G.StatusRuntimeException: EXCEEDED
    at io.G.stub.CCalls.toStatusRuntimeException(CCalls.java:227)
    at io.G.stub.CCalls.getUnchecked(CCalls.java:208)
    at io.G.stub.CCalls.blockingUnaryCall(CCalls.java:141)
2018-03-06 11:36:46:824 SEVERE: getConfigInteger(): eGSWindowsPortNumber    (com.Y.W.Y_Z_config_s.YZConfigs.getInteger)
2018-03-06 11:36:46:844 SEVERE: Failed to get (com.Y.W.Z_H.ZHGC.create)"""

def date_match(s):
    """Returns true if the beginning of this string matches a date and time."""
    return bool(re.match("\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}", s))

def yeild_matches(full_log):
    log = []
    for line in full_log.split("\n"):
        if date_match(line): # if this line starts with a date
            if len(log) > 0: # if theres already a log...
                yield "\n".join(log) # ... yield the log ...
                log = [] # ... and set the log back to nothing.

        log.append(line) # add the current line to log (list)

    yield "\n".join(log) # return the last log (theres no date at the end of the string to end the last log)

logs = list(yeild_matches(sample))

for i, l in enumerate(logs):
    print("Match {}:\n{}\n".format(i + 1, l))

yield_matches 会将每一行添加到名为log 的列表中，直到找到另一个日期。当它找到一个日期时，它yields 当前日志，并将日志设置为空。

输出如下：

Match 1:
2018-03-06 11:36:40:048 INFO:Starting.  (com.X.s.f.o.o)

Match 2:
2018-03-06 11:36:42:931 SEVERE: Error attempting to s: StatusRuntimeException   (com.Y.W.Z_H.ZHGC.sHToVe)
io.G.StatusRuntimeException: EXCEEDED
    at io.G.stub.CCalls.toStatusRuntimeException(CCalls.java:227)
    at io.G.stub.CCalls.getUnchecked(CCalls.java:208)
    at io.G.stub.CCalls.blockingUnaryCall(CCalls.java:141)

Match 3:
2018-03-06 11:36:46:159 SEVERE: Error attempt: StatusRuntimeException   (com.Y.W.Z_H.ZHGC.sHToVe)
io.G.StatusRuntimeException: EXCEEDED
    at io.G.stub.CCalls.toStatusRuntimeException(CCalls.java:227)
    at io.G.stub.CCalls.getUnchecked(CCalls.java:208)
    at io.G.stub.CCalls.blockingUnaryCall(CCalls.java:141)

Match 4:
2018-03-06 11:36:46:824 SEVERE: getConfigInteger(): eGSWindowsPortNumber    (com.Y.W.Y_Z_config_s.YZConfigs.getInteger)

Match 5:
2018-03-06 11:36:46:844 SEVERE: Failed to get (com.Y.W.Z_H.ZHGC.create)

【讨论】：

谢谢！的确，给猫剥皮的方法不止一种。感谢您的输入。 :)
我会扩展它以删除字符串中的唯一部分，例如日期/时间。

【解决方案2】：

通过以下信息，我能够弄清楚：

python: multiline regular expression

https://www.safaribooksonline.com/library/view/python-cookbook-3rd/9781449357337/ch02s08.html

如果以下正则表达式以日期 ^\d{4}-\d{2}-\d{2} 开头并继续向前查找 (?=...) 直到第一次找到另一个日期条目 .+? 并将其作为匹配项返回，则它匹配日志条目。这匹配多行字符串！ :D

^\d{4}-\d{2}-\d{2}.+?(?=\d{4}-\d{2}-\d{2})

以下正则表达式将与@Sean Breckenridge 的解决方案执行相同的操作，但这次摆脱了我要摆脱的字符串的独特部分。很有用！

(?<=\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}:\d{3}).+?(?=\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}:\d{3}|\Z)

【讨论】：