在巨大的文本文件中查找和替换多行

Question

我有一个巨大的游戏服务器日志文件（将近 6 GB），其中包含数百万个错误（当时每秒产生数百个错误），此外还有需要保留的有用记录。我想删除所有包含错误的行，同时保留显示聊天消息或其他信息的行。

但是，我不能轻易删除要转储的行，因为错误消息并不总是相同的，并且总是需要不同数量的行。简而言之，我根本无法确定哪些行包含错误。我需要一个正则表达式来做到这一点。我一直在寻找适合我目的的程序。不过，我还没有找到。例如，sed（流编辑器）可以完成这样的工作，因为它不需要太多资源来处理如此巨大的文件。但是，它不支持多行查找和替换。

因此，是否有支持在多行大文本文件中查找和替换正则表达式的程序？还是建议您编写自己的脚本来完成这项工作？

日志文件如下：

2011-03-02 01:43:00 [INFO] <admin> CraftBook is causing errors. 
2011-03-02 01:43:01 [SEVERE] Could not pass event REDSTONE_CHANGE to CraftBookMechanisms
java.lang.NoSuchMethodError: com.sk89q.worldedit.blocks.BlockType.isRedstoneBlock(I)Z
    at com.sk89q.craftbook.bukkit.MechanicListenerAdapter$MechanicBlockListener.onBlockRedstoneChange(MechanicListenerAdapter.java:174)
    at net.minecraft.server.BlockButton.a(BlockButton.java:170)
    at net.minecraft.server.ItemInWorldManager.a(ItemInWorldManager.java:160)
    at net.minecraft.server.NetServerHandler.a(NetServerHandler.java:482)
    at net.minecraft.server.Packet15Place.a(SourceFile:57)
    at net.minecraft.server.NetworkManager.a(SourceFile:230)
    at net.minecraft.server.NetServerHandler.a(NetServerHandler.java:75)
    at net.minecraft.server.NetworkListenThread.a(SourceFile:100)
    at net.minecraft.server.MinecraftServer.h(MinecraftServer.java:357)
    at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:272)
    at net.minecraft.server.ThreadServerApplication.run(SourceFile:366)
2011-03-02 01:43:01 [INFO] <admin> Is it working yet? 
2011-03-02 01:43:01 [INFO] <admin> Not really. 
2011-03-02 01:43:01 [SEVERE] Could not pass event REDSTONE_CHANGE to CraftBookMechanisms
java.lang.NoSuchMethodError: com.sk89q.worldedit.blocks.BlockType.isRedstoneBlock(I)Z
    at com.sk89q.craftbook.bukkit.MechanicListenerAdapter$MechanicBlockListener.onBlockRedstoneChange(MechanicListenerAdapter.java:174)
    at net.minecraft.server.MinecraftServer.h(MinecraftServer.java:348)
    at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:272)
    at net.minecraft.server.ThreadServerApplication.run(SourceFile:366)
2011-03-02 01:43:02 [INFO] <admin> I hope we find a solution as soon as ever possible. 

期望的结果如下：

2011-03-02 01:43:00 [INFO] <admin> CraftBook is causing errors.
2011-03-02 01:43:01 [INFO] <admin> Is it working yet? 
2011-03-02 01:43:01 [INFO] <admin> Not really. 
2011-03-02 01:43:02 [INFO] <admin> I hope we find a solution as soon as ever possible.

如您所见，日志文件一遍又一遍地包含相同的错误。即使它总是以日期和时间开头，后跟 [SEVERE] 无法将事件 REDSTONE_CHANGE 传递给 CraftBookMechanisms 并以 at net.minecraft.server.ThreadServerApplication.run(SourceFile:366) 结束，中间的错误信息每次都不一样。这就是为什么我不能只用空字符串替换错误消息。

是否有一个正则表达式可以帮助我删除所有包含错误的行但保留剩余的行？这样一来，我的日志文件的大小将缩小到 50 MB 以下，就像以前所有这些错误都是由我的服务器由于插件损坏引起的一样。

Answer 1

这个 Python 脚本会遍历从标准输入读取的日志文件，将过滤后的日志消息打印到标准输出。

它使用正则表达式来匹配标记日志消息开头的行（例如以2011-03-02 01:43:00 [ 开头的行）。

如果日志消息开头的行包含[SEVERE] Could not pass event REDSTONE_CHANGE to CraftBookMechanisms，则脚本会丢弃该行与包含下一条日志消息开头的行之间的所有行。否则，它输出该行。您可以将其视为具有两种状态的有限状态机，这对应于脚本是跳过行还是输出行。

import sys
import re

START_OF_MESSAGE_RE = r"^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}"
ERROR_RE = START_OF_MESSAGE_RE + r' \[SEVERE\] Could not pass event REDSTONE_CHANGE to CraftBookMechanisms$'
skip_until_next_message = False

for line in sys.stdin:
    line = line.rstrip()
    if re.match(START_OF_MESSAGE_RE, line):
        if re.match(ERROR_RE, line):
            skip_until_next_message = True
        else:
            skip_until_next_message = False
    if not skip_until_next_message:
        print line

我在日志文件中添加了一些特殊情况进行测试。这是我测试它的日志文件：

2011-03-02 01:43:00 [INFO] <admin> CraftBook is causing errors. 
2011-03-02 01:43:01 [SEVERE] Could not pass event REDSTONE_CHANGE to CraftBookMechanisms
java.lang.NoSuchMethodError: com.sk89q.worldedit.blocks.BlockType.isRedstoneBlock(I)Z
    at com.sk89q.craftbook.bukkit.MechanicListenerAdapter$MechanicBlockListener.onBlockRedstoneChange(MechanicListenerAdapter.java:174)
    at net.minecraft.server.BlockButton.a(BlockButton.java:170)
    at net.minecraft.server.ItemInWorldManager.a(ItemInWorldManager.java:160)
    at net.minecraft.server.NetServerHandler.a(NetServerHandler.java:482)
    at net.minecraft.server.Packet15Place.a(SourceFile:57)
    at net.minecraft.server.NetworkManager.a(SourceFile:230)
    at net.minecraft.server.NetServerHandler.a(NetServerHandler.java:75)
    at net.minecraft.server.NetworkListenThread.a(SourceFile:100)
    [SEVERE] Could not pass event REDSTONE_CHANGE to CraftBookMechanisms
    at net.minecraft.server.MinecraftServer.h(MinecraftServer.java:357)
    at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:272)
    at net.minecraft.server.ThreadServerApplication.run(SourceFile:366)
2011-03-02 01:43:01 [INFO] <admin> Is it working yet? 
2011-03-02 01:43:01 [INFO] <admin> Not really. 
2011-03-02 01:43:01 [SEVERE] Could not pass event REDSTONE_CHANGE to CraftBookMechanisms
java.lang.NoSuchMethodError: com.sk89q.worldedit.blocks.BlockType.isRedstoneBlock(I)Z
    at com.sk89q.craftbook.bukkit.MechanicListenerAdapter$MechanicBlockListener.onBlockRedstoneChange(MechanicListenerAdapter.java:174)
    at net.minecraft.server.MinecraftServer.h(MinecraftServer.java:348)
    at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:272)
    at net.minecraft.server.ThreadServerApplication.run(SourceFile:366)
2011-03-02 01:43:02 [INFO] <admin> I hope we find a solution as soon as ever possible. 
2011-03-02 01:43:01 [SEVERE] Another multi
line
log
message
2011-03-02 01:43:01 [INFO] <admin> Here's the error: [SEVERE] Could not pass event REDSTONE_CHANGE to CraftBookMechanisms

这是输出：

$ python minecraftlog.py < minecraft.log 
2011-03-02 01:43:00 [INFO] <admin> CraftBook is causing errors.
2011-03-02 01:43:01 [INFO] <admin> Is it working yet?
2011-03-02 01:43:01 [INFO] <admin> Not really.
2011-03-02 01:43:02 [INFO] <admin> I hope we find a solution as soon as ever possible.
2011-03-02 01:43:01 [SEVERE] Another multi
line
log
message
2011-03-02 01:43:01 [INFO] <admin> Here's the error: [SEVERE] Could not pass event REDSTONE_CHANGE to CraftBookMechanisms

Answer 2

似乎更好的方法是匹配您想要保留的行，间接“删除”您不关心的行：

以下 Perl 脚本就足够了：

while (<>) {
  next unless /^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s\[INFO\]/;
  print;
}