C#：使用 RegEx 进行日志解析，还想从日志文件中提取一些其他值答案

【问题标题】：C#: Log parsing with RegEx, want to extract some other values from the log file as wellC#：使用 RegEx 进行日志解析，还想从日志文件中提取一些其他值
【发布时间】：2018-03-03 14:21:38
【问题描述】：

我对 C# 有点陌生，我已经向 stackoverflow 提交了一个问题，但我解决了这个问题 (C#: Log parsing date/time with regex, when there's an anomaly of >1 second, I want to save that whole line to a new file)。

现在，日志输出如下所示：

<FLTR> <TID: 0000000461> <RPC ID: 0072606623> <Queue: Fast      > <Client-RPC: 390620   > <USER: lkoporec                                     > <Overlay-Group: 1         > /* Mon Feb 26 2018 13:53:04.4230 */ <Filter Level:0 Number Of Filters:10> Checking "SHR:LHP:InitSvc_GetServiceContextFlag" (0)
<API > <TID: 0000000373> <RPC ID: 0072606628> <Queue: List      > <Client-RPC: 390620   > <USER: Remedy Application Service                   > <Overlay-Group: 1         > /* Mon Feb 26 2018 13:53:06.1140 */ +BE      ARBulkEntry -- user Remedy Application Service from Normalization Engine (protocol 24) at IP address 10.122.26.197 using RPC // :q:0.1s 

<FLTR> <TID: 0000000445> <RPC ID: 0072611490> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:53:43.3600 */ <Filter Level:0 Number Of Filters:3> Checking "FB:SetGroupForSmartReportingKPIs_3" (530)
<API > <TID: 0000000382> <RPC ID: 0072611493> <Queue: List      > <Client-RPC: 390620   > <USER: darzupan                                     > <Overlay-Group: 1         > /* Mon Feb 26 2018 13:53:46.5240 */ +EXPQRY  ARExpandQueryMenu --  from Mid-tier (protocol 23) at IP address 10.122.0.61 using RPC // :q:0.0s 

<FLTR> <TID: 0000000457> <RPC ID: 0072646830> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:58:19.5660 */ <Filter Level:0 Number Of Filters:1> Checking "FB:SetGroupForSmartReportingKPIs_GetLabel" (502)
<FLTR> <TID: 0000000345> <RPC ID: 0072647870> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:58:30.4330 */ <Filter Level:0 Number Of Filters:3> Checking "FB:SetGroupForSmartReportingKPIs_3" (530)

<FLTR> <TID: 0000000441> <RPC ID: 0072647871> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:58:30.4480 */ --> Disabled -- filter is ignored
<FLTR> <TID: 0000000460> <RPC ID: 0072648393> <Queue: Fast      > <Client-RPC: 390620   > <USER: esribar                                      > <Overlay-Group: 1         > /* Mon Feb 26 2018 13:58:35.3710 */ <Filter Level:0 Number Of Filters:123> Checking "INT:SRMHPD:INC:UpdateRequest_99_SetSRAppRegistryGUID" (99)

<FLTR> <TID: 0000000460> <RPC ID: 0072648393> <Queue: Fast      > <Client-RPC: 390620   > <USER: esribar                                      > <Overlay-Group: 1         > /* Mon Feb 26 2018 13:58:35.4250 */ Application-Copy-Field-Value 600000001 1000003890
<SQL > <TID: 0000000457> <RPC ID: 0000000000> <Queue: Admin     > <Client-RPC: 390600   > <USER:                                              > <Overlay-Group: 0         > /* Mon Feb 26 2018 13:58:41.8550 */ Generating prepared statement

<FLTR> <TID: 0000000447> <RPC ID: 0072648992> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:58:41.9020 */ --> Disabled -- filter is ignored
<SQL > <TID: 0000000346> <RPC ID: 0072649991> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:58:51.9350 */ COMMIT TRANSACTION

<API > <TID: 0000000439> <RPC ID: 0072650203> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:58:53.7280 */ +CE      ARCreateEntry -- schema FB:History from Flashboards (protocol 22) at IP address 10.122.26.197 using RPC // :q:0.0s 
<SQL > <TID: 0007483686> <RPC ID: 0000000000> <Queue: Escalation> <Client-RPC: 0        > <USER:                                              > <Overlay-Group: 0         > /* Mon Feb 26 2018 14:06:48.7630 */ Generating prepared statement

<SQL > <TID: 0000000447> <RPC ID: 0000000000> <Queue: Admin     > <Client-RPC: 390600   > <USER:                                              > <Overlay-Group: 0         > /* Mon Feb 26 2018 14:06:52.1940 */ OK
<SQL > <TID: 0000000457> <RPC ID: 0072764627> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 14:18:10.5700 */ OK

看到那些 TID 了吗？每个出现的元素都有自己的线程 ID，因此一个元素可以有例如 100 个 TID，分散在一个 .log 文件中。

当发生超时 (previousLine) 时，如何捕获 TID，在原始 .log 文件中搜索该 TID，并将引用该 TID 的每一行导出到一个新文件中（当然，为每个确实发生超时）？

使用新的正则表达式/日期时间函数？是否可以只获取 TID（因为它们对于元素、RPC ID 总是相同的，而其余的则不是 - 一个 TID 在此输出“TID：0000000447”中出现两次，但彼此不同）。

这是我的代码到目前为止的样子：

using System;
using System.Globalization;
using System.IO;
using System.Text.RegularExpressions;

namespace LogParser
{
class Program
{
    static void Main(string[] args)
    {
        Console.OutputEncoding = System.Text.Encoding.UTF8;
        var counterLine = 0;
        var counterTimeout = 0;
        var line = string.Empty;
        var previousLine = string.Empty;
        var previousDt = DateTime.MaxValue;
        var tidLine = string.Empty;
        var regex = new Regex(@"\d{2}:\d{2}:\d{2}\.\d{4}");

        try
        {
            var file = new System.IO.StreamReader(args[0]);
            Console.WriteLine("Profesionalno branje logov se začenja:\n");

            using (var writer = new StreamWriter(Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop), "log_output.txt"), true))
            {
                while ((line = file.ReadLine()) != null)
                {
                    counterLine++;
                    foreach (Match m in regex.Matches(line))
                    {
                        var dt = new DateTime();
                        if (DateTime.TryParseExact(m.Value, "HH:mm:ss.ffff", null, DateTimeStyles.None, out dt))
                        {
                            if ((dt - previousDt).TotalSeconds > 1)
                            {
                                counterTimeout++;
                                Console.WriteLine(previousLine);
                                Console.WriteLine(line + "\n\n\n");
                                writer.WriteLine(previousLine);
                                writer.WriteLine(line + Environment.NewLine);
                            }

                            previousLine = line;
                            previousDt = dt;
                        }
                    }

                }
            }

            file.Close();
            Console.WriteLine("\nBranje logov je končano. Prebrali smo: {0} vrstic ter izpisali " +
                              "{1} vrstic, kjer je bil timeout v datoteko.", counterLine, counterTimeout);
        }
        catch (Exception e)
        {
            Console.OpenStandardError();
            Console.WriteLine(e.Message);
        }

        if (args.Length < 1)
        {
            Console.OpenStandardError();
            Console.WriteLine("Uporaba: {0} LOG_FILE", AppDomain.CurrentDomain.FriendlyName);
            Console.ReadKey();
            return;
        }
    }
}
}

最终结果应如下所示（对于每个 TID）：

<FLTR> <TID: 0000000447> <RPC ID: 0072599594> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:52:08.5920 */ <Filter Level:0 Number Of Filters:2> Checking "FB:SetGroupForSmartReportingKPIs" (505)
<FLTR> <TID: 0000000447> <RPC ID: 0072599594> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:52:08.5920 */ --> Disabled -- filter is ignored
<FLTR> <TID: 0000000447> <RPC ID: 0072599594> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:52:08.5930 */ <Filter Level:0 Number Of Filters:2> Checking "FB:SetGroupForSmartReportingKPIs_1" (510)
<FLTR> <TID: 0000000447> <RPC ID: 0072599594> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:52:08.5930 */ --> Disabled -- filter is ignored
<FLTR> <TID: 0000000447> <RPC ID: 0072599594> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:52:08.5930 */ <Filter Level:0 Number Of Filters:2> Checking "FB:SetGroupForSmartReportingKPIs_2" (520)
<FLTR> <TID: 0000000447> <RPC ID: 0072599594> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         >    --> Passed -- perform actions
<FLTR> <TID: 0000000447> <RPC ID: 0072599594> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         >         0 : Set Fields
<SQL > <TID: 0000000447> <RPC ID: 0072599594> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:52:08.5940 */ Generating prepared statement
<SQL > <TID: 0000000447> <RPC ID: 0072599594> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:52:08.5950 */ OK
<SQL > <TID: 0000000447> <RPC ID: 0072599594> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:52:08.5950 */ Binding [5] parameters to prepared statement
<SQL > <TID: 0000000447> <RPC ID: 0072599594> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:52:08.5950 */ OK
<SQL > <TID: 0000000447> <RPC ID: 0072599594> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:52:08.5950 */ WITH AR_SQL_Alias$1 AS (SELECT T72.C179, T72.C1, ROW_NUMBER() OVER (ORDER BY T72.C1 ASC) AS 'AR_RowNumber_Alias$1' FROM T72 WHERE ((N'{78DF8F98-D170-48A2-A7A8-0B2E2BB6931C}' = T72.C179) AND ((T72.C41050 = N'CMDB:Total_Base_CI') OR (T72.C41050 = N'CMDB:Total_Relationship_CI')))) SELECT AR_SQL_Alias$1.C179, AR_SQL_Alias$1.C1 FROM AR_SQL_Alias$1 WHERE (AR_SQL_Alias$1.AR_RowNumber_Alias$1 BETWEEN 0 AND 3)
<SQL > <TID: 0000000447> <RPC ID: 0072599594> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:52:08.5960 */ OK
<FLTR> <TID: 0000000447> <RPC ID: 0072599594> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         >               Label Group5 (40111) = 
<FLTR> <TID: 0000000447> <RPC ID: 0072599594> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:52:08.5960 */ <Filter Level:0 Number Of Filters:3> Checking "FB:SetGroupForSmartReportingKPIs_3" (530)
<FLTR> <TID: 0000000447> <RPC ID: 0072599594> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         >    --> Failed qualification
<FLTR> <TID: 0000000447> <RPC ID: 0072599594> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:52:08.5960 */ <Filter Level:0 Number Of Filters:4> Checking "FB:SetGroupForSmartReportingKPIs_4" (540)
<FLTR> <TID: 0000000447> <RPC ID: 0072599594> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         >    --> Failed qualification
<SQL > <TID: 0000000447> <RPC ID: 0000000000> <Queue: Admin     > <Client-RPC: 390600   > <USER:                                              > <Overlay-Group: 0         > /* Mon Feb 26 2018 13:52:08.5980 */ OK
<SQL > <TID: 0000000447> <RPC ID: 0072599594> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:52:08.5980 */ INSERT INTO T251(C7,C40000,C40100,C40104,C40106,C40108,C40102,C40101,C40103,C40105,C40107,C40109,C2,C8,C5,C3,C6,C1) VALUES(0,N'{78DF8F98-D170-48A2-A7A8-0B2E2BB6931C}',1000000164,1000000217,1000000001,303497300,1358879400,2.00,N'2',N'TS3111 Merilni sistemi',N'Telekom Slovenije, d.d.',N'INTERNET_11894970',N'Remedy Application Service',N'x',N'Remedy Application Service',1519649528,1519649528,N'000000118010138')
<SQL > <TID: 0000000447> <RPC ID: 0072599594> <Queue: Fast      > <Client-RPC: 390619   > <USER: Remedy Application Service -- Impersonated by Remedy Application Service --> <Overlay-Group: 1         > /* Mon Feb 26 2018 13:52:08.5990 */ OK

等等等等（有很多行，所以我把它们删掉了）。

【问题讨论】：

预期输出？
我希望第二个输出看起来如何？像这样： /* Mon Feb 26 2018 13:58:41.9020 / --> Disabled -- 过滤器被忽略 / Mon Feb 26 2018 14:06:52.1940 */ OK
以及 2GB 日志文件中其余的 0000000447 TID。在它输出它找到的第一个（从第一个 OUTPUT）之后，它会创建一个新的空白行，并粘贴第二个 TID、空白行、第三个 TID 等。但是当它获得 TID 时，它也会复制整行来自日志文件，就像我现在正在创建的输出一样。
编辑您的帖子并添加最终结果。你想让正则表达式输出什么？
你去。重要的是它在我的第一个输出中获取超时时发生的 TID，并在原始日志中搜索它并将所有它从 .log 复制到一个新文件中。它必须对在第一个输出中发现的作为超时（至少一次）的每个 TID 执行此操作。

标签： c# regex parsing

【解决方案1】：

我在遵循您想要的内容时遇到了一些麻烦，我只是复制了您的输入和所需的输出，但我无法理解您所寻求的模式到底是什么。如果你能给出一个简化的例子，那会有所帮助，因为 Regex 几乎就是一个可变模式。

根据您的要求，我推荐以下工作流程：

使用 while 循环，其中 Regex.IsMatch 是条件。当您始终匹配您的条件时，循环会继续并提取您的信息。

对于匹配模式本身，根据您的要求，查找超时模式的外观（我无法确定），您可以使用它来获取下一行：

(timeoutpattern.*[\r\n]+).*

这将匹配超时模式本身，但在循环中是必需的。

当这个匹配被创建并保存为一个字符串变量时，使用 Regex.Replace 将它从原始字符串中删除并替换为""

然后从原始字符串中替换 (timeoutpattern).*?用 "" 完全删除它（你会得到你想要的输出）。

【讨论】：

嗨！ This is how the first log looks like（我正在解析的那个）。在我的 OP 中，第一条信息是我第一次解析日志后的样子。然后是我的代码。我的 OP 中的第三条信息是我希望 second 输出的样子（提取所有 TID（线程 ID）），匹配一个超时（我想这样做每次超时）。有人告诉我我应该使用字典，并将所有行存储在那里，然后在我发现超时后从那里提取它们。最好的问候！