从不同事件和分隔符中提取 Splunk 字段答案

【问题标题】：Splunk field extractions from different events & delimiters从不同事件和分隔符中提取 Splunk 字段
【发布时间】：2020-11-18 15:41:00
【问题描述】：

我的关键事件时间戳的 Splunk 日志格式如下：

[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=Hour = 18-nov-2020 11:00:00]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=Id = 126566]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=zipBefore = 18-nov-2020 12:27:08.776174]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=zipAfter = 18-nov-2020 12:36:52.718122]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=StartTime = 18-nov-2020 12:17:10.603227]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=EndTime = 18-nov-2020 12:36:53.094513]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=beginThread = 18-nov-2020 12:17:10.905782]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=endThread = 18-nov-2020 12:24:22.628907]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=LogTime = CASE1~6~18-nov-2020 12:17:11.377070~0~18-nov-2020 12:17:12.608526,CASE1~0~18-nov-2020 12:17:11.365409~0~18-nov-2020 12:17:12.654285,CASE3~0~18-nov-2020 12:17:12.644921~11~18-nov-2020 12:17:13.636655,CASE2~5~18-nov-2020 12:17:13.295225~700000~18-nov-2020 12:23:29.370142,CASE2~2~18-nov-2020 12:17:12.815714~700000~18-nov-2020 12:23:31.400500]

我想将所有关键事件时间戳提取到如下表字段中，以便我可以在它们之间做一些区别等：

Hour                  Id      StartTime                    EndTime                      beginThread                 endThread                   zipBefore                    zipAfter
18-nov-2020 11:00:00  126566  18-nov-2020 12:17:10.603227  18-nov-2020 12:36:53.094513  18-nov-2020 12:17:10.905782 18-nov-2020 12:24:22.628907 18-nov-2020 12:27:08.776174  18-nov-2020 12:36:52.718122

另外，我在日志中的最后一个事件有不同的情况、线程和时间戳，我需要根据分隔符分别提取，如下所示：

Case Thread StartTime                    Count EndTime
CASE1     6 18-nov-2020 12:17:11.377070      0 18-nov-2020 12:17:12.608526
CASE1     0 18-nov-2020 12:17:11.365409      0 18-nov-2020 12:17:12.654285
CASE2     5 18-nov-2020 12:17:13.295225 700000 18-nov-2020 12:23:29.370142
CASE2     2 18-nov-2020 12:17:12.815714 700000 18-nov-2020 12:23:31.400500
CASE3     0 18-nov-2020 12:17:12.644921     11 18-nov-2020 12:17:13.636655

【问题讨论】：

标签： splunk data-extraction splunk-query

【解决方案1】：

以下是完成第一项任务的示例查询。然而，一个问题是它仅适用于一组事件。由于这8个事件之间没有明显的联系，因此没有将一个事务的8与另一个事务的8分开。

| makeresults | eval data="[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=Hour = 18-nov-2020 11:00:00]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=Id = 126566]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=zipBefore = 18-nov-2020 12:27:08.776174]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=zipAfter = 18-nov-2020 12:36:52.718122]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=StartTime = 18-nov-2020 12:17:10.603227]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=EndTime = 18-nov-2020 12:36:53.094513]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=beginThread = 18-nov-2020 12:17:10.905782]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=endThread = 18-nov-2020 12:24:22.628907]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=LogTime = CASE1~6~18-nov-2020 12:17:11.377070~0~18-nov-2020 12:17:12.608526,CASE1~0~18-nov-2020 12:17:11.365409~0~18-nov-2020 12:17:12.654285,CASE3~0~18-nov-2020 12:17:12.644921~11~18-nov-2020 12:17:13.636655,CASE2~5~18-nov-2020 12:17:13.295225~700000~18-nov-2020 12:23:29.370142,CASE2~2~18-nov-2020 12:17:12.815714~700000~18-nov-2020 12:23:31.400500]" | eval data=split(data,"!") | mvexpand data | eval _raw=data
```The above just sets up test data```
| rex "\[Message=(?<Message>[^\]]+)"
| rex field=Message "(?<field>\w+)\s+=\s+(?<value>.*)"
| eval {field}=value
| table Hour Id StartTime EndTime beginThread endThread zipBefore zipAfter
| filldown | tail 1

第二部分和第一部分类似，只是解析不同。

| makeresults | eval data="[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=Hour = 18-nov-2020 11:00:00]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=Id = 126566]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=zipBefore = 18-nov-2020 12:27:08.776174]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=zipAfter = 18-nov-2020 12:36:52.718122]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=StartTime = 18-nov-2020 12:17:10.603227]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=EndTime = 18-nov-2020 12:36:53.094513]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=beginThread = 18-nov-2020 12:17:10.905782]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=endThread = 18-nov-2020 12:24:22.628907]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=LogTime = CASE1~6~18-nov-2020 12:17:11.377070~0~18-nov-2020 12:17:12.608526,CASE1~0~18-nov-2020 12:17:11.365409~0~18-nov-2020 12:17:12.654285,CASE3~0~18-nov-2020 12:17:12.644921~11~18-nov-2020 12:17:13.636655,CASE2~5~18-nov-2020 12:17:13.295225~700000~18-nov-2020 12:23:29.370142,CASE2~2~18-nov-2020 12:17:12.815714~700000~18-nov-2020 12:23:31.400500]" | eval data=split(data,"!") | mvexpand data | eval _raw=data
```The above just sets up test data```
| rex "\[Message=(?<Message>[^\]]+)"
| rex field=Message "(?<field>\w+)\s+=\s+(?<value>.*)"
| eval {field}=value
```We only care about LogTime messages```
| search LogTime=*
```Divide the message on commas and make separate events```
| eval LogTime=split(LogTime, ",") | mvexpand LogTime
```Parse the events```
| rex field=LogTime "(?<Case>[^~]+)~(?<Thread>[^~]+)~(?<StartTime>[^~]+)~(?<Count>[^~]+)~(?<EndTime>[^~]+)(?:,|$)"
| table Case Thread StartTime Count EndTime

【讨论】：

谢谢@RichG！这很有帮助。
嗨 RichG，为了进一步找到时差，我正在尝试 strptime(EndTime, \"%dT-%b-%Y %H:%M:%S.%6Q\") 哪个似乎不适用于我的格式：“18-nov-2020 12:17:13.636655”。我的缩写月份名称是否需要为 Nov 而不是 nov 才能正常工作？任何建议表示赞赏。谢谢。
只要格式字符串与数据匹配，小写就可以正常工作。 | makeresults | eval EndTime="18-nov-2020 12:17:13.636655" | eval epoch=strptime(EndTime, "%d-%b-%Y %H:%M:%S.%6Q") | table EndTime epoch
嗨 RichG，与第一个任务相关，我想使用源作为 8 个事件之间的连接并将这些事件时间戳列出到每个源的表字段中..`Source Hour Id StartTime EndTime beginThread endThread zipBefore zipAfter ABC.log 18-nov-2020 11:00:00 126566 18-nov-2020 12:17:10.603227 18-nov-2020 12:36:53.094513 18-nov-2020 12:17:10.905782 18-nov -2020 12:24:22.628907 18-nov-2020 12:27:08.776174 18-nov-2020 12:36:52.718122 XYZ.log 20-nov-2020 ....`