需要输入来处理 ELK 堆栈中的多行 java 堆栈日志

Question

我正在尝试通过 ELK 堆栈处理以下日志。我了解 logstash 处理和 grok 模式的基础知识，并且能够为各个行编写 grok。但无法理解从这些日志中提取一些信息信息的总体策略。

Thread #1: t@-1680123584, lwp=22843, ref=0x1b0f8550, session=9F0071A66D89544155D149CCE2453E9A:mx2135649930e123d964:(WebServiceFacade.java:84), ms=0x78714bc0
<Start Stack Trace>
  <1 - ADK Verbose Trace Entry>
  stateless dispatch for invokeClass.bosInterface executing
  Active: 37 minutes  0.00 seconds 
  User:
    ServiceUser1
  Tenant:
    
  Session:
    9F0071A66D89544155D149CCE2453E9A:mx2135649930e123d964:(WebServiceFacade.java:84)
  Parameters:
    bosContext _cntx:
      user:
        ContextUser1
      depth:
        3
      session id:
        9F0071A66D89544155D149CCE2453E9A:mx2135649930e123d964:(WebServiceFacade.java:84)
    bosUTF _className:
      TestClassName1
    bosStringList _construct:
      2 entries
          $$MXRIP$$|java.util.HashMap
          1
    bosUTF _methodName:
      TestMethodName1
    bosStringList _params:
      2 entries
          $$MXRIP$$|java.util.HashMap
          16
    uint8 _local:
      1

  Error Stack:
  
Thread #2: t@-1686439616, lwp=22837, ref=0x1b7a81c0, session=2FF1BFBFCC010E7815678741BB95907F:mx115420087975768b5:(WebServiceFacade.java:84), ms=0x78caa910
<Start Stack Trace>
  <1 - ADK Verbose Trace Entry>
  stateless dispatch for invokeClass.bosInterface executing
  Active: 34 minutes  45.00 seconds 
  User:
    ServiceUser2
  Tenant:
    
  Session:
    2FF1BFBFCC010E7815678741BB95907F:mx115420087975768b5:(WebServiceFacade.java:84)
  Parameters:
    bosContext _cntx:
      user:
        ContextUser2
      depth:
        2
      session id:
        2FF1BFBFCC010E7815678741BB95907F:mx115420087975768b5:(WebServiceFacade.java:84)
    bosUTF _className:
      ClassName2
    bosStringList _construct:
      2 entries
          $$MXRIP$$|java.util.HashMap
          7
    bosUTF _methodName:
      TestMethodName2
    bosStringList _params:
      2 entries
          $$MXRIP$$|java.util.HashMap
          6
    uint8 _local:
      1

 Error Stack:

如果有人可以帮助我了解总体策略，我如何从这些日志中获取以下信息，例如： 对于线程 1：

Active => 37 minutes  0.00 seconds
User => ServiceUser1
Tenant => 
Session => 9F0071A66D89544155D149CCE2453E9A:mx2135649930e123d964:(WebServiceFacade.java:84)
Parameters:
user => ContextUser1
bosUTF _className => TestClassName1
bosUTF _methodName => TestMethodName1

下一个线程的类似信息将在同一个日志文件中。

Answer 1

您需要了解多行解析以将每个线程分离为一个日志事件

input{
    file{
        path => "/home/test_logs.log"
        tags => ["samplelog"]
        sincedb_path => "/dev/null"
        start_position => "beginning"
        codec => multiline {
                pattern => "^\s"
                negate => true
                what => "previous"
        }

    }

}

然后你将继续过滤

filter{

  if "samplelog" in [tag]{

        grok{

         break_on_match => false

         #log parsing

         match => { "message" => "<grok pattern here>"}

        }
        
        #For possibly using the date from log
        date{

         match => ["Timestamp", "UNIX_MS"]

         target => "@time"

        }

     }

}