使用 awk/perl 过滤 tomcat 日志

Question

我正在尝试从 tomcat 日志中提取错误|异常。由于 tomcat 日志在多行中有错误详细信息。我想将每个新的日志条目都视为我的记录，即日期可以作为记录分隔符。

Oct 4, 2012 4:00:38 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load com.google.common.base.Stopwatch.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1531)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1491)
        at com.myweb.ontest.stats.RpcInterceptor.intercept(RpcInterceptor.java:45)
        at com.myweb.ontest.platform.SupplySource$Iface$$EnhancerByCGLIB$$6b5e8142.finalize()
        at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
        at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83)
        at java.lang.ref.Finalizer.access$100(Finalizer.java:14)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)
Oct 4, 2012 4:00:38 PM Org.apache.catalina.loader .... 

请告诉我如何使用 awk/perl 脚本来完成。

谢谢

Answer 1

我首先在记录之间添加一个空行：

awk '/^[A-Z][a-z]/ && f++{print ""}1' file

因此您可以在后续的 awk 脚本中使用空行作为记录分隔符轻松处理它：

awk '/^[A-Z][a-z]/ && f++{print ""}1' file |
awk -v RS= -F'\n' '{print "Record #" NR; for (i=1;i<=NF;i++) print "Field #" i, "[" $i "]"}'
Record #1
Field #1 [Oct 4, 2012 4:00:38 PM org.apache.catalina.loader.WebappClassLoader loadClass]
Field #2 [INFO: Illegal access: this web application instance has been stopped already.  Could not load com.google.common.base.Stopwatch.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.]
Field #3 [java.lang.IllegalStateException]
Field #4 [        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1531)]
Field #5 [        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1491)]
Field #6 [        at com.myweb.ontest.stats.RpcInterceptor.intercept(RpcInterceptor.java:45)]
Field #7 [        at com.myweb.ontest.platform.SupplySource$Iface$$EnhancerByCGLIB$$6b5e8142.finalize()]
Field #8 [        at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)]
Field #9 [        at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83)]
Field #10 [        at java.lang.ref.Finalizer.access$100(Finalizer.java:14)]
Field #11 [        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)]
Record #2
Field #1 [Oct 7, 2012 4:00:38 PM org.apache.catalina.loader.WebappClassLoader loadClass]
Field #2 [INFO: just a dummy record]
Field #3 [java.lang.IllegalStateException]
Field #4 [        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1531)]
Field #5 [        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1491)]
Field #6 [        at com.myweb.ontest.stats.RpcInterceptor.intercept(RpcInterceptor.java:45)]
Field #7 [        at com.myweb.ontest.platform.SupplySource$Iface$$EnhancerByCGLIB$$6b5e8142.finalize()]
Field #8 [        at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)]
Field #9 [        at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83)]
Field #10 [        at java.lang.ref.Finalizer.access$100(Finalizer.java:14)]
Field #11 [        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)]

您没有发布您希望输出的样子，因此如果上述内容没有为您提供足够的信息来执行此操作，请发布您的示例输入的预期输出

Answer 2

假设您要将这些错误放入 MySQL 数据库中，我会这样做。

#!/usr/bin/perl -Tw

use strict;
use warnings;
use English qw( -no_match_vars $OS_ERROR );
use Readonly;
use DBI;

Readonly my $LOG_FILENAME => '/var/log/tomcat.log';

Readonly my $DATE_REGEX => qr{
    \w{3} \s \d+, \s        # Oct 4,
    \d{4}         \s        # 2012
    \d+:\d+:\d+   \s \w{2}  # 4:00:38 PM
}xms;

my $sth;
{
    my $dbh = DBI->connect(
        'DBI:mysql:database=errors;host=localhost',
        'error_monitor',
        '*********',
        { 'RaiseError' => 1 }
    );

    die $DBI::errstr
        if !$dbh;

    $sth = $dbh->prepare_cached( q{
        INSERT INTO error (
            date,
            text
        ) VALUES (
            ?,
            ?
        )
    } );

    die $DBI::errstr
        if !$sth;
}

my $fh;
{
    open $fh, '<', $LOG_FILENAME
        or die "open $LOG_FILENAME: $OS_ERROR";
}

my ( $count, %record ) = ( 0, () );

while ( my $line = <$fh> ) {

    if ( $line =~ m{\A ( $DATE_REGEX ) \s ( .+ ) }xms ) {

        my ( $date, $text ) = ( $1, $2 );

        if (%record) {
            $sth->execute( @record{qw( date text )} );
            %record = ();
            $count++;
        }

        @record{qw( date text )} = ( $date, $text );
    }
    elsif ( exists $record{date} ) {

        $record{text} .= $line;
    }
    else {

        warn "malformed message: $line";
    }
}

if (%record) {
    $sth->execute( @record{qw( date text )} );
    $count++;
}

close $fh
    or die "close $LOG_FILENAME: $OS_ERROR";

print "inserted $count error records\n"
    or die "print: $OS_ERROR";

__END__

Answer 3

在 perl 中你可以这样做：-

use strict;
my $log = join('', <STDIN>);
my %entries = split(/([\r\n]*\w{3} \d+, \d+ \d+:\d+:\d+ [AP]M )/m, $log);

foreach my $key(keys %entries)
{
    print "$key\n";
    print "$entries{$key}\n";
}

上面的代码从标准输入读取整个日志，按日期拆分并存储日期，输入哈希并在最后打印哈希。