使用 perl 和哈希数据结构从日志文件中打印记录

【问题标题】：Print records from log file using perl and hash data structure使用 perl 和哈希数据结构从日志文件中打印记录
【发布时间】：2018-12-31 07:58:30
【问题描述】：

我有以下脚本，最初由 @zdim 编写，我对其进行了一些调整。

#!/usr/bin/perl

use warnings;
use strict;

my $file = "/home/tsec/prototype/logs/extractedlogs/cowrieresult.log";
open (LOG, $file);

# Assemble results for required output in data structure:
# %rept = { $port => { $usr => { $status => $freq } };

my %testhash;#new code
my %rept;
my ($ip, $port);

while (my $line = <LOG>)
{
    if ($line =~ /New connection/) {
        ($ip, $port) = $line =~ /New connection:\s+([^:]+):(\d+)/;
        #new code here
        if($ip){
                $testhash{$ip}++;
        }
        #end
        next;
    }

    my ($usr, $status) =  $line =~ m/login\ attempt \s+ \[ ( [^\]]+ ) \] \s+ (\w+)/x;
    if ($usr and $status) {
        $rept{$port}{$usr}{$status}++;
    }
    else { warn "Line with an unexpected format:\n$line" }
}
#close(LOG);
#open (LOG, $file);
#my $frequency = 0;
#while (my $line = <LOG>){
#       if($line =~ /login attempt/){

        #split string, get the ip and match it with original $ip
#       my ($testip) = (split /[\s,:\[\]\/]+/, $line)[-6];
        #print "$testip\n";
        #this two lines above print ips from login attempt line.
#       if($testip =~ /$ip/){
#               $frequency++;
#       }
        #elsif($testip =~ /^(?!$ip)/) {
                # stop frequency counter and start another one?
        #       print "$frequency\n";
        #       $frequency = 0;
        #}

#       }
#}
#print "$frequency\n";
#close(LOG);

#new code
print "ConnectionsOnIP\n";
foreach my $ip (sort keys %testhash){
        print "$testhash{$ip}\n";
}

print "\n";

#new code
print "Port,Status,AttemptOnPort,AttemptsOnIP,Malicious\n";
foreach my $ip (sort keys %testhash){
foreach my $port (sort keys %rept) {
    foreach my $usr (sort keys %{$rept{$port}}) {
        foreach my $stat ( sort keys %{$rept{$port}{$usr}} ) {
                if($port ne ""){
            print "$port,$stat,$rept{$port}{$usr}{$stat},$testhash{$ip}\n";
                }
        }
   }

}
}
#new code

可以看出，除了最后一个变量（AttemptsOnIP）之外，我希望得到当前正在工作的所需输出我希望 AttemptsOnIP 变量在某种程度上执行 AttemptsOnPort 所做的事情：

Port,Status,AttemptsOnPort,ConnectionsOnIP,Malicious
15853,failed,4,18
15853,succeeded,4,18
18693,failed,1,18
18942,failed,1,18
18942,succeeded,1,18
31130,succeeded,1,18
43041,failed,1,18
43041,succeeded,1,18
46321,failed,1,18
46321,succeeded,1,18
47417,failed,3,18
47417,succeeded,3,18
48713,failed,1,18
48713,succeeded,1,18
53653,failed,1,18
53653,succeeded,1,18
60563,failed,1,18
60563,succeeded,1,18

我创建了一个名为 testhash 的哈希并将其传递给 ip 变量以增加它。现在我想根据单行输出的 IP 来增加这个散列变量。这是日志文件：

2016-05-02 10:20:56+0000 [SSHService ssh-userauth on HoneyPotTransport,14,183.3.202.172] login attempt [root/!@] failed
2016-05-02 10:20:57+0000 [SSHService ssh-userauth on HoneyPotTransport,15,183.3.202.172] login attempt [root/!@] failed
2016-05-02 10:20:57+0000 [SSHService ssh-userauth on HoneyPotTransport,14,183.3.202.172] login attempt [root/123456] succeeded
2016-05-02 10:20:58+0000 [SSHService ssh-userauth on HoneyPotTransport,15,183.3.202.172] login attempt [root/123456] succeeded
2016-05-02 10:43:32+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New connection: 183.3.202.172:55157 (172.17.0.5:2222) [session: 43283650]
2016-05-02 10:43:46+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New connection: 183.3.202.172:10319 (172.17.0.5:2222) [session: c7702f86]
2016-05-02 10:43:53+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New connection: 183.3.202.172:46321 (172.17.0.5:2222) [session: fe7bb804]
2016-05-02 10:43:57+0000 [SSHService ssh-userauth on HoneyPotTransport,17,183.3.202.172] login attempt [root/!@] failed
2016-05-02 10:43:58+0000 [SSHService ssh-userauth on HoneyPotTransport,17,183.3.202.172] login attempt [root/123456] succeeded
2016-05-02 10:43:59+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New connection: 183.3.202.172:18693 (172.17.0.5:2222) [session: d74eae96]
2016-05-02 10:44:02+0000 [SSHService ssh-userauth on HoneyPotTransport,18,183.3.202.172] login attempt [root/!@] failed
2016-05-02 10:44:03+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New connection: 183.3.202.172:31130 (172.17.0.5:2222) [session: 3bde7820]
2016-05-02 10:44:03+0000 [SSHService ssh-userauth on HoneyPotTransport,18,183.3.202.172] login attempt [root/123456] succeeded
2016-05-02 10:44:05+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New connection: 183.3.202.172:47417 (172.17.0.5:2222) [session: 3e177c02]
2016-05-02 10:44:06+0000 [SSHService ssh-userauth on HoneyPotTransport,19,183.3.202.172] login attempt [root/!@] failed
2016-05-02 10:44:09+0000 [SSHService ssh-userauth on HoneyPotTransport,19,183.3.202.172] login attempt [root/123456] succeeded
2016-05-02 10:44:10+0000 [SSHService ssh-userauth on HoneyPotTransport,21,183.3.202.172] login attempt [root/!@] failed
2016-05-02 10:44:11+0000 [SSHService ssh-userauth on HoneyPotTransport,21,183.3.202.172] login attempt [root/123456] succeeded
2016-05-02 10:44:13+0000 [SSHService ssh-userauth on HoneyPotTransport,20,183.3.202.172] login attempt [root/!@] failed
2016-05-02 10:44:14+0000 [SSHService ssh-userauth on HoneyPotTransport,20,183.3.202.172] login attempt [root/123456] succeeded
2016-05-02 11:06:55+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New connection: 183.3.202.172:13849 (172.17.0.5:2222) [session: b20915b6]
2016-05-02 11:07:06+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New connection: 183.3.202.172:61338 (172.17.0.5:2222) [session: cd38fe51]
2016-05-02 11:07:14+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New connection: 183.3.202.172:23048 (172.17.0.5:2222) [session: 01b12825]
2016-05-02 11:07:21+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New connection: 183.3.202.172:60563 (172.17.0.5:2222) [session: ad64232b]
2016-05-02 11:07:26+0000 [SSHService ssh-userauth on HoneyPotTransport,23,183.3.202.172] login attempt [root/!@] failed
2016-05-02 11:07:27+0000 [SSHService ssh-userauth on HoneyPotTransport,23,183.3.202.172] login attempt [root/123456] succeeded
2016-05-02 11:07:33+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New connection: 183.3.202.172:53653 (172.17.0.5:2222) [session: 9c48415b]
2016-05-02 11:07:41+0000 [SSHService ssh-userauth on HoneyPotTransport,26,183.3.202.172] login attempt [root/!@] failed
2016-05-02 11:07:47+0000 [SSHService ssh-userauth on HoneyPotTransport,26,183.3.202.172] login attempt [root/123456] succeeded
2016-05-02 11:12:25+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New connection: 183.3.202.172:18942 (172.17.0.5:2222) [session: a4dc4901]
2016-05-02 11:12:34+0000 [SSHService ssh-userauth on HoneyPotTransport,27,183.3.202.172] login attempt [root/!@] failed
2016-05-02 11:12:36+0000 [SSHService ssh-userauth on HoneyPotTransport,27,183.3.202.172] login attempt [root/123456] succeeded
2016-05-02 11:32:40+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New connection: 183.3.202.172:40091 (172.17.0.5:2222) [session: aeb36234]
2016-05-02 11:32:43+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New connection: 183.3.202.172:53505 (172.17.0.5:2222) [session: 9022c831]
2016-05-02 11:32:48+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New connection: 183.3.202.172:15131 (172.17.0.5:2222) [session: cf62fb9a]
2016-05-02 11:32:48+0000 [cowrie.ssh.transport.HoneyPotSSHFactory] New connection: 183.3.202.172:15853 (172.17.0.5:2222) [session: f2f6c254]
2016-05-02 11:32:50+0000 [SSHService ssh-userauth on HoneyPotTransport,28,183.3.202.172] login attempt [root/!@] failed
2016-05-02 11:32:52+0000 [SSHService ssh-userauth on HoneyPotTransport,28,183.3.202.172] login attempt [root/123456] succeeded
2016-05-02 11:32:55+0000 [SSHService ssh-userauth on HoneyPotTransport,29,183.3.202.172] login attempt [root/!@] failed
2016-05-02 11:32:55+0000 [SSHService ssh-userauth on HoneyPotTransport,30,183.3.202.172] login attempt [root/!@] failed
2016-05-02 11:32:56+0000 [SSHService ssh-userauth on HoneyPotTransport,30,183.3.202.172] login attempt [root/123456] succeeded
2016-05-02 11:32:57+0000 [SSHService ssh-userauth on HoneyPotTransport,31,183.3.202.172] login attempt [root/!@] failed
2016-05-02 11:32:59+0000 [SSHService ssh-userauth on HoneyPotTransport,31,183.3.202.172] login attempt [root/123456] succeeded

因此，前两行的输出应如下所示

Port,Status,AttemptsOnPort,ConnectionsOnIP,Malicious
    15853,failed,4,(total no of times the IP using this port is seen in log, even if it used other ports)
    15853,succeeded,4,18

【问题讨论】：

  • 看来您需要为每个看到的 IP 添加一个计数器：用户、状态。因此，您将使用计数器 $by_ip{$ip}{$usr}{$status}++ 添加具有这些级别的哈希。您将其添加到$rept{$port}{$usr}{$status}++ 旁边的下一行。这可以使用已经捕获的$ip，我不明白你为什么需要再次从last attempt 行中获取它（必须相同，不是吗？）。不要忘记声明这个新的哈希，my %by_ip。我可以发布这个，但先自己尝试并报告。你现在拥有它的方式，它计算任何端口、用户、状态的 IP - 所有这些，一次性。
  • @zdim 不确定我是否正确，我在$rept{$port}{$usr}{$status}++ 下添加了哈希，这就是我要打印的地方：print "Port,Status,AttemptOnPort,AttemptsOnIP,Malicious\n"; foreach my $ip (sort keys %by_ip){ foreach my $port (sort keys %rept) { foreach my $usr (sort keys %{$rept{$port}}) { foreach my $stat ( sort keys %{$rept{$port}{$usr}} ) { if($port ne ""){ print "$port,$stat,$rept{$port}{$usr}{$stat},$by_ip{$ip}{$usr}{$stat},\n$ } } } } }
  • @zdim 这是我得到的output
  • 好吧，我明白了。添加该行后，您有两个 distinct 数据结构（尽管非常相似）。您尝试的嵌套迭代很棘手，并且很可能使某些事情翻倍，而使其他事情不匹配。我会调查的。 （也许对代码稍作修改会更容易提取所需的输出。）
  • @zdim 是的，事实上这正是正在发生的事情，双重数据并引发一些错误。如果您能指出我如何做到这一点，将不胜感激。

标签： perl logging hash output


【解决方案1】：

此代码以以下格式打印报告。如果不需要，请删除 (IP) 字段。

端口、状态、AttemptOnPort、(IP)、ConnectionsOnIP

为每个用户打印这样的一行。但是，ConnectionsOnIP 是 所有 用户和端口看到的这个 IP 的总数。该代码还单独打印关于 IP 的单独报告。有关相关问题，请参阅 cmets。

use strict;
use warnings;

my $file = 'logfile.txt';
open my $fh_in, '<', $file;

# Assemble results for required output in data structure:
# %rept = {
#    $port => {
#       $ip => {
#           $usr => { 
#               $status => $freq 
#          },
#       },
#   },
# };
# Auxiliary: %ip_tot = { $ip => { $status => $freq } } 

my (%rept, %ip_tot);
my ($ip, $port);

while (my $line = <$fh_in>) 
{
    if ($line =~ /New connection/) {
        ($ip, $port) = $line =~ /New connection:\s+([^:]+):(\d+)/;
        next;
    }   
    elsif (!$ip or !$port) { next }  # First lines come before New connection

    my ($usr, $status) = $line =~ m/login attempt\s+\[([^\]]+)\]\s+(\w+)/;
    if ($usr and $status) {
        $rept{$port}{$ip}{$usr}{$status}++;
        $ip_tot{$ip}{$status}++;
    }   
    else { warn "Line with an unexpected format:\n$line" }
}

print "Port,Status,AttemptOnPort,(IP),ConnectionsOnIP\n";
foreach my $port (sort keys %rept) {
    foreach my $ip (sort keys %{$rept{$port}}) {
        foreach my $usr (sort keys %{$rept{$port}{$ip}}) {
            foreach my $stat ( sort keys %{$rept{$port}{$ip}{$usr}} ) { 
                print "$port,$stat,$rept{$port}{$ip}{$usr}{$stat}";
                print "$,(ip),$ip_tot{$ip}{$stat}\n"; 
            }   
        }   
    }   
}

print "\n";
print "IP,Status,Occurences\n";
foreach my $ip (sort keys %ip_tot) {
    foreach my $stat ( sort keys %{$ip_tot{$ip}} ) {
        print "$ip,$stat,$ip_tot{$ip}{$stat}\n"; 
    }
}

将提供的输入作为logfile.txt 打印出来

端口、状态、AttemptOnPort、(IP)、ConnectionsOnIP 15853,失败,4,(183.3.202.172),12 15853,成功,3,(183.3.202.172),11 18693,失败,1,(183.3.202.172),12 18942,失败,1,(183.3.202.172),12 18942,成功,1,(183.3.202.172),11 31130,成功,1,(183.3.202.172),11 46321,失败,1,(183.3.202.172),12 46321,成功,1,(183.3.202.172),11 47417,失败,3,(183.3.202.172),12 47417,成功,3,(183.3.202.172),11 53653,失败,1,(183.3.202.172),12 53653,成功,1,(183.3.202.172),11 60563,失败,1,(183.3.202.172),12 60563,成功,1,(183.3.202.172),11 IP、状态、事件 183.3.202.172，失败，12 183.3.202.172，成功，11

正则表达式解释。这是有效的代码，由/x 提供。 \s+ 在 cmets 中被忽略。

my ($usr, $status) =  $line =~ m/ 
    login\ attempt \s+         # literal, serves as a 'post' to help matching 
    \[                         # literal [ within which our pattern is 
        (                      # start capture 
            [^\]]+             # any char which is not ], 1 or more times  
        )                      # end of capture 
    \] \s+                     # closing literal ] 
    (\w+)                      # next capture: any 'word' char, 1 or more times 
/x;

核心是否定字符类，[ ^\] ]。它说：匹配任何一个字符 ([...]) 是 not (^) 括号 (\])，需要转义 (\) 以表示文字字符.它后面的+ 表示一次或多次次。例如

my $str = 'a5_".-]B1'; 
if ($str =~ m/([^\]]+)/) { say "Got: $1" }

这会打印出Got: a5_".-。第一个 ] 之前的所有内容都匹配（并捕获）。这是一种指定非贪婪匹配的方法，直到第一次出现给定字符。请注意，像 .+] 这样的东西会匹配到 last ] 的所有内容，它是 greedy。

见Regular Expressions Tutorial。在 SO 中搜索特定问题和小型教程。

【讨论】：

    猜你喜欢
    • 2011-01-26
    • 2023-03-08
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2012-07-22
    • 1970-01-01
    • 1970-01-01
    • 2016-02-21
    相关资源
    最近更新 更多
    热门标签
    Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode