非数字字符的fail2ban模式匹配输入与数字答案

【问题标题】：fail2ban pattern for non-digit character matches input with digit非数字字符的fail2ban模式匹配输入与数字
【发布时间】：2020-06-25 14:03:10
【问题描述】：

这是我的输入字符串：

myserver:/etc/fail2ban # cat one_line.txt
12.123.123.12 - - [25/Jul/2017:09:26:40 +0200] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"

我试图演化出一个以^<HOST>.*$ 开头并以^<HOST> - - \[2.*$ 结尾的正则表达式，但是当它结束时发生了一些奇怪的事情。

最后一次模式迭代不再匹配。当我这样声明时，我可以强制匹配^<HOST> - - \[\D.*$。

哈？这意味着非数字字符跟在左方括号后面！

我打印了我采取的步骤，以便向您展示它是如何演变的。

希望有人能给我解释一下这种奇怪的行为。

环境信息：

myserver:/etc/fail2ban # fail2ban-regex --version
fail2ban-regex 0.9.4
myserver:/etc/fail2ban # python --version
Python 2.7.17

现在跟随我的过滤器表达式的迭代演变：

迭代 1：匹配 - 符合预期

myserver:/etc/fail2ban # fail2ban-regex -l heavydebug one_line.txt '^<HOST>.*$'

Running tests
=============

D: Setting usedns = warn for Filter(None)
D: Created Filter(None)
Use   failregex line : ^<HOST>.*$
Use         log file : one_line.txt
Use         encoding : UTF-8

D: Working on line u'12.123.123.12 - - [25/Jul/2017:09:26:40 +0200] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"'
D: Matched time template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
D: Got time 1500967600.000000 for "u'25/Jul/2017:09:26:40 +0200'" using template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
D: Looking for failregex match of [(u'12.123.123.12 - - [', u'25/Jul/2017:09:26:40 +0200', u'] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"')]
D: Matched FailRegex('^(?:::f{4,6}:)?(?P<host>[\\w\\-.^_]*\\w).*$')
D: Sorting the template list
D: Winning template: Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? with 1 hits

Results
=======

Failregex: 1 total
|-  #) [# of hits] regular expression
|   1) [1] ^<HOST>.*$
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [1] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-

Lines: 1 lines, 0 ignored, 1 matched, 0 missed
[processed in 0.00 sec]

迭代 2：匹配 - 符合预期

myserver:/etc/fail2ban # fail2ban-regex -l heavydebug one_line.txt '^<HOST> - - .*$'

Running tests
=============

D: Setting usedns = warn for Filter(None)
D: Created Filter(None)
Use   failregex line : ^<HOST> - - .*$
Use         log file : one_line.txt
Use         encoding : UTF-8

D: Working on line u'12.123.123.12 - - [25/Jul/2017:09:26:40 +0200] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"'
D: Matched time template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
D: Got time 1500967600.000000 for "u'25/Jul/2017:09:26:40 +0200'" using template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
D: Looking for failregex match of [(u'12.123.123.12 - - [', u'25/Jul/2017:09:26:40 +0200', u'] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"')]
D: Matched FailRegex('^(?:::f{4,6}:)?(?P<host>[\\w\\-.^_]*\\w) - - .*$')
D: Sorting the template list
D: Winning template: Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? with 1 hits

Results
=======

Failregex: 1 total
|-  #) [# of hits] regular expression
|   1) [1] ^<HOST> - - .*$
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [1] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-

Lines: 1 lines, 0 ignored, 1 matched, 0 missed
[processed in 0.00 sec]

迭代 3：匹配 - 符合预期

myserver:/etc/fail2ban # fail2ban-regex -l heavydebug one_line.txt '^<HOST> - - \[.*$'

Running tests
=============

D: Setting usedns = warn for Filter(None)
D: Created Filter(None)
Use   failregex line : ^<HOST> - - \[.*$
Use         log file : one_line.txt
Use         encoding : UTF-8

D: Working on line u'12.123.123.12 - - [25/Jul/2017:09:26:40 +0200] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"'
D: Matched time template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
D: Got time 1500967600.000000 for "u'25/Jul/2017:09:26:40 +0200'" using template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
D: Looking for failregex match of [(u'12.123.123.12 - - [', u'25/Jul/2017:09:26:40 +0200', u'] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"')]
D: Matched FailRegex('^(?:::f{4,6}:)?(?P<host>[\\w\\-.^_]*\\w) - - \\[.*$')
D: Sorting the template list
D: Winning template: Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? with 1 hits

Results
=======

Failregex: 1 total
|-  #) [# of hits] regular expression
|   1) [1] ^<HOST> - - \[.*$
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [1] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-

Lines: 1 lines, 0 ignored, 1 matched, 0 missed
[processed in 0.00 sec]

迭代 4.1：失败 - 为什么？？？

myserver:/etc/fail2ban # fail2ban-regex -l heavydebug one_line.txt '^<HOST> - - \[2.*$'

Running tests
=============

D: Setting usedns = warn for Filter(None)
D: Created Filter(None)
Use   failregex line : ^<HOST> - - \[2.*$
Use         log file : one_line.txt
Use         encoding : UTF-8

D: Working on line u'12.123.123.12 - - [25/Jul/2017:09:26:40 +0200] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"'
D: Matched time template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
D: Got time 1500967600.000000 for "u'25/Jul/2017:09:26:40 +0200'" using template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
D: Looking for failregex match of [(u'12.123.123.12 - - [', u'25/Jul/2017:09:26:40 +0200', u'] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"')]
D: Sorting the template list
D: Winning template: Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? with 1 hits

Results
=======

Failregex: 0 total

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [1] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-

Lines: 1 lines, 0 ignored, 0 matched, 1 missed
[processed in 0.00 sec]

|- Missed line(s):
|  12.123.123.12 - - [25/Jul/2017:09:26:40 +0200] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
`-

迭代 4.2：失败 - 为什么？？？

myserver:/etc/fail2ban # fail2ban-regex -l heavydebug one_line.txt '^<HOST> - - \[\d.*$'

Running tests
=============

D: Setting usedns = warn for Filter(None)
D: Created Filter(None)
Use   failregex line : ^<HOST> - - \[\d.*$
Use         log file : one_line.txt
Use         encoding : UTF-8

D: Working on line u'12.123.123.12 - - [25/Jul/2017:09:26:40 +0200] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"'
D: Matched time template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
D: Got time 1500967600.000000 for "u'25/Jul/2017:09:26:40 +0200'" using template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
D: Looking for failregex match of [(u'12.123.123.12 - - [', u'25/Jul/2017:09:26:40 +0200', u'] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"')]
D: Sorting the template list
D: Winning template: Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? with 1 hits

Results
=======

Failregex: 0 total

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [1] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-

Lines: 1 lines, 0 ignored, 0 matched, 1 missed
[processed in 0.00 sec]

|- Missed line(s):
|  12.123.123.12 - - [25/Jul/2017:09:26:40 +0200] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
`-

Itertaion 4.3：匹配 - 为什么？？？

myserver:/etc/fail2ban # fail2ban-regex -l heavydebug one_line.txt '^<HOST> - - \[\D.*$'

Running tests
=============

D: Setting usedns = warn for Filter(None)
D: Created Filter(None)
Use   failregex line : ^<HOST> - - \[\D.*$
Use         log file : one_line.txt
Use         encoding : UTF-8

D: Working on line u'12.123.123.12 - - [25/Jul/2017:09:26:40 +0200] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"'
D: Matched time template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
D: Got time 1500967600.000000 for "u'25/Jul/2017:09:26:40 +0200'" using template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
D: Looking for failregex match of [(u'12.123.123.12 - - [', u'25/Jul/2017:09:26:40 +0200', u'] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"')]
D: Matched FailRegex('^(?:::f{4,6}:)?(?P<host>[\\w\\-.^_]*\\w) - - \\[\\D.*$')
D: Sorting the template list
D: Winning template: Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? with 1 hits

Results
=======

Failregex: 1 total
|-  #) [# of hits] regular expression
|   1) [1] ^<HOST> - - \[\D.*$
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [1] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-

Lines: 1 lines, 0 ignored, 1 matched, 0 missed
[processed in 0.00 sec]

提前致谢。

【问题讨论】：

标签： python regex pattern-matching mismatch fail2ban

【解决方案1】：

根据https://fail2ban.readthedocs.io/en/latest/filters.html#regular-expressions

[...]
Regular expressions (failregex, ignoreregex) assume that the date/time has been removed from the log line (this is just how fail2ban works internally ATM).

If the format is like ‘<date...> error 1.2.3.4 is evil’ then you need to match the < at the start so regex should be similar to ‘^<> <HOST> is evil$’ using <HOST> where the IP/domain name appears in the log line.
[...]

就是这样！

【讨论】：