需要创建正则表达式来分析 Rails 服务器日志

Question

我有一个rails服务器日志文件，格式如下。

Started <REQUEST_TYPE_1> <URL_1> for <IP_1> at <TIMESTAMP_1>
  Processing by <controller#action_1> as <REQUEST_FORMAT_1>
  Parameters: <parameters_1>
<Some logs from code>
Rendered <some_template_1> (<timetaken_1>)
Completed <RESPONSE_CODE_1> in <TIME_1>


Started <REQUEST_REQUEST_TYPE_2> <URL_2> for <IP_2> at <TIMESTAMP_2>
  Processing by <controller#action_2> as <REQUEST_FORMAT_2>
  Parameters: <parameters_2>
<Some logs from code>
Completed <RESPONSE_CODE_2> in <TIME_2>

现在，我需要解析这个日志并从上面的日志中提取所有REQUEST_TYPE、URL、IP、TIMESTAMP、REQUEST_FORMAT、RESPONSE_CODE。我正在努力在 java/ruby 中为它创建一个好的正则表达式。 <> 在实际输入中不存在。我已经添加了实际数据的可读性和屏蔽。

示例请求：

Started GET "/google.com/2" for 127.0.0.1 at Tue Dec 01 12:01:13 +0530 2015
  Processing by MyController#method as JS
  Parameters: {"abc" => "xyz"}
[LOG] 3 : User text log
Completed 200 OK in 26ms (Views: 3.3ms | ActiveRecord: 2.9ms)


Started POST "/google.com/543" for 127.0.1.1 at Tue Dec 01 13:13:16 +0530 2015
  Processing by MyController#method_2 as JSON
  Parameters: {"efg" => "uvw"}
Completed 404 Not Authorized in 65ms (Views: 1.5ms | ActiveRecord: 1.0ms)

预期输出：

request_types = ['GET', 'POST']
urls = ['/google.com/2','/google.com/543']
ips = ['127.0.0.1','127.0.1.1']
timestamps = ['Tue Dec 01 12:01:13 +0530 2015','Tue Dec 01 13:13:16 +0530 2015']
request_formats = ['JS','JSON']
response_codes = ['200 OK','404 Not Authorized']

我能够编写以下正则表达式，但它没有按预期工作。

request_types = /Started \w+/  //Expected array of all request types
urls = /"\/.*\/"/ //Expected array of all urls types
ips = /"d{1,3}.d{1,3}.d{1,3}.d{1,3}"/ //Expected array of all ips types
timestamps =  /at \w+/
request_formats =/as \w+/
response_codes = /Completed \w+/

我希望在创建正则表达式以从 JAVA/RUBY 中的给定输入中提取此参数时获得一些帮助。如果可能的话，我更喜欢 java。

Answer 1

这是一个 Java sn-p，展示了如何将日志中的详细信息放入 Java 中的单独数组列表中：

String re = "(?sm)^Started\\s+(?<requesttype>\\S+)\\s+\"(?<url>\\S+)\"\\s+for\\s+(?<ip>\\d+(?:\\.\\d+)+)\\s+at\\s+(?<tsp>[a-zA-Z]+\\s+[a-zA-Z]+\\s+\\d+\\s+\\d+:\\d+:\\d+\\s+\\+\\d+\\s\\d{4})\\s+(?:Processing\\s+by\\s+\\S+)\\s+as\\s+(?<requestformat>\\S+)(?:\\s+Parameters:\\s+\\S+)?(?:(?:(?:(?!\nStarted ).)*Completed\\s)(?<responsecode>\\d+(?:(?!\\sin\\s).)*))?";
String str = "Started GET \"/google.com/2\" for 127.0.0.1 at Tue Dec 01 12:01:13 +0530 2015\n  Processing by MyController#method as JS\n  Parameters: {\"abc\" => \"xyz\"}\n[LOG] 3 : User text log\nCompleted 200 OK in 26ms (Views: 3.3ms | ActiveRecord: 2.9ms)\n\n\nStarted POST \"/google.com/543\" for 127.0.1.1 at Tue Dec 01 13:13:16 +0530 2015\n  Processing by MyController#method_2 as JSON\n  Parameters: {\"efg\" => \"uvw\"}\nCompleted 404 Not Authorized in 65ms (Views: 1.5ms | ActiveRecord: 1.0ms)";
Pattern pattern = Pattern.compile(re);
Matcher matcher = pattern.matcher(str);
List<String> requesttypes = new ArrayList<String>();
List<String> urls = new ArrayList<String>();
List<String> ips = new ArrayList<String>();
List<String> timestamps = new ArrayList<String>(); 
List<String> requestformats = new ArrayList<String>(); 
List<String> responsecodes = new ArrayList<String>();
while (matcher.find()){
    requesttypes.add(matcher.group("requesttype"));
    urls.add(matcher.group("url"));
    ips.add(matcher.group("ip"));
    timestamps.add(matcher.group("tsp"));
    requestformats.add(matcher.group("requestformat"));
    responsecodes.add(matcher.group("responsecode"));
    System.out.println("-----------------------");
    System.out.println(matcher.group("requesttype"));
    System.out.println(matcher.group("url")); 
    System.out.println(matcher.group("ip")); 
    System.out.println(matcher.group("tsp")); 
    System.out.println(matcher.group("requestformat")); 
    System.out.println(matcher.group("responsecode")); 
} 

请参阅IDEONE demo。您甚至可以在完成匹配后打印数组，例如System.out.println(urls):

System.out.println(requesttypes);
System.out.println(urls);
System.out.println(ips);
System.out.println(urls);
System.out.println(timestamps);
System.out.println(requestformats);
System.out.println(responsecodes);

见this demo。输出是：

[GET, POST]
[/google.com/2, /google.com/543]
[127.0.0.1, 127.0.1.1]
[/google.com/2, /google.com/543]
[Tue Dec 01 12:01:13 +0530 2015, Tue Dec 01 13:13:16 +0530 2015]
[JS, JSON]
[200 OK, 404 Not Authorized]

正则表达式匹配：

• (?sm)^ - 行首（由于^ 和?m 选项）
• Started\\s+ - 文字 Started 字符串和 1+ 个空格
• (?<requesttype>\\S+) - 组“请求类型”包含 1+ 个非空白字符
• \\s+\" - 1+ 空格后跟 "
• (?<url>\\S+) - 组“url”包含 1+ 个非空格
• \"\\s+for\\s+ - " 后跟 1+ 个空格 + for + 1+ 个空格
• (?<ip>\\d+(?:\\.\\d+)+) - IP 组包含数字 + . + 数字（.+数字 1+ 次）
• \\s+at\\s+ - at 这个词被空格包围
• (?<tsp>[a-zA-Z]+\\s+[a-zA-Z]+\\s+\\d+\\s+\\d+:\\d+:\\d+\\s+\\+\\d+\\s\\d{4}) - 以不同顺序保存字母和数字的时间戳组，用空格 acc 分隔。输入示例
  • \\s+ - 1+ 个空格
• (?:Processing\\s+by\\s+\\S+)\\s+as\\s+ - Processing by 后跟一些单词（1+ 个非空格），然后是单词 as，周围有空格
• (?<requestformat>\\S+) - 由非空白符号组成的组“请求格式”
• (?:\\s+Parameters:\\s+\\S+)? - 可选组 Paramters: 后跟空格和一些单词
• (?:(?:(?:(?!\nStarted ).)*Completed\\s)(?<responsecode>\\d+(?:(?!\\sin\\s).)*))? - 一个可选组（因为包含在(?:...)? 中），匹配直到Completed 的任何字符，但没有Started （由于经过调和的贪婪令牌(?:(?!\nStarted ).)*），然后匹配Completed后跟一个空格，然后 (?<responsecode>\\d+(?:(?!\\sin\\s).)*) 匹配并捕获到组“响应代码”数字后跟任何字符，直到整个单词 in 被空格包围。