同事整理的，在此分享。logback，log4j2 等slf4j的日志实现都可以以json格式输出日志， 这里采用的是logback。当然也可以以文本行的格式输出，然后在logstash里通过grok解析，但是直接以json格式输出，在logstash处理时效率会高一点。
Logback 输出 Json格式日志文件
为了让 logback 输出JSON 格式的日志文件，需要在pom.xml 加入如下依赖
<dependency> <groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>4.8</version>
<scope>runtime</scope>
</dependency> |
logback日志配置示例
<appender name="errorFile" class="ch.qos.logback.core.rolling.RollingFileAppender">
<filter class="ch.qos.logback.classic.filter.LevelFilter">
<level>ERROR</level>
<onMatch>ACCEPT</onMatch>
<onMismatch>DENY</onMismatch>
</filter>
<file>${log.dir}/elk/error.log</file> <!-- 当前的日志文件文件放在 elk文件下，该日志的内容会被filebeat传送到es -->
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <! -- 历史日志会放到 bak 文件下，最多保存7天的历史，最多占用 1G的空间 -->
<fileNamePattern>${log.dir}/bak/error.%d{yyyy-MM-dd}.log</fileNamePattern>
<maxHistory>7</maxHistory>
<totalSizeCap>1GB</totalSizeCap>
</rollingPolicy>
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<providers>
<pattern>
<pattern>
{
"tags": ["errorlog"],
"project": "myproject",
"timestamp": "%date{\"yyyy-MM-dd'T'HH:mm:ss,SSSZ\"}",
"log_level": "%level",
"thread": "%thread",
"class_name": "%class",
"line_number": "%line",
"message": "%message",
"stack_trace": "%exception{5}",
"req_id": "%X{reqId}",
"elapsed_time": "#asLong{%X{elapsedTime}}"
} </pattern>
</pattern>
</providers>
</encoder>
</appender> |
Json 字段说明:
|
名称
|
说明
|
备注
|
|
|
|
|
|
|---|---|---|---|---|---|---|---|
| tags | 用于说明这条日志是属于哪一类日志 | ||||||
timestamp |
日志记录时间 | ||||||
project |
系统名称，该日志来自于哪个系统 | ||||||
log_level |
输出日志级别 | ||||||
thread |
输出产生日志的线程名。 | ||||||
class_name |
输出执行记录请求的调用者的全限定名 |
|
|||||
line_number |
输出执行日志请求的行号 |
|
|||||
message |
输出应用程序提供的信息 | ||||||
stack_trace |
异常栈信息 | ||||||
req_id |
请求ID，用于追踪请求 | 需要引入aop-logging | |||||
elapsed_time |
该方法执行时间,单位: 毫秒 | 需要引入aop-logging |
%X{key}: 表示该项来自于SLF4j MDC,需要引入 aop-logging
<dependency> <groupId>com.cloud</groupId> <artifactId>xspring-aop-logging</artifactId> <version>0.7.1</version>
</dependency>针对web应用，在 web.xml 中加入 ReqIdFilter,该过滤器会在MDC 加入 reqId<filter> <filter-name>aopLogReqIdFilter</filter-name>
<filter-class>com.github.nickvl.xspring.core.log.aop.ReqIdFilter</filter-class>
</filter><filter-mapping> <filter-name>aopLogReqIdFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>or register in springboot like this:
@Beanpublic FilterRegistrationBean getDemoFilter(){
ReqIdFilter reqIdFilter=new ReqIdFilter();
FilterRegistrationBean registrationBean=new FilterRegistrationBean();
registrationBean.setFilter(reqIdFilter);
List<String> urlPatterns=new ArrayList<String>();
urlPatterns.add("/*");
registrationBean.setUrlPatterns(urlPatterns);
registrationBean.setOrder(100);
return registrationBean;
}如果需要记录该方法执行时间: elapsed_time，如果在该类或者方法上加入如下注解：import com.github.nickvl.xspring.core.log.aop.annotation.LogDebug;
import com.github.nickvl.xspring.core.log.aop.annotation.LogInfo;
@LogInfo // 当logger 设为level=INFO 会输出
@LogException(value = {@Exc(value = Exception.class, stacktrace = false)}, warn = {@Exc({IllegalArgumentException.class})}) //
当logger 设为level=error 会输出针对dubbo 消费者的日志记录,dubbo消费者是通过 javassist 生成的动态类型,如果要监控该dubbo接口的传入参数，返回值，和调用时间 需要引入aop-logging,以及在 eye-rpc包中的接口上给对应的类或方法 加上上面的注解。dubbo 消费者的日志会输出如下配置: <logger name="com.alibaba.dubbo.common.bytecode" level="INFO" additivity="false">
<appender-ref ref="dubboApiFile"/>
</logger> |
ElasticSearch 模板设置
curl -XPUT http://localhost:9200/_template/log -d '{
"mappings": {
"_default_": {
"_all": {
"enabled": false
},
"_meta": {
"version": "5.1.1"
},
"dynamic_templates": [
{
"strings_as_keyword": {
"mapping": {
"ignore_above": 1024,
"type": "keyword"
},
"match_mapping_type": "string"
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"beat": {
"properties": {
"hostname": {
"ignore_above": 1024,
"type": "keyword"
},
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"version": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"input_type": {
"ignore_above": 1024,
"type": "keyword"
},
"message": {
"norms": false,
"type": "text"
},
"offset": {
"type": "long"
},
"source": {
"ignore_above": 1024,
"type": "keyword"
},
"tags": {
"ignore_above": 1024,
"type": "keyword"
},
"type": {
"ignore_above": 1024,
"type": "keyword"
}
}
}
},
"order": 0,
"settings": {
"index.refresh_interval": "5s"
},
"template": "log-*"
}'curl -XPUT http://localhost:9200/_template/log-java -d '
{ "mappings": {
"_default_": {
"properties": {
"log_level": {
"ignore_above": 1024,
"type": "keyword"
},
"project": {
"ignore_above": 1024,
"type": "keyword"
},
"thread": {
"ignore_above": 1024,
"type": "keyword"
},
"req_id": {
"ignore_above": 1024,
"type": "keyword"
},
"class_name": {
"ignore_above": 1024,
"type": "keyword"
},
"line_number": {
"type": "long"
},
"exception_class":{
"ignore_above": 1024,
"type": "keyword"
},
"elapsed_time": {
"type": "long"
},
"stack_trace": {
"type": "keyword"
}
}
}
},
"order": 1,
"settings": {
"index.refresh_interval": "5s"
},
"template": "log-java-*"
}' |
logstatsh 设置
logstash-java-log
if [fields][logType] == "java" {
json {
source => "message"
remove_field => ["offset"]
}
date {
match => ["timestamp","yyyy-MM-dd'T'HH:mm:ss,SSSZ"]
remove_field => ["timestamp"]
}
if [stack_trace] {
mutate {
add_field => { "exception_class" => "%{stack_trace}" }
}
}
if [exception_class] {
mutate {
gsub => [
"exception_class", "\n", "",
"exception_class", ":.*", ""
]
}
}
} |
filebeat 设置
filebeat.yml
filebeat.prospectors:- input_type: log paths:
- /eyebiz/logs/eyebiz-service/elk/*.log # eyebiz-service 日志
- /eyebiz/logs/eyebiz-web/elk/*.log # eyebiz-web 日志
fields:
logType: "java"
docType: "log-java-dev"
|