ELK日志处理之使用logstash收集log4J日志
Log4j 主要由三部分组成:
- loggers:负责采集日志信息。
- appenders:负责将日志信息发布到不同地方。
- layouts:负责以各种风格格式化日志信息。
创建maven工程,pom中导入日志依赖:
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
在resource目录下新建log4j.properties,加入以下配置:
### 设置###
log4j.rootLogger = debug,stdout,D,E,logstash
### 输出信息到控制抬 ###
log4j.appender.stdout = org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target = System.out
log4j.appender.stdout.layout = org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern = [%-5p] %d{yyyy-MM-dd HH:mm:ss,SSS} method:%l%n%m%n
### 输出DEBUG 级别以上的日志到=/Users/elk/log4j/debug.log###
log4j.appender.D = org.apache.log4j.DailyRollingFileAppender
log4j.appender.D.File = /Users/bee/Documents/elk/log4j/debug.log
log4j.appender.D.Append = true
log4j.appender.D.Threshold = DEBUG
log4j.appender.D.layout = org.apache.log4j.PatternLayout
log4j.appender.D.layout.ConversionPattern = %-d{yyyy-MM-dd HH:mm:ss} [ %t:%r ] - [ %p ] %m%n
### 输出ERROR 级别以上的日志到=/Users/elk/log4j/error.log ###
log4j.appender.E = org.apache.log4j.DailyRollingFileAppender
log4j.appender.E.File =/Users/bee/Documents/elk/log4j/error.log
log4j.appender.E.Append = true
log4j.appender.E.Threshold = ERROR
log4j.appender.E.layout = org.apache.log4j.PatternLayout
log4j.appender.E.layout.ConversionPattern = %-d{yyyy-MM-dd HH:mm:ss} [ %t:%r ] - [ %p ] %m%n
#输出日志到logstash
log4j.appender.logstash=org.apache.log4j.net.SocketAppender
log4j.appender.logstash.RemoteHost=127.0.0.1
log4j.appender.logstash.port=4560
log4j.appender.logstash.ReconnectionDelay=60000
log4j.appender.logstash.LocationInfo=true
配置文件中,把日志输出了四份:
- 第一份输出到控制台
- 第二份把DEBUG 级别以上的日志到文件
- 第三份把输出ERROR 级别以上的日志到文件
- 第四份输出到logstash
在java目录下添加TestLoggerMain.java,内容如下:
import java.util.ArrayList;
import java.util.List;
import org.apache.log4j.Logger;
public class TestLoggerMain {
public static final Logger logger = Logger.getLogger(TestLoggerMain.class);
public static void main(String[] args) {
logger.debug("This is a debug message!");
logger.info("This is info message!");
logger.warn("This is a warn message!");
logger.error("This is error message!");
List<Integer> list = new ArrayList<Integer>();
list.add(1);
list.add(2);
list.add(3);
try {
for (Integer num : list) {
list.remove(num);
}
} catch (Exception e) {
logger.error(e);
}
}
}
配置logstash
创建logstash.conf配置文件
input { #日志数据输入来源log4j
log4j {
host => "127.0.0.1"
port => 4561
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch{
hosts => ["localhost:9200"]
index => "log4j-%{+YYYY.MM.dd}"
document_type => "log4j_type"
}
}
首先启动Elasticsearch,然后启动logstash和kibaba
logstash输出:
进入kibaba查询:
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 16,
"successful": 16,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 1,
"hits": [
{
"_index": ".kibana",
"_type": "config",
"_id": "5.2.0",
"_score": 1,
"_source": {
"buildNum": 14695
}
},
{
"_index": "log4j-2019.01.08",
"_type": "log4j_type",
"_id": "AWgsBq4xsF3SfWsXEnpg",
"_score": 1,
"_source": {
"method": "queryData",
"thread": "http-nio-8025-exec-1",
"priority": "ERROR",
"type": "simple",
"message": "nested exception is org.apache.ibatis.type.TypeException: Could not set parameters for mapping: ParameterMapping{property='level', mode=IN, javaType=class java.lang.Integer, jdbcType=null, numericScale=null, resultMapId='null', jdbcTypeName='null', expression='null'}. Cause: org.apache.ibatis.type.TypeException: Error setting non null for parameter #1 with JdbcType null . Try setting a different JdbcType for this parameter or a different configuration property. Cause: java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer",
"path": "com.ekingwin.bas.cloud.solr.service.impl.JiucaiyunSolrServiceImpl",
"@timestamp": "2019-01-08T05:54:20.018Z",
"file": "JiucaiyunSolrServiceImpl.java:328",
"@version": "1",
"host": "127.0.0.1:56347",
"logger_name": "com.ekingwin.bas.cloud.solr.service.impl.JiucaiyunSolrServiceImpl",
"class": "com.ekingwin.bas.cloud.solr.service.impl.JiucaiyunSolrServiceImpl",
"timestamp": 1546926859986,
}
}
]
}
}