ELK日志处理之使用logstash收集log4J日志

Log4j 主要由三部分组成:

  1. loggers:负责采集日志信息。
  2. appenders:负责将日志信息发布到不同地方。
  3. layouts:负责以各种风格格式化日志信息。

创建maven工程,pom中导入日志依赖:

<dependency>
    <groupId>log4j</groupId>
    <artifactId>log4j</artifactId>
    <version>1.2.17</version>
</dependency>

 在resource目录下新建log4j.properties,加入以下配置:

### 设置###
log4j.rootLogger = debug,stdout,D,E,logstash

### 输出信息到控制抬 ###
log4j.appender.stdout = org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target = System.out
log4j.appender.stdout.layout = org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern = [%-5p] %d{yyyy-MM-dd HH:mm:ss,SSS} method:%l%n%m%n

### 输出DEBUG 级别以上的日志到=/Users/elk/log4j/debug.log###
log4j.appender.D = org.apache.log4j.DailyRollingFileAppender
log4j.appender.D.File = /Users/bee/Documents/elk/log4j/debug.log
log4j.appender.D.Append = true
log4j.appender.D.Threshold = DEBUG 
log4j.appender.D.layout = org.apache.log4j.PatternLayout
log4j.appender.D.layout.ConversionPattern = %-d{yyyy-MM-dd HH:mm:ss}  [ %t:%r ] - [ %p ]  %m%n

### 输出ERROR 级别以上的日志到=/Users/elk/log4j/error.log  ###
log4j.appender.E = org.apache.log4j.DailyRollingFileAppender
log4j.appender.E.File =/Users/bee/Documents/elk/log4j/error.log 
log4j.appender.E.Append = true
log4j.appender.E.Threshold = ERROR 
log4j.appender.E.layout = org.apache.log4j.PatternLayout
log4j.appender.E.layout.ConversionPattern = %-d{yyyy-MM-dd HH:mm:ss}  [ %t:%r ] - [ %p ]  %m%n

#输出日志到logstash
log4j.appender.logstash=org.apache.log4j.net.SocketAppender
log4j.appender.logstash.RemoteHost=127.0.0.1
log4j.appender.logstash.port=4560
log4j.appender.logstash.ReconnectionDelay=60000
log4j.appender.logstash.LocationInfo=true

配置文件中,把日志输出了四份:

  1. 第一份输出到控制台
  2. 第二份把DEBUG 级别以上的日志到文件
  3. 第三份把输出ERROR 级别以上的日志到文件
  4. 第四份输出到logstash

在java目录下添加TestLoggerMain.java,内容如下: 

import java.util.ArrayList;
import java.util.List;

import org.apache.log4j.Logger;

public class TestLoggerMain {
	public static final Logger logger = Logger.getLogger(TestLoggerMain.class);

	public static void main(String[] args) {

		logger.debug("This is a debug message!");
		logger.info("This is info message!");
		logger.warn("This is a warn message!");
		logger.error("This is error message!");
		List<Integer> list = new ArrayList<Integer>();
		list.add(1);
		list.add(2);
		list.add(3);

		try {
			for (Integer num : list) {
				list.remove(num);
			}

		} catch (Exception e) {
			logger.error(e);
		}
	}

}

配置logstash

创建logstash.conf配置文件

input {                                      #日志数据输入来源log4j
    log4j {
        host => "127.0.0.1"
        port => 4561
    }

}



output {
    stdout {
      codec => rubydebug
    }
    elasticsearch{
        hosts => ["localhost:9200"]
        index => "log4j-%{+YYYY.MM.dd}"
        document_type => "log4j_type"
    }
}

首先启动Elasticsearch,然后启动logstash和kibaba

logstash输出:

ELK日志处理之使用logstash收集log4J日志

进入kibaba查询:

{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 16,
    "successful": 16,
    "failed": 0
  },
  "hits": {
    "total": 2,
    "max_score": 1,
    "hits": [
      {
        "_index": ".kibana",
        "_type": "config",
        "_id": "5.2.0",
        "_score": 1,
        "_source": {
          "buildNum": 14695
        }
      },
      {
        "_index": "log4j-2019.01.08",
        "_type": "log4j_type",
        "_id": "AWgsBq4xsF3SfWsXEnpg",
        "_score": 1,
        "_source": {
          "method": "queryData",
          "thread": "http-nio-8025-exec-1",
          "priority": "ERROR",
          "type": "simple",
          "message": "nested exception is org.apache.ibatis.type.TypeException: Could not set parameters for mapping: ParameterMapping{property='level', mode=IN, javaType=class java.lang.Integer, jdbcType=null, numericScale=null, resultMapId='null', jdbcTypeName='null', expression='null'}. Cause: org.apache.ibatis.type.TypeException: Error setting non null for parameter #1 with JdbcType null . Try setting a different JdbcType for this parameter or a different configuration property. Cause: java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer",
          "path": "com.ekingwin.bas.cloud.solr.service.impl.JiucaiyunSolrServiceImpl",
          "@timestamp": "2019-01-08T05:54:20.018Z",
          "file": "JiucaiyunSolrServiceImpl.java:328",
          "@version": "1",
          "host": "127.0.0.1:56347",
          "logger_name": "com.ekingwin.bas.cloud.solr.service.impl.JiucaiyunSolrServiceImpl",
          "class": "com.ekingwin.bas.cloud.solr.service.impl.JiucaiyunSolrServiceImpl",
          "timestamp": 1546926859986,
        }
      }
    ]
  }
}