ELK部署logstash安装部署及应用(二)
Logstash 安装部署注意事项:
Logstash基本概念:
logstash收集日志基本流程: input-->codec-->filter-->codec-->output
input:从哪里收集日志。
filter:发出去前进行过滤
output:输出至Elasticsearch或Redis消息队列
codec:输出至前台,方便边实践边测试
数据量不大日志按照月来进行收集
如果通过logstash来采集日志,那么每个客户端都需要安装logstash
安装需要前置系统环境
安装java
下载并安装GPG key [[email protected]_agent yum.repos.d]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
配置yum仓库
[[email protected]_agent yum.repos.d]# echo '[logstash-2.3]> name=Logstash repository for 2.3.x packages > baseurl=https://packages.elastic.co/logstash/2.3/centos> gpgcheck=1> gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch > enabled=1' >>/etc/yum.repos.d/logstash.repo
安装logstash
[[email protected]_agent yum.repos.d]# yum install -y logstash
查看多种输出输出方式
实例1
[[email protected]_agent yum.repos.d]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }'
输出内容:
[[email protected]_agent yum.repos.d]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }'OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N Settings: Default pipeline workers: 1Pipeline main started hello2017-06-27T04:10:21.010Z master_agent hello hello word 2017-06-27T04:10:48.513Z master_agent hello wordquit2017-06-27T04:11:12.959Z master_agent quitexit2017-06-27T04:11:19.064Z master_agent exit
实例2
[[email protected]_agent yum.repos.d]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug} }'
[[email protected]_agent yum.repos.d]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug} }'OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N Settings: Default pipeline workers: 1Pipeline main started hello word { "message" => "hello word", "@version" => "1", "@timestamp" => "2017-06-27T04:13:14.560Z", "host" => "master_agent"}
实例3 不输出在屏幕上,直接记录在es里
[[email protected]_agent yum.repos.d]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["172.16.1.201:9200"] } }'OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N Settings: Default pipeline workers: 1Pipeline main started hello word!! Pipeline main has been shutdown
实例5
记录两份,屏幕输出一份、es记录一份 [[email protected]_agent yum.repos.d]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["172.16.1.201:9200"] } stdout{ codec => rubydebug } }'
[[email protected]_agent yum.repos.d]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["172.16.1.201:9200"] } stdout{ codec => rubydebug } }'OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N Settings: Default pipeline workers: 1Pipeline main started hello fly salt { "message" => "hello fly salt", "@version" => "1", "@timestamp" => "2017-06-27T04:32:50.297Z", "host" => "master_agent"}
查看web页面记录的内容
Logstash配置文件
配置Logstash,将创建一个配置文件,指定您想要使用的插件和设置为每个插件。 你可以参考事件字段在配置和使用条件符合一定时处理事件标准;当您运行logstash,你使用-f指定您的配置文件。 参考链接
实例如下:
[[email protected]_agent yum.repos.d]# vim /etc/logstash/conf.d/logstash.conf ##新建一个配置语法文件,让输入输出标准化
input { stdin { } }output { elasticsearch { hosts => ["localhost:9200"] } stdout { codec => rubydebug } }
[[email protected]_agent yum.repos.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf ### 执行命令配置语法文件 手动输入hao123.com,查看结果
[[email protected]_agent yum.repos.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N Settings: Default pipeline workers: 1Pipeline main started hao123.com { "message" => "hao123.com", "@version" => "1", "@timestamp" => "2017-06-27T05:06:02.695Z", "host" => "master_agent"}
input插件
创建一个文件夹用于专门存放logfile配置文件 存放的路径
[[email protected]_agent yum.repos.d]# mkdir -p /home/elk/logfile/
Input plugins file 收集实例:
编辑一个新的配置文件
[[email protected]_agent yum.repos.d]# mkdir -p /home/elk/logfile/[[email protected]_agent yum.repos.d]# cd /home/elk/logfile/[[email protected]_agent logfile]# ls[[email protected]_agent logfile]# vim file.conf
编写配置文件内容
input { file { path => "/var/log/messages" ###读取log日志文件路径 type => "system" ### 设置个类型 start_position => "beginning" ### 从日志最开始地方读取 } }output { elasticsearch { hosts => ["172.16.1.201:9200"] index => "system-%{+YYYY.MM.dd}" ##设置个索引每天创建 } }
注意:如果采集的日志文件日志量不是很大的情况下,采用每月建一从索引,日志量很大的情况下可以按照每天来建立索引
转载于:https://blog.51cto.com/yibeishui/1953414