ELK 搭建
一.环境准备
两台主机:
主机1 elk-master 192.168.93.14
主机2 elk-node 192.168.93.19
主机hosts
192.168.93.14 elk-master 安装elasticsearch和kibana
192.168.93.19 elk-node 安装elasticsearch和logstash
备注:版本为6.8.0-1
二.部署
1.安装jdk
下载jdk8 地址 https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
jdk-8u211-linux-x64.tar.gz
#tar xvf jdk-8u211-linux-x64.tar.gz
#mv jdk1.8.0_211 /usr/local/jdk1.8
#vim /etc/profile //添加如下
export JAVA_HOME=/usr/local/jdk1.8
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
export JRE_HOME=$JAVA_HOME/jre
#source /etc/profile
#java -version
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)
#ln -s /usr/local/jdk1.8/bin/java /usr/bin/
2.elasticsearch
1) 安装
[[email protected] ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
[[email protected] ~]# vim /etc/yum.repos.d/elasticsearch.repo //添加如下内容
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
[[email protected] ~]# yum install elasticsearch -y
2)配置
elasticsearch配置文件主要有两个,一个是/etc/elastcisearch/elasticsearch.yml 主要是配置节点信息,另一个是/etc/sysconfig/elasticsearch 配置elasticsearch本身的信息.
[[email protected] ~]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name: lt-elk
node.name: elk-master
node.master: true
node.data: false
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.93.14", "192.168.93.19"]
其他保持默认即可.
数据节点node配置如下
cluster.name: lt-elk
node.name: elk-node
node.master: false
node.data: true
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.93.14", "192.168.93.19"]
启动服务
[[email protected] ~]# systemctl enable elasticsearch.service
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
[[email protected] ~]# systemctl start elasticsearch.service
[[email protected] ~]# systemctl enable elasticsearch.service
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
[[email protected] ~]# systemctl start elasticsearch.service
查看下端口
[[email protected] ~]# netstat -lntp | grep java
tcp6 0 0 :::9200 :::* LISTEN 21817/java
tcp6 0 0 :::9300 :::* LISTEN 21817/java
// 9200 数据传输端口, 9300集群通信端口
测试下:
[[email protected] ~]# curl "http://192.168.93.14:9200"
{
"name" : "elk-master",
"cluster_name" : "lt-elk",
"cluster_uuid" : "PMzMag2vQsanNWbhzu3MZA",
"version" : {
"number" : "6.8.0",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "65b6179",
"build_date" : "2019-05-15T20:06:13.172855Z",
"build_snapshot" : false,
"lucene_version" : "7.7.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
查看下集群
[[email protected] ~]# curl "http://192.168.93.14:9200/_cluster/health?pretty"
{
"cluster_name" : "lt-elk",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
3.kibana
安装kibana
[[email protected] ~]# yum install kibana -y
配置
[[email protected] ~]# vim /etc/kibana/kibana.yml
server.port: 5601
server.host: "192.168.93.14"
elasticsearch.url: "http://192.168.93.14:9200"
kibana.index: ".kibana"
logging.dest: /var/log/kibana.log
启动
[[email protected] ~]# systemctl enable kibana.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /etc/systemd/system/kibana.service.
[[email protected] ~]# systemctl start kibana.service
4.安装logstash
[[email protected] ~]# yum install logstash -y
[[email protected] ~]# vim /etc/logstash/conf.d/syslog.conf
input{
syslog{
tyep => "system-log"
port => 10514
}
}
output{
stdout{
codec => rubydebug
}
}
[[email protected] logstash]# vim /etc/logstash/logstash.yml
http.host : 192.168.93.19
[[email protected] logstash]# systemctl restart logstash
验证下配置
[[email protected] ~]# cd /usr/share/logstash/bin
[[email protected] bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-06-07T11:59:55,496][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2019-06-07T12:00:00,480][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[[email protected] bin]#
// 显示OK,表示验证
配置rsyslog
[[email protected] logstash]# vim /etc/rsyslog.conf
*.* @192.168.93.19:10514
[[email protected] logstash]# systemctl restart rsyslog
验证配置
[[email protected] bin]# cd /usr/share/logstash/bin/
[[email protected] bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf
另找一台机器ssh登录到note上,
观察输出,类似如下
{
"facility_label" => "syslogd",
"message" => "action 'action 8' resumed (module 'builtin:omfwd') [v8.24.0-34.el7 try http://www.rsyslog.com/e/2359 ]\n",
"@timestamp" => 2019-06-07T14:21:18.000Z,
"@version" => "1",
"timestamp" => "Jun 7 22:21:18",
"logsource" => "elk-node",
"host" => "192.168.93.19",
"facility" => 5,
"type" => "system-log",
"program" => "rsyslogd",
"priority" => 46,
"severity_label" => "Informational",
"severity" => 6
}
{
"facility_label" => "security/authorization",
"message" => "New session 3 of user root.\n",
"@timestamp" => 2019-06-07T14:21:18.000Z,
"@version" => "1",
"timestamp" => "Jun 7 22:21:18",
"logsource" => "elk-node",
"host" => "192.168.93.19",
"facility" => 4,
"type" => "system-log",
"program" => "systemd-logind",
"priority" => 38,
"severity_label" => "Informational",
"severity" => 6
}
{
"facility_label" => "system",
"message" => "Started Session 3 of user root.\n",
"@timestamp" => 2019-06-07T14:21:18.000Z,
"@version" => "1",
"timestamp" => "Jun 7 22:21:18",
"logsource" => "elk-node",
"host" => "192.168.93.19",
"facility" => 3,
"type" => "system-log",
"program" => "systemd",
"priority" => 30,
"severity_label" => "Informational",
"severity" => 6
}
{
"facility_label" => "security/authorization",
"message" => "pam_unix(sshd:session): session opened for user root by (uid=0)\n",
"@timestamp" => 2019-06-07T14:21:18.000Z,
"@version" => "1",
"timestamp" => "Jun 7 22:21:18",
"logsource" => "elk-node",
"host" => "192.168.93.19",
"pid" => "12518",
"facility" => 10,
"type" => "system-log",
"program" => "sshd",
"priority" => 86,
"severity_label" => "Informational",
"severity" => 6
}
{
"facility_label" => "syslogd",
"message" => "action 'action 8' resumed (module 'builtin:omfwd') [v8.24.0-34.el7 try http://www.rsyslog.com/e/2359 ]\n",
"@timestamp" => 2019-06-07T14:21:18.000Z,
"@version" => "1",
"timestamp" => "Jun 7 22:21:18",
"logsource" => "elk-node",
"host" => "192.168.93.19",
"facility" => 5,
"type" => "system-log",
"program" => "rsyslogd",
"priority" => 46,
"severity_label" => "Informational",
"severity" => 6
}
{
"facility_label" => "security/authorization",
"message" => "Accepted password for root from 192.168.93.14 port 42568 ssh2\n",
"@timestamp" => 2019-06-07T14:21:18.000Z,
"@version" => "1",
"timestamp" => "Jun 7 22:21:18",
"logsource" => "elk-node",
"host" => "192.168.93.19",
"pid" => "12518",
"facility" => 10,
"type" => "system-log",
"program" => "sshd",
"priority" => 86,
"severity_label" => "Informational",
"severity" => 6
}
证明配置成功
修改配置文件,让收集的日志信息输出到master服务器中，而不是当前终端：
[[email protected] bin]# vim /etc/logstash/conf.d/syslog.conf
input{
syslog{
type => "system-log"
port => 10514
}
}
output{
elasticsearch{
hosts => ["192.168.93.14:9200"]
index => "system-log-%{+YYYY.MM}"
}
}
检查配置文件,并启动
[[email protected] bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-06-07T22:30:17,139][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2019-06-07T22:30:21,755][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[[email protected] bin]# systemctl start logstash
这里要修改下权限,不然虽然能成功启动,但监听不到端口
//chown logstash /var/log/logstash/logstash-plain.log
//chown -R logstash /var/lib/logstash/
检查下端口,如下,启动成功
[[email protected] bin]# netstat -lntp | grep 9600
tcp6 0 0 192.168.93.19:9600 :::* LISTEN 12587/java
[[email protected] bin]#
[[email protected] bin]# netstat -lntp | grep 10514
tcp6 0 0 :::10514 :::* LISTEN 12587/java
5.配置kibana索引
建立索引index
完成后,点击Discover
使用beats采集日志
[[email protected] ~]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.7.0-x86_64.rpm
[[email protected] ~]# rpm -ivh filebeat-6.7.0-x86_64.rpm
准备中... ################################# [100%]
正在升级/安装...
1:filebeat-6.7.0-1 ################################# [100%]
修改配置文件
[[email protected] ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
paths:
- /var/log/messages
output.elasticsearch:
hosts: ["192.168.93.14:9200"]
启动filebeat
[[email protected] ~]# systemctl start filebeat
服务端检查
[[email protected] ~]# curl "192.168.93.14:9200/_cat/indices?v"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open nginx-log-2019.06.07 q5t0CEG8SPajfnuaqQrTYw 5 1 7 0 92.2kb 46.1kb
green open system-log-2019.06 hG2UzU9AQ2SpQS6tat-gNQ 5 1 27 0 347.2kb 173.6kb
green open .kibana_task_manager bkj5j4MmQf6Kf5wb3_UeIA 1 1 2 0 25.1kb 12.5kb
green open .kibana_1 v5BpxX-JTWqJP3OMyvoClQ 1 1 6 0 57.5kb 28.7kb
green open filebeat-6.8.0-2019.06.07 MjFxU6A1Q6OhKrU0mNGATw 3 1 0 0 80.6kb 460b
看到出现filebeat-6.8.0-2019.06.07 索引文件表示正常.