elk群集收集HTTP日志部署

主机名	软件包
node-1      192.168.100.11	elasticsearch-2.3.3.tar.gz jdk-8u181-linux-x64.tar.gz  kibana-4.5.1-linux-x64.tar.gz  logstash-2.3.3.tar.gz
node-2      192.168.100.12	elasticsearch-2.3.3.tar.gz jdk-8u181-linux-x64.tar.gz
syslog-client   192.168.100.13	logstash-2.3.3.tar.gz jdk-8u181-linux-x64.tar.gz

设置主机名和hosts文件

[[email protected] ~]# vim /etc/hosts   ##三台机器同步设置
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.100.11 node-1
192.168.100.12 node-2
192.168.100.13 syslog-client

[[email protected] ~]# hostnamectl set-hostname node-1
[[email protected] ~]# hostnamectl set-hostname node-2
[[email protected] ~]# hostnamectl set-hostname syslog-client

设置Java环境三台一样步骤

[[email protected] ~]# vim /etc/profile
export JAVA_HOME=/usr/local/java
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=$JAVA_HOME/jre/lib/ext:$JAVA_HOME/lib/tools.jar
[[email protected] ~]# source /etc/profile
[[email protected] ~]# tar zxvf jdk-8u181-linux-x64.tar.gz
[[email protected] ~]# mv jdk1.8.0_181/ /usr/local/java

安装elasticsearch（node-1和node-2一样步骤，node-2就不写了）

[[email protected] ~]# tar zxvf elasticsearch-2.3.3.tar.gz -C /usr/local/
[[email protected] ~]# useradd user1
[[email protected] ~]# passwd user1
更改用户 user1 的密码 。
新的 密码：
无效的密码： 密码少于 8 个字符
重新输入新的 密码：
passwd：所有的身份验证令牌已经成功更新。
[[email protected] elasticsearch-2.3.3]# chown -R user1:user1 /usr/local/elasticsearch-2.3.3
[[email protected] ~]# vim /usr/local/elasticsearch-2.3.3/config/elasticsearch.yml
 cluster.name: my-application   #取消注释
 node.name: node-1              #取消注释node-2的修改为node-2
 path.data: /usr/local/elasticsearch-2.3.3/data     #取消注释修改路径
 path.logs: /usr/local/elasticsearch-2.3.3/logs     #取消注释修改路径
 bootstrap.mlockall: true         #取消注释              
 network.host: 192.168.100.11     #取消注释修改为本机ip
 http.port: 9200
 discovery.zen.ping.unicast.hosts: ["192.168.100.11", "192.168.100.12"] #取消注释改为node1和2的IP
 discovery.zen.minimum_master_nodes: 1    #取消注释 改为1
[[email protected] elasticsearch-2.3.3]# su - user1
[[email protected] ~]$ /usr/local/elasticsearch-2.3.3/bin/elasticsearch


node-2配置步骤相同  注意把IP改为本机的就OK

打开浏览器访问本机IP的9200端口查看elasticsearch

 

 查看集群的健康情况

http://192.168.100.11:9200/_cluster/health?pretty

查看群集的状态信息

http://192.168.100.11:9200/_cluster/state?pretty

安装elasticsearch-kopf插件查询elasticsearch集群数据

[[email protected] elasticsearch-2.3.3]# ./bin/plugin install lmenezes/elasticsearch-kopf
-> Installing lmenezes/elasticsearch-kopf...
Trying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip ...
Downloading .............................................................................................................................................................................................DONE
Verifying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
Installed kopf into /usr/local/elasticsearch-2.3.3/plugins/kopf
[[email protected] elasticsearch-2.3.3]# ls plugins/  #安装完成会看到目录下的kopf
kopf

访问http://192.168.100.11:9200/_plugin/kopf

 安装logstash（syslog-client）

[[email protected] ~]# tar zxvf logstash-2.3.3
[[email protected] ~]# cd logstash-2.3.3/bin/
[[email protected] bin]# vim log_system.conf
input {
        file {
          path=>"/var/log/messages"
          type=>"system"
          start_position=>"beginning"
             }
      }

output {
        elasticsearch {
          hosts=>["192.168.100.11:9200"]
          index=>"system-%{+YYYY.MM.dd}"
             }
       }
[[email protected] bin]# ./logstash -f log_system.conf 
Settings: Default pipeline workers: 1
Pipeline main started

安装kibana-4.5.1（node-1）

[[email protected] ~]# tar zxvf kibana-4.5.1-linux-x64.tar.gz -C /usr/local/
[[email protected] ~]# vim /usr/local/kibana-4.5.1-linux-x64/config/kibana.yml 
 elasticsearch.url: "http://192.168.100.11:9200"   #改为本机elasticsearch的IP
 kibana.index: ".kibana"   #取消注释
[[email protected] ~]# /usr/local/kibana-4.5.1-linux-x64/bin/kibana 
  log   [22:09:17.854] [info][status][plugin:kibana] Status changed from uninitialized to green - Ready
  log   [22:09:17.965] [info][status][plugin:elasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [22:09:18.080] [info][status][plugin:kbn_vislib_vis_types] Status changed from uninitialized to green - Ready
  log   [22:09:18.096] [info][status][plugin:markdown_vis] Status changed from uninitialized to green - Ready
  log   [22:09:18.106] [info][status][plugin:metric_vis] Status changed from uninitialized to green - Ready
  log   [22:09:18.110] [info][status][plugin:spyModes] Status changed from uninitialized to green - Ready
  log   [22:09:18.203] [info][status][plugin:statusPage] Status changed from uninitialized to green - Ready
  log   [22:09:18.216] [info][status][plugin:table_vis] Status changed from uninitialized to green - Ready
  log   [22:09:18.225] [info][listening] Server running at http://0.0.0.0:5601
  log   [22:09:23.220] [info][status][plugin:elasticsearch] Status changed from yellow to yellow - No existing Kibana index found
  log   [22:09:28.876] [info][status][plugin:elasticsearch] Status changed from yellow to green - Kibana index ready

访问本机的5601端口

 

在syslog-client安装HTTP并访问

查看搜集的日志信息