Linux下轻量级监控系统搭建（Telegraf+Influxdb+Grafana）

Linux下监控系统搭建（Telegraf+Influxdb+Grafana）

 

一、安装文件准备（可提前去官网下载好）

 

telegraf-1.12.4-1.x86_64.rpm

influxdb-1.7.8.x86_64.rpm 单机的免费，集群的收费

grafana-6.4.3-1.x86_64.rpm

kapacitor-1.5.3.x86_64.rpm （TIGK技术栈的告警服务）

 

二、安装

 

1、创建存放软件目录

mkdir /home/ldw/monitor

把下载的安装文件上传到服务器的monitor目录下

登录到monitor所在目录下赋权

chmod -R 777 monitor

 

2、安装

安装命令：(如果是分布式监控，需要在其他client端安装telegraf)

rpm -ivh telegraf-1.12.4-1.x86_64.rpm

rpm -ivh influxdb-1.7.8.x86_64.rpm

rpm -ivh grafana-6.4.3-1.x86_64.rpm

rpm -ivh kapacitor-1.5.3.x86_64.rpm

 

安装过程：(登录到安装软件所在目录下)

[[email protected] monitor]# rpm -ivh telegraf-1.12.4-1.x86_64.rpm

准备中...                          ################################# [100%]

正在升级/安装...

   1:telegraf-1.12.4-1                ################################# [100%]

Created symlink from /etc/systemd/system/multi-user.target.wants/telegraf.service to /usr/lib/systemd/system/telegraf.service.

 

[[email protected] monitor]# rpm -ivh influxdb-1.7.8.x86_64.rpm

准备中...                          ################################# [100%]

正在升级/安装...

   1:influxdb-1.7.8-1                 ################################# [100%]

Created symlink from /etc/systemd/system/influxd.service to /usr/lib/systemd/system/influxdb.service.

Created symlink from /etc/systemd/system/multi-user.target.wants/influxdb.service to /usr/lib/systemd/system/influxdb.service.

 

[[email protected] monitor]# rpm -ivh grafana-6.4.3-1.x86_64.rpm

警告：grafana-6.4.3-1.x86_64.rpm: 头V4 RSA/SHA1 Signature, ** ID 24098cb6: NOKEY

准备中...                          ################################# [100%]

正在升级/安装...

   1:grafana-6.4.3-1                  ################################# [100%]

### NOT starting on installation, please execute the following statements to configure grafana to start automatically using systemd

 sudo /bin/systemctl daemon-reload

 sudo /bin/systemctl enable grafana-server.service

### You can start grafana-server by executing

 sudo /bin/systemctl start grafana-server.service

POSTTRANS: Running script

 

[[email protected] monitor]# rpm -ivh kapacitor-1.5.3.x86_64.rpm

准备中...                          ################################# [100%]

正在升级/安装...

   1:kapacitor-1.5.3-1                ################################# [100%]

 

监控软件安装后的配置文件地址如下：

/etc/telegraf/telegraf.conf

/etc/influxdb/influxdb.conf

/etc/grafana/grafana.ini

/etc/kapacitor/kapacitor.conf

 

监控软件安装后的log文件地址如下：

/var/log/telegraf/telegraf.log

/var/log/influxdb/influxdb.log

/var/log/grafana/grafana.log

 

Grafana插件地址：

/var/lib/grafana/plugins

 

Influxdb的后台文件保存位置：

/var/lib/influxdb/meta  #元数据/raft数据库的存储位置

/var/lib/influxdb/data  #TSM存储引擎存储TSM文件的目录

/var/lib/influxdb/wal   #TSM存储引擎存储WAL文件的目录

 

 

三、配置

 

1、Telegraf配置

[agent]

#修改数据采集间隔

interval = "5s"

 

[outputs.influxdb]

#修改对应的influxdb的url,IP修改成安装influxdb服务器的IP地址

urls = ["http://10.67.31.74:8086"]

#修改对应的influxdb的数据库名称，使用默认的telegraf就可以，后续启动influxdb数据库的时候要创建telegraf名称的数据库就可以。

database = "telegraf"

 

2、Influxdb配置

# Determines whether HTTP endpoint is enabled.主要作用是接收telegraf的数据并存储，提供API给Grafana调用数据

enabled = true

# The bind address used by the HTTP service.打开HTTP API使用的端口

bind-address = ":8086"

 

3、Grafana配置

# The public facing domain name used to access grafana from a browser 从浏览器访问grafana的面向公众的域名

;domain = 10.67.31.74

# The full public facing url you use in browser, used for redirects and emails 浏览器中使用的面向公众的完整url，用于重定向和电子邮件

;root_url = http://10.67.31.74:3000

默认的登录用户名密码都是admin，不用修改

 

 

四、启动

 

启动命令：

systemctl start telegraf

systemctl start influxdb

systemctl start grafana-server

 

查看启动情况

systemctl status telegraf

systemctl status influxdb

systemctl status grafana-server

 

停止命令：

systemctl stop telegraf

systemctl stop influxdb

systemctl stop grafana-server

 

 

五、Influxdb数据库配置

 

启动influxdb后，需要配置下数据库

 

[[email protected] ~]# influx

   Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.

   Connected to http://localhost:8086 version 1.0.2

   InfluxDB shell version: 1.0.2

   > create user "telegraf" with password 'telegraf'

   > show users;

   user     admin

   telegraf false

   > create database telegraf

   > show databases

   name: databases

   ---------------

   name

   _internal

   telegraf

 

#使用数据库

>use telegraf

 

#显示该数据库中所有的表

>show measurements

 

 

六、Grafana使用

 

登录Grafana

http://10.67.31.74:3000

用户名密码：admin/admin

 

登录后配置数据源

 

配置数据源：

 

 

提前下载了合适的Dashboard文件，直接导入。选用server-single_rev3.json

 

 

 

然后可以自己起个模板名字，选择influxdb类型数据库，点击import进行导入。

 

导入成功后，就可以进行模板的配置。

这个模板server-single_rev3.json有特殊的配置要求，需要重新配置telegraf，以下是配置信息，需要到linux后台重新配置telegraf.conf文件。

 

telegraf.conf重新配置：

--------------------------------------------------------------------------------------------------------

[global_tags]

 

  host = "$HOSTNAME"

  ##注意每个client都要配置自己的hostname

[agent]

 

  interval = "5m"

 

[[outputs.influxdb]]

 

  urls = ["http://mydomain.invalid:8086"]

 

  database = "servermonitor"

 

[[inputs.cpu]]

 

  percpu = false

 

  totalcpu = true

 

  collect_cpu_time = true

 

  fielddrop = ["time_guest","time_guest_nice","time_irq","time_nice","time_softirq","time_steal","usage_guest","usage_guest_nice","usage_irq","usage_nice","usage_softirq","usage_steal"]

 

  interval = "2s"

 

[[inputs.disk]]

 

  mount_points = ["/","/var","/data"]

 

  fielddrop=["used","inodes_used"]

 

[[inputs.mem]]

 

  fielddrop=["active","buffered","cached","free","inactive","used","used_percent"]

 

[[inputs.processes]]

 

[[inputs.swap]]

 

  fielddrop=["free","total"]

 

[[inputs.system]]

 

  fielddrop=["n_users","uptime_format"]

 

[[inputs.nstat]]

 

  interval = "2s"

 

  #proc_net_netstat = "" # this is of interest.

    ##注意：这条不知道别配，先注释掉，否则配置成空，telegraf会启动不了。

  fieldpass = ["IpExtOutOctets","IpExtInOctets"]

 

telegraf.conf文件配置完成后要重启telegraf。

 

可以通过脚本或者手动，重新启动telegraf+influxdb+grafana.重新登录grafana就可以看到下面的截图，保留自己想监控的指标，其他指标删除了就可以了。

这个模板的好处就是可以通过左上角的hostname来随时切换无服务。进行不同服务器的监控指标查看。

 

 

上面模板各个指标的配置条件导出：

CPU：

SELECT mean("n_cpus") FROM "system" WHERE ("host" =~ /^$host$/) AND $timeFilter GROUP BY time($interval), "host" fill(none)

SELECT mean("usage_system") FROM "cpu" WHERE ("host" =~ /^$host$/) AND $timeFilter GROUP BY time($interval), "host" fill(none)

SELECT mean("usage_user") FROM "cpu" WHERE ("host" =~ /^$host$/) AND $timeFilter GROUP BY time($interval), "host" fill(none)

SELECT mean("usage_iowait") FROM "cpu" WHERE ("host" =~ /^$host$/) AND $timeFilter GROUP BY time($interval), "host" fill(none)

 

RAM：

SELECT mean("available") FROM "mem" WHERE ("host" =~ /^$host$/) AND $timeFilter GROUP BY time($interval), "host" fill(null)

SELECT mean("total") FROM "mem" WHERE ("host" =~ /^$host$/) AND $timeFilter GROUP BY time($interval), "host" fill(null)

 

swap：

SELECT derivative(mean("in"), 1s) FROM "swap" WHERE ("host" =~ /^$host$/) AND $timeFilter GROUP BY time($interval), "host" fill(null)

SELECT derivative(mean("out"), 1s) FROM "swap" WHERE ("host" =~ /^$host$/) AND $timeFilter GROUP BY time($interval), "host" fill(null)

SELECT mean("used_percent") FROM "swap" WHERE ("host" =~ /^$host$/) AND $timeFilter GROUP BY time($interval), "host" fill(null)

 

Disk：

SELECT mean("total") FROM "disk" WHERE ("host" =~ /^$host$/) AND $timeFilter GROUP BY time($interval), "path" fill(null)

SELECT mean("free") FROM "disk" WHERE ("host" =~ /^$host$/) AND $timeFilter GROUP BY time($interval), "path" fill(null)

SELECT mean("inodes_total") FROM "disk" WHERE ("host" =~ /^$host$/) AND $timeFilter GROUP BY time($interval), "path" fill(null)

SELECT mean("inodes_free") FROM "disk" WHERE ("host" =~ /^$host$/) AND $timeFilter GROUP BY time($interval), "path" fill(null)

 

Processes：

SELECT mean("total") FROM "processes" WHERE ("host" =~ /^$host$/) AND $timeFilter GROUP BY time($interval) fill(null)

SELECT mean("running") FROM "processes" WHERE ("host" =~ /^$host$/) AND $timeFilter GROUP BY time($interval) fill(null)

SELECT mean("blocked") FROM "processes" WHERE ("host" =~ /^$host$/) AND $timeFilter GROUP BY time($interval) fill(null)

SELECT mean("stopped") FROM "processes" WHERE ("host" =~ /^$host$/) AND $timeFilter GROUP BY time($interval) fill(null)

SELECT max("blocked") FROM "processes" WHERE $timeFilter GROUP BY time($interval), "host" fill(null)

 

 

七、脚本

 

附件是一键启动、停止监控脚本。参考。

/home/ldw/monitor/script

 

脚本内容参考：

start.sh

ssh [email protected] 'systemctl start telegraf'&ssh [email protected] 'systemctl start influxdb'&ssh [email protected] 'systemctl start grafana-server'&ssh [email protected] 'systemctl start telegraf'&

stop.sh

ssh [email protected] 'systemctl stop telegraf'&ssh [email protected] 'systemctl stop influxdb'&ssh [email protected] 'systemctl stop grafana-server'&ssh [email protected] 'systemctl stop telegraf'&