Elasticsearch集群部署

1 序言

Elasticsearch集群部署，只需要配置elasticsearch.yml配置文件，各个节点分别启动即可自动识别

2 环境

Window 7下创建三个虚拟机：centos6.7，jdk1.8，elasticsearch6.1.3

虚拟机配置如下：

类别	主机名	ip	内存	硬盘	主要目录	注释
Master	master	192.168.43.30	2G	50G	/soft/elasticsearch-6.1.3/data	索引目录
					/soft/elasticsearch-6.1.3/log	日志目录
segments	slave1	192.168.43.31	2G	50G	/soft/elasticsearch-6.1.3/data	索引目录
					/soft/elasticsearch-6.1.3/log	日志目录
segments	slave2	192.168.43.32	2G	50G	/soft/elasticsearch-6.1.3/data	索引目录
					/soft/elasticsearch-6.1.3/log	日志目录

3 创建es用户

用户组：[[email protected] ~]# groupadd elasticsearch

用户：[[email protected] ~]# useradd es

密码：[[email protected] ~]# passwd es

用户添加到用户组：[[email protected] ~]# usermod -G elasticsearch es

设置用户sudo权限：

[[email protected] ~]# visudo

在root ALL=(ALL) ALL 一行下面

添加es用户如下:

es ALL=(ALL) ALL

4 安装es

4.1 安装包处理

下载：https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.1.3.tar.gz

上传：分别上传到各节点/soft目录下

解压：tar -xvf /soft/elasticsearch-6.1.3.tar.gz

修改文件的所属用户:sudo chown -R es:elasticsearch /soft/elasticsearch-6.1.3

4.2 es配置

切换用户：su - es

进入安装包目录：cd /soft/elasticsearch-6.1.3

修改配置文件：vim config/elasticsearch.yml，具体内容如下：

# ======================== Elasticsearch Configuration =========================

# NOTE: Elasticsearch comes with reasonable defaults for most settings.

# Before you set out to tweak and tune the configuration, make sure you

# understand what are you trying to accomplish and the consequences.

# The primary way of configuring a node is via this file. This template lists

# the most important settings you may want to configure for a production cluster.

# Please consult the documentation for further information on configuration options:

# https://www.elastic.co/guide/en/elasticsearch/reference/index.html

# ---------------------------------- Cluster -----------------------------------

# Use a descriptive name for your cluster:

cluster.name: my-es

# ------------------------------------ Node ------------------------------------

# Use a descriptive name for the node:

node.name: node-1

# Add custom attributes to the node:

node.attr.rack: r1

# ----------------------------------- Paths ------------------------------------

# Path to directory where to store the data (separate multiple locations by comma):

path.data: /soft/elasticsearch-6.1.3/data

# Path to log files:

path.logs: /soft/elasticsearch-6.1.3/logs

# ----------------------------------- Memory -----------------------------------

# Lock the memory on startup:

bootstrap.memory_lock: false

# Make sure that the heap size is set to about half the memory available

# on the system and that the owner of the process is allowed to use this

# limit.

# Elasticsearch performs poorly when the system is swapping the memory.

# ---------------------------------- Network -----------------------------------

# Set the bind address to a specific IP (IPv4 or IPv6):

network.host: 192.168.43.30

# Set a custom port for HTTP:

http.port: 9200

# For more information, consult the network module documentation.

# --------------------------------- Discovery ----------------------------------

# Pass an initial list of hosts to perform discovery when new node is started:

# The default list of hosts is ["127.0.0.1", "[::1]"]

node.master: true

node.data: true

node.ingest: true

transport.tcp.port: 9300

discovery.zen.ping.unicast.hosts: ["192.168.43.30:9300","192.168.43.31:9300","192.168.43.32:9300"]

# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):

discovery.zen.minimum_master_nodes: 2

# For more information, consult the zen discovery module documentation.

# ---------------------------------- Gateway -----------------------------------

# Block initial recovery after a full cluster restart until N nodes are started:

#gateway.recover_after_nodes: 3

# For more information, consult the gateway module documentation.

# ---------------------------------- Various -----------------------------------

# Require explicit names when deleting indices:

#action.destructive_requires_name: true

http.cors.enabled: true

http.cors.allow-origin: "*"

bootstrap.system_call_filter: false

备注：另外两个节点配置：node.name: node-2、node.name: node-3；node.attr.rack: r2、r3；network.host: 192.168.43.31、network.host: 192.168.43.32，其它保持不变

4.3 elasticsearch.yml配置项说明

cluster.name:elasticsearch

配置es的集群名称，默认是elasticsearch，es会自动发现在同一网段下的es，如果在同一网段下有多个集群，就可以用这个属性来区分不同的集群。
node.name:”FranzKafka”

节点名，默认随机指定一个name列表中名字，该列表在es的jar包中config文件夹里name.txt文件中，其中有很多作者添加的有趣名字。
node.master:true

指定该节点是否有资格被选举成为node，默认是true，es是默认集群中的第一台机器为master，如果这台机挂了就会重新选举master。
node.data:true

指定该节点是否存储索引数据，默认为true。
index.number_of_shards:5
设置默认索引分片个数，默认为5片。
index.number_of_replicas:1
设置默认索引副本个数，默认为1个副本。
path.conf:/path/to/conf
设置配置文件的存储路径，默认是es根目录下的config文件夹。
path.data:/path/to/data
设置索引数据的存储路径，默认是es根目录下的data文件夹，可以设置多个存储路径，用逗号隔开，例：
path.data:/path/to/data1,/path/to/data2
path.work:/path/to/work
设置临时文件的存储路径，默认是es根目录下的work文件夹。
path.logs:/path/to/logs
设置日志文件的存储路径，默认是es根目录下的logs文件夹
path.plugins:/path/to/plugins
设置插件的存放路径，默认是es根目录下的plugins文件夹
bootstrap.mlockall:true
设置为true来锁住内存。因为当jvm开始swapping时es的效率会降低，所以要保证它不swap，可以把ES_MIN_MEM和ES_MAX_MEM两个环境变量设置成同一个值，并且保证机器有足够的内存分配给es。同时也要允许elasticsearch的进程可以锁住内存，linux下可以通过`ulimit-lunlimited`命令。
network.bind_host:192.168.0.1
设置绑定的ip地址，可以是ipv4或ipv6的，默认为0.0.0.0。network.publish_host:192.168.0.1
设置其它节点和该节点交互的ip地址，如果不设置它会自动判断，值必须是个真实的ip地址。
network.host:192.168.0.1
这个参数是用来同时设置bind_host和publish_host上面两个参数。
transport.tcp.port:9300
设置节点间交互的tcp端口，默认是9300。
transport.tcp.compress:true
设置是否压缩tcp传输时的数据，默认为false，不压缩。
http.port:9200
设置对外服务的http端口，默认为9200。
http.max_content_length:100mb
设置内容的最大容量，默认100mb
http.enabled:false
是否使用http协议对外提供服务，默认为true，开启。
gateway.type:local
gateway的类型，默认为local即为本地文件系统，可以设置为本地文件系统，分布式文件系统，hadoop的HDFS，和amazon的s3服务器，其它文件系统的设置方法下次再详细说。
gateway.recover_after_nodes:1
设置集群中N个节点启动时进行数据恢复，默认为1。
gateway.recover_after_time:5m
设置初始化数据恢复进程的超时时间，默认是5分钟。
gateway.expected_nodes:2
设置这个集群中节点的数量，默认为2，一旦这N个节点启动，就会立即进行数据恢复。
cluster.routing.allocation.node_initial_primaries_recoveries:4
初始化数据恢复时，并发恢复线程的个数，默认为4。
cluster.routing.allocation.node_concurrent_recoveries:2
添加删除节点或负载均衡时并发恢复线程的个数，默认为4。
indices.recovery.max_size_per_sec:0
设置数据恢复时限制的带宽，如入100mb，默认为0，即无限制。
indices.recovery.concurrent_streams:5
设置这个参数来限制从其它分片恢复数据时最大同时打开并发流的个数，默认为5。
discovery.zen.minimum_master_nodes:1
设置这个参数来保证集群中的节点可以知道其它N个有master资格的节点。默认为1，对于大的集群来说，可以设置大一点的值（2-4）
discovery.zen.ping.timeout:3s
设置集群中自动发现其它节点时ping连接超时时间，默认为3秒，对于比较差的网络环境可以高点的值来防止自动发现时出错。
discovery.zen.ping.multicast.enabled:false
设置是否打开多播发现节点，默认是true。
discovery.zen.ping.unicast.hosts:[“host1″,”host2:port”,”host3[portX-portY]”]
设置集群中master节点的初始列表，可以通过这些节点来自动发现新加入集群的节点

4.4 修改系统设置

1.切换回root用户

2.修改sysctl.conf：

vim /etc/sysctl.conf

在文件最后添加以下内容

vm.max_map_count=655360

保存后退出，再执行 sysctl -p 是设置立即生效

sysctl -p

3.修改limits.conf：

vim /etc/security/limits.conf

尾部追加内容如下

* soft nofile 65536

* hard nofile 65536

* soft nproc 131072

* hard nproc 131072

4.修改90-nproc.conf

vi /etc/security/limits.d/90-nproc.conf

修改如下内容：

* soft nproc 1024

#修改为

* soft nproc 4096

5.其余两个节点均需修改，完成后重启

6.其余两个节点均需修改

4.5 启动es

[[email protected] ~]# su - es

[[email protected] ~]$ cd /soft/elasticsearch-6.1.3

[[email protected] elasticsearch-6.1.3]$ ./bin/elasticsearch

4.6 查看启动是否成功

1.节点执行：“[[email protected] ~]# curl -X GET "192.168.43.30:9200/"”，结果如下：

{

"name" : "node-1",

"cluster_name" : "my-es",

"cluster_uuid" : "SzSwRqiqQbmA-LGe7Y2hpA",

"version" : {

"number" : "6.1.3",

"build_hash" : "af51318",

"build_date" : "2018-01-26T18:22:55.523Z",

"build_snapshot" : false,

"lucene_version" : "7.1.0",

"minimum_wire_compatibility_version" : "5.6.0",

"minimum_index_compatibility_version" : "5.0.0"

"tagline" : "You Know, for Search"

}

2.节点执行：[[email protected] ~]# curl '192.168.43.30:9200/_cat/health?v'

Elasticsearch集群部署

4.7 关闭es

采用杀进程的方式

[[email protected] ~]# jps

3143 Jps

2988 Elasticsearch

[[email protected] ~]# kill -9 2988

5 安装elasticsearch-head插件

5.1 安装node

安装elasticsearch-head插件需要nodejs的支持，所以此处讲解一下安装nodejs步骤

下载：https://nodejs.org/en/download/

Elasticsearch集群部署

上传解压：[[email protected] soft]# tar -xvf node-v10.15.0-linux-x64

[[email protected] soft]# cd node-v10.15.0-linux-x64/bin/

[[email protected] bin]# pwd

/soft/node-v10.15.0-linux-x64/bin

添加环境变量：[[email protected] bin]# vim /etc/profile

Elasticsearch集群部署

[[email protected] bin]# source /etc/profile

查看node -v查看安装是否成功：[[email protected] bin]# node -v

Elasticsearch集群部署

5.2 安装elasticsearch-head插件

下载：https://github.com/mobz/elasticsearch-head

上传并解压：[[email protected] soft]# unzip elasticsearch-head-master.zip

安装：

[[email protected] soft]# cd elasticsearch-head-master

[[email protected] elasticsearch-head-master]# npm install

npm WARN deprecated [email protected]: CoffeeScript on NPM has moved to "coffeescript" (no hyphen)

npm WARN deprecated [email protected]: Use the built-in module in node 9.0.0 or newer, instead

> [email protected] install /soft/elasticsearch-head-master/node_modules/phantomjs-prebuilt

> node install.js

PhantomJS not found on PATH

Downloading https://github.com/Medium/phantomjs/releases/download/v2.1.1/phantomjs-2.1.1-linux-x86_64.tar.bz2

Saving to /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2

Receiving...

[========================================] 100%

Received 22866K total.

Extracting tar contents (via spawned process)

Removing /soft/elasticsearch-head-master/node_modules/phantomjs-prebuilt/lib/phantom

Copying extracted folder /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2-extract-1548727009699/phantomjs-2.1.1-linux-x86_64 -> /soft/elasticsearch-head-master/node_modules/phantomjs-prebuilt/lib/phantom

Phantom installation failed { [Error: EACCES: permission denied, link '/tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2-extract-1548727009699/phantomjs-2.1.1-linux-x86_64' -> '/soft/elasticsearch-head-master/node_modules/phantomjs-prebuilt/lib/phantom']

errno: -13,

code: 'EACCES',

syscall: 'link',

path:

'/tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2-extract-1548727009699/phantomjs-2.1.1-linux-x86_64',

dest:

'/soft/elasticsearch-head-master/node_modules/phantomjs-prebuilt/lib/phantom' } Error: EACCES: permission denied, link '/tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2-extract-1548727009699/phantomjs-2.1.1-linux-x86_64' -> '/soft/elasticsearch-head-master/node_modules/phantomjs-prebuilt/lib/phantom'

npm WARN [email protected] license should be a valid SPDX license expression

npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/fsevents):

npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

npm ERR! code ELIFECYCLE

npm ERR! errno 1

npm ERR! [email protected] install: `node install.js`

npm ERR! Exit status 1

npm ERR!

npm ERR! Failed at the [email protected] install script.

npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:

npm ERR! /root/.npm/_logs/2019-01-29T01_56_56_057Z-debug.log

执行该命名可能会出现以上错误，此时忽略[email protected]，执行命令如下：

[[email protected] elasticsearch-head-master]# npm install [email protected] --ignore-scripts

npm notice created a lockfile as package-lock.json. You should commit this file.

npm WARN [email protected] license should be a valid SPDX license expression

npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/fsevents):

npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

+ [email protected]

added 62 packages from 64 contributors, removed 4 packages and audited 1717 packages in 6.853s

found 23 vulnerabilities (16 low, 1 moderate, 6 high)

run `npm audit fix` to fix them, or `npm audit` for details

启动：[[email protected] elasticsearch-head-master]# npm run start

> [email protected] start /soft/elasticsearch-head-master

> grunt server

>> Local Npm module "grunt-contrib-jasmine" not found. Is it installed?

Running "connect:server" (connect) task

Waiting forever...

Started connect web server on http://192.168.43.30:9100:9100

查看：http://192.168.43.30:9100/

Elasticsearch集群部署

5.3 启动连接elasticsearch

切换到es启动elasticsearch集群：[[email protected] elasticsearch-6.1.3]# ./bin/elasticsearch

http://192.168.43.30:9100/，并连接到http://192.168.43.30:9200/，如图：

Elasticsearch集群部署

6 安装Kibana

6.1 定义

Kibana是Elasticsearch的开源数据可视化插件，为查看存储在ElasticSearch提供了友好的Web界面，并提供了条形图，线条和散点图，饼图和地图等分析工具

6.2 安装

下载：https://artifacts.elastic.co/downloads/kibana/kibana-6.1.3-linux-x86_64.tar.gz

上传并解压：[[email protected] soft]# tar -xvf kibana-6.1.3-linux-x86_64

配置：[[email protected] soft]# cd kibana-6.1.3-linux-x86_64/config/

[[email protected] config]# vim kibana.yml ，修改以下属性：

server.port: 5601

server.host: "192.168.43.30"

elasticsearch.url: "http://192.168.43.30:9200"

启动：先启动elasticsearch集群，再启动kibana

[[email protected] kibana-6.1.3-linux-x86_64]# ./bin/kibana

log [02:58:25.944] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready

log [02:58:26.018] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [02:58:26.073] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready

log [02:58:26.135] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready

log [02:58:26.713] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready

log [02:58:26.720] [info][listening] Server running at http://192.168.43.30:5601

log [02:58:26.836] [info][status][plugin:[email protected]] Status changed from yellow to green - Ready

验证：http://192.168.43.30:5601

Elasticsearch集群部署

Elasticsearch集群部署

1 序言

2 环境

3 创建es用户

4 安装es

5 安装elasticsearch-head插件

6 安装Kibana

相关推荐