从零开始搭建完整分布式系统
文章附件下载:https://download.****.net/download/wq8987/10442232
nginx下载:https://download.****.net/download/wq8987/10442237
构架方案:
说明:
- 负载均衡(2c4g):2台lvs+keepalived作为负载均衡服务器,一主一备,实现高可用;
- 反向代理(4c8g):2台nginx作为web服务反向代理服以实现WEB服务的高可用;
- WEB服务器集群(4c8g):2台服务器,每台服务器启用多应用的方式实现服务集群;
- 缓存服务器(4c16g):2台服务器实现redis服务端集群;
- 数据库(8c16g):2台MYSQL;
- #uname -r
- 2.6.18-53.el5PAE
- #ln -s /usr/src/kernels/2.6.18-53.el5PAE-i686/ /usr/src/linux
- #tar zxvf ipvsadm-1.24.tar.gz
- #cd ipvsadm-1.24
- #make && make install
- #find / -name ipvsadm
#vi /usr/local/sbin/lvs-dr.sh
- #!/bin/bash
- # description: start LVS of DirectorServer
- #Written by :[email protected]
- # website director vip.
- SNS_VIP=61.164.122.8
- SNS_RIP1=61.164.122.9
- SNS_RIP2=61.164.122.10
- NETWORK_NAME=eth0 #网卡接口名称:ifconfig查看即可
- . /etc/rc.d/init.d/functions
- logger $0 called with $1
- case "$1" in
- start)
- # set squid vip
- /sbin/ipvsadm --set 30 5 60
- /sbin/ifconfig $NETWORK_NAME:0 $SNS_VIP broadcast $SNS_VIP netmask 255.255.255.255
- broadcast $SNS_VIP up
- /sbin/route add -host $SNS_VIP dev eth0:0
- /sbin/ipvsadm -A -t $SNS_VIP:80 -s wrr -p 3
- /sbin/ipvsadm -a -t $SNS_VIP:80 -r $SNS_RIP1:80 -g -w 1
- /sbin/ipvsadm -a -t $SNS_VIP:80 -r $SNS_RIP2:80 -g -w 1
- touch /var/lock/subsys/ipvsadm >/dev/null 2>&1
- ;;
- stop)
- /sbin/ipvsadm -C
- /sbin/ipvsadm -Z
- ifconfig $NETWORK_NAME:0 down
- ifconfig $NETWORK_NAME:1 down
- route del $SNS_VIP
- route del $SS_VIP
- rm -rf /var/lock/subsys/ipvsadm >/dev/null 2>&1
- echo "ipvsadm stoped"
- ;;
- status)
- if [ ! -e /var/lock/subsys/ipvsadm ];then
- echo "ipvsadm stoped"
- exit 1
- else
- echo "ipvsadm OK"
- fi
- ;;
- *)
- echo "Usage: $0 {start|stop|status}"
- exit 1
- esac
- exit 0
keepalived安装:
- #tar zxvf keepalived-1.1.15.tar.gz
- #cd keepalived-1.1.15
- #./configure && make && make install
- #find / -name keepalived # 查看keepalived位置
- #cp /usr/local/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/
- #cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/
- #mkdir /etc/keepalived
- #cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/
- #cp /usr/local/sbin/keepalived /usr/sbin/
rpm -e --nodeps keyutils-libs-1.4-4.el6.x86_64rpm -ivh keyutils-libs-1.4-5.el6.x86_64.rpmrpm -ivh keyutils-libs-devel-1.4-5.el6.x86_64.rpmrpm -ivh libsepol-devel-2.0.41-4.el6.x86_64.rpmrpm -e --nodeps libselinux-utils-2.0.94-5.3.el6_4.1.x86_64rpm -Uvh libselinux-2.0.94-5.8.el6.x86_64.rpmrpm -ivh libselinux-devel-2.0.94-5.8.el6.x86_64.rpmrpm -e --nodeps krb5-libs-1.10.3-10.el6_4.6.x86_64rpm -ivh krb5-libs-1.10.3-42.el6.x86_64.rpmrpm -e --nodeps libcom_err-1.41.12-18.el6.x86_64rpm -ivh libcom_err-1.41.12-22.el6.x86_64.rpmrpm -ivh libcom_err-devel-1.41.12-22.el6.x86_64.rpmrpm -ivh krb5-devel-1.10.3-42.el6.x86_64.rpmrpm -ivh zlib-devel-1.2.3-29.el6.x86_64.rpmrpm -e --nodeps openssl-1.0.1e-15.el6.x86_64rpm -ivh openssl-1.0.1e-42.el6.x86_64.rpm
rpm -ivh openssl-devel-1.0.1e-42.el6.x86_64.rpm
rpm -ivh popt-devel-1.13-7.el6.x86_64.rpm
vi /etc/keepalived/keepalived.conf
- ! Configuration File for keepalived
- global_defs {
- notification_email {
- [email protected]
- }
- notification_email_from [email protected]
- smtp_server 127.0.0.1
- # smtp_connect_timeout 30
- router_id LVS_DEVEL
- }
- # 20170615 written by :wangqiang
- # VIP1
- vrrp_instance VI_1 {
- state MASTER #备份服务器上将MASTER改为BACKUP
- interface eth0
- virtual_router_id 51
- priority 100 # 备份服务上将100改为99
- advert_int 1
- authentication {
- auth_type PASS
- auth_pass 1111
- }
- virtual_ipaddress {
- 61.164.122.8
- #(如果有多个VIP,继续换行填写.)
- }
- }
- virtual_server 61.164.122.8 80 {
- delay_loop 6 #(每隔10秒查询realserver状态)
- lb_algo wrr #(lvs 算法)
- lb_kind DR #(Direct Route)
- persistence_timeout 60 #(同一IP的连接60秒内被分配到同一台realserver)
- protocol TCP #(用TCP协议检查realserver状态)
- real_server 61.164.122.9 80 {
- weight 3 #(权重)
- TCP_CHECK {
- connect_timeout 10 #(10秒无响应超时)
- nb_get_retry 3
- delay_before_retry 3
- connect_port 80
- }
- }
- real_server 61.164.122.10 80 {
- weight 3
- TCP_CHECK {
- connect_timeout 10
- nb_get_retry 3
- delay_before_retry 3
- connect_port 80
- }
- }
- }
nginx安装
cp -f nginx.tar /usr/localcd /usr/local/tar -xf nginx.tarcd nginxmake && make install
#user nobody;worker_processes 4;worker_cpu_affinity 00000001 00000010 00000100 00001000;worker_rlimit_nofile 204800;error_log logs/error.log debug;#pid /var/run/nginx.pid;events {worker_connections 204800;}http {#cache config..##proxy_connect_timeout 5;proxy_read_timeout 60;proxy_send_timeout 5;proxy_buffer_size 16k;proxy_buffers 4 64k;proxy_busy_buffers_size 128k;proxy_temp_file_write_size 128k;proxy_temp_path /home/temp_dir;proxy_cache_path /home/cache levels=1:2 keys_zone=cache_one:200m inactive=1d max_size=30g;#proxy_cache_key $host$uri$is_args$args; #$host$uri$is_args$args; #增加设置web缓存的key值,nginx根据key值md5哈希存储缓存#..cache config##log_format main '$remote_addr - $remote_user [$time_local] "$request" ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent" "$http_x_forwarded_for"''"$upstream_cache_status"';access_log logs/access.log main;sendfile on;#tcp_nopush on;keepalive_timeout 65;server_tokens off;upstream ngServer {#ip_hash;server 10.151.31.47:8080;server 10.151.31.46:8080;}include conf.d/*.conf;}
- #vi /usr/local/sbin/realserver.sh
- #!/bin/bash
- # description: Config realserver lo and apply noarp
- #Written by :[email protected]yeepay.com
- SNS_VIP=61.164.122.8
- . /etc/rc.d/init.d/functions
- case "$1" in
- start)
- ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP
- /sbin/route add -host $SNS_VIP dev lo:0
- echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
- echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
- echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
- echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
- sysctl -p >/dev/null 2>&1
- echo "RealServer Start OK"
- ;;
- stop)
- ifconfig lo:0 down
- route del $SNS_VIP >/dev/null 2>&1
- echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
- echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
- echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
- echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
- echo "RealServer Stoped"
- ;;
- *)
- echo "Usage: $0 {start|stop}"
- exit 1
- esac
- exit 0
1. 运行平台:CentOS 6.3 x86_64,基本等同于RHEL 6.3
2. 安装方法:
安装MySQL主要有两种方法:一种是通过源码自行编译安装,这种适合高级用户定制mysql的特性,这里不做说明;另一种是通过编译过的二进制文件进行安装。二进制文件安装的方法又分为两种:一种是不针对特定平台的通用安装方法,使用的二进制文件是后缀为.tar.gz的压缩文件;第二种是使用RPM或其他包进行安装,这种安装进程会自动完成系统的相关配置,所以比较方便。
3. 下载安装包:
a. 官方下载地址:
http://dev.mysql.com/downloads/mysql/#downloads
或镜像文件下载:
http://dev.mysql.com/downloads/mirrors.html
2. 下载文件(根据操作系统选择相应的发布版本):
a. 通用安装方法
mysql-5.5.29-linux2.6-x86_64.tar.gz
b. RPM安装方法:
MySQL-server-5.5.29-2.el6.x86_64.rpm
MySQL-client-5.5.29-2.el6.x86_64.rpm
4. 通用安装步骤
a. 检查是否已安装,grep的-i选项表示匹配时忽略大小写
[[email protected] JavaEE]#rpm -qa|grep -i mysql
mysql-libs-5.1.61-4.el6.x86_64
*可见已经安装了库文件,应该先卸载,不然会出现覆盖错误。注意卸:载时使用了--nodeps选项,忽略了依赖关系:
[[email protected] JavaEE]#rpm -e mysql-libs-5.1.61-4.el6.x86_64 --nodeps
b. 添加mysql组和mysql用户,用于设置mysql安装目录文件所有者和所属组。
[[email protected] JavaEE]#groupadd mysql
[[email protected] JavaEE]#useradd -r -g mysql mysql
*useradd -r参数表示mysql用户是系统用户,不可用于登录系统。
c. 将二进制文件解压到指定的安装目录,我们这里指定为/usr/local
[[email protected] ~]# cd/usr/local/
[[email protected] local]#tar zxvf /path/to/mysql-5.5.29-linux2.6-x86_64.tar.gz
*加压后在/usr/local/生成了解压后的文件夹mysql-5.5.29-linux2.6-x86_64,这名字太长,我们为它建立一个符号链接mysql,方便输入。
[[email protected] local]#ln -s mysql-5.5.29-linux2.6-x86_64 mysql
d. /usr/local/mysql/下的目录结构
Directory |
Contents of Directory |
bin |
Client programs and the mysqld server |
data |
Log files, databases |
docs |
Manual in Info format |
man |
Unix manual pages |
include |
Include (header) files |
lib |
Libraries |
scripts |
mysql_install_db |
share |
Miscellaneous support files, including error messages, sample configuration files, SQL for database installation |
sql-bench |
Benchmarks |
e. 进入mysql文件夹,也就是mysql所在的目录,并更改所属的组和用户。
[[email protected] local]#cd mysql
[[email protected] mysql]#chown -R mysql .
[[email protected] mysql]#chgrp -R mysql .
f. 执行mysql_install_db脚本,对mysql中的data目录进行初始化并创建一些系统表格。注意mysql服务进程mysqld运行时会访问data目录,所以必须由启动mysqld进程的用户(就是我们之前设置的mysql用户)执行这个脚本,或者用root执行,但是加上参数--user=mysql。
[[email protected] mysql]scripts/mysql_install_db --user=mysql
*如果mysql的安装目录(解压目录)不是/usr/local/mysql,那么还必须指定目录参数,如
[[email protected] mysql]scripts/mysql_install_db --user=mysql \
--basedir=/opt/mysql/mysql \
--datadir=/opt/mysql/mysql/data
*将mysql/目录下除了data/目录的所有文件,改回root用户所有,mysql用户只需作为mysql/data/目录下所有文件的所有者。
[[email protected] mysql]chown -R root .
[[email protected] mysql]chown -R mysql data
g. 复制配置文件
[[email protected] mysql] cp support-files/my-medium.cnf /etc/my.cnf
h. 将mysqld服务加入开机自启动项。
*首先需要将scripts/mysql.server服务脚本复制到/etc/init.d/,并重命名为mysqld。
[[email protected]] cp support-files/mysql.server /etc/init.d/mysqld
*通过chkconfig命令将mysqld服务加入到自启动服务项中。
[[email protected] mysql]#chkconfig --add mysqld
*注意服务名称mysqld就是我们将mysql.server复制到/etc/init.d/时重命名的名称。
*查看是否添加成功
[[email protected] mysql]#chkconfig --list mysqld
mysqld 0:off 1:off 2:on 3:on 4:on 5:on 6:off
i. 重启系统,mysqld就会自动启动了。
*检查是否启动
[[email protected] mysql]#netstat -anp|grep mysqld
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 2365/mysqld
unix 2 [ ACC ] STREAM LISTENING 14396 2365/mysqld /tmp/mysql.sock
*如果不想重新启动,那可以直接手动启动。
[[email protected] mysql]#service mysqld start
Starting MySQL.. SUCCESS!
j. 运行客户端程序mysql,在mysql/bin目录中,测试能否连接到mysqld。
[[email protected] mysql]#/usr/local/mysql/bin/mysql
Welcome to the MySQLmonitor. Commands end with ; or \g.
Your MySQL connection idis 2
Server version:5.5.29-log MySQL Community Server (GPL)
Copyright (c) 2000, 2012,Oracle and/or its affiliates. All rights reserved.
oracle is a registeredtrademark of Oracle Corporation and/or its affiliates. Other names may betrademarks of their respective owners.
Type 'help;' or '\h' forhelp. Type '\c' to clear the current input statement.
mysql> quit
Bye
*此时会出现mysql>命令提示符,可以输入sql语句,输入quit或exit退出。为了避免每次都输入mysql的全路径/usr/local/mysql/bin/mysql,可将其加入环境变量中,在/etc/profile最后加入两行命令:
MYSQL_HOME=/usr/local/mysql
export PATH=$PATH:$MYSQL_HOME/bin
这样就可以在shell中直接输入mysql命令来启动客户端程序了
[[email protected] mysql]#mysql
Welcome to the MySQLmonitor. Commands end with ; or \g.
Your MySQL connection idis 3
Server version:5.5.29-log MySQL Community Server (GPL)
Copyright (c) 2000, 2012,Oracle and/or its affiliates. All rights reserved.
Oracle is a registeredtrademark of Oracle Corporation and/or its
affiliates. Other namesmay be trademarks of their respective
owners.
Type 'help;' or '\h' forhelp. Type '\c' to clear the current input statement.
mysql>
5. RPM安装步骤
a. 检查是否已安装,grep的-i选项表示匹配时忽略大小写
[[email protected] JavaEE]#rpm -qa|grep -i mysql
mysql-libs-5.1.61-4.el6.x86_64
可见已经安装了库文件,应该先卸载,不然会出现覆盖错误。注意卸载时使用了--nodeps选项,忽略了依赖关系:
[[email protected] JavaEE]#rpm -e mysql-libs-5.1.61-4.el6.x86_64 --nodeps
2. 安装MySQL的服务器端软件,注意切换到root用户:
[[email protected] JavaEE]#rpm -ivh MySQL-server-5.5.29-2.el6.x86_64.rpm
安装完成后,安装进程会在Linux中添加一个mysql组,以及属于mysql组的用户mysql。可通过id命令查看:
[[email protected] JavaEE]#id mysql
uid=496(mysql)gid=493(mysql) groups=493(mysql)
MySQL服务器安装之后虽然配置了相关文件,但并没有自动启动mysqld服务,需自行启动:
[[email protected] JavaEE]#service mysql start
Starting MySQL.. SUCCESS!
可通过检查端口是否开启来查看MySQL是否正常启动:
[[email protected] JavaEE]#netstat -anp|grep 3306
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 34693/mysqld
c. 安装MySQL的客户端软件:
[[email protected] JavaEE]#rpm -ivh MySQL-client-5.5.29-2.el6.x86_64.rpm
如果安装成功应该可以运行mysql命令,注意必须是mysqld服务以及开启:
[[email protected] JavaEE]#mysql
Welcome to the MySQLmonitor. Commands end with ; or \g.
Your MySQL connection idis 1
Server version: 5.5.29MySQL Community Server (GPL)
Copyright (c) 2000, 2012,Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademarkof Oracle Corporation and/or its affiliates. Other names may be trademarks oftheir respective owners.
Type 'help;' or '\h' forhelp. Type '\c' to clear the current input statement.
mysql>
d. RPM安装方式文件分布
Directory |
Contents of Directory |
/usr/bin |
Client programs and scripts |
/usr/sbin |
The mysqld server |
/var/lib/mysql |
Log files, databases |
/usr/share/info |
Manual in Info format |
/usr/share/man |
Unix manual pages |
/usr/include/mysql |
Include (header) files |
/usr/lib/mysql |
Libraries |
/usr/share/mysql |
Miscellaneous support files, including error messages, character set files, sample configuration files, SQL for database installation |
/usr/share/sql-bench |
Benchmarks |
Redis集群部署文档(centos6系统)
(要让集群正常工作至少需要3个主节点,在这里我们要创建6个redis节点,其中三个为主节点,三个为从节点,对应的redis节点的ip和端口对应关系如下)
127.0.0.1:7000
127.0.0.1:7001
127.0.0.1:7002
127.0.0.1:7003
127.0.0.1:7004
127.0.0.1:7005
1:下载redis。官网下载3.0.0版本,之前2.几的版本不支持集群模式
下载地址:https://github.com/antirez/redis/archive/3.0.0-rc2.tar.gz
2:上传服务器,解压,编译
tar -zxvf redis-3.0.0-rc2.tar.gz mv redis-3.0.0-rc2.tar.gz redis3.0 cd /usr/local/redis3.0 make make install |
3:创建集群需要的目录
mkdir -p /usr.local/cluster cd /usr.local/cluster mkdir 7000 mkdir 7001 mkdir 7002 mkdir 7003 mkdir 7004 mkdir 7005 |
4:修改配置文件redis.conf
cp /usr/local/redis3.0/redis.conf /usr.local/cluster vi redis.conf ##修改配置文件中的下面选项 port 7000 daemonize yes cluster-enabled yes cluster-config-file nodes.conf cluster-node-timeout 5000 appendonly yes ##修改完redis.conf配置文件中的这些配置项之后把这个配置文件分别拷贝到7000/7001/7002/7003/7004/7005目录下面 cp /usr/local/cluster/redis.conf /usr/local/cluster/7000 cp /usr/local/cluster/redis.conf /usr/local/cluster/7001 cp /usr/local/cluster/redis.conf /usr/local/cluster/7002 cp /usr/local/cluster/redis.conf /usr/local/cluster/7003 cp /usr/local/cluster/redis.conf /usr/local/cluster/7004 cp /usr/local/cluster/redis.conf /usr/local/cluster/7005 ##注意:拷贝完成之后要修改7001/7002/7003/7004/7005目录下面redis.conf文件中的port参数,分别改为对应的文件夹的名称 |
5:分别启动这6个redis实例
cd /usr/local/cluster/7000 redis-server redis.conf cd /usr/local/cluster/7001 redis-server redis.conf cd /usr/local/cluster/7002 redis-server redis.conf cd /usr/local/cluster/7003 redis-server redis.conf cd /usr/local/cluster/7004 redis-server redis.conf cd /usr/local/cluster/7005 redis-server redis.conf ##启动之后使用命令查看redis的启动情况ps -ef|grep redis 如下图显示则说明启动成功 |
6:执行redis的创建集群命令创建集群
cd /usr/local/redis3.0/src ./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 |
6.1执行上面的命令的时候会报错,因为是执行的ruby的脚本,需要ruby的环境
错误内容:/usr/bin/env: ruby: No such file or directory
所以需要安装ruby的环境,这里推荐使用yum install ruby安装
yum install ruby |
6.2然后再执行第6步的创建集群命令,还会报错,提示缺少rubygems组件,使用yum安装
错误内容:
./redis-trib.rb:24:in `require': no such file to load -- rubygems (LoadError)
from ./redis-trib.rb:24
yum install rubygems |
6.3再次执行第6步的命令,还会报错,提示不能加载redis,是因为缺少redis和ruby的接口,使用gem 安装
错误内容:
/usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require': no such file to load -- redis (LoadError)
from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
from ./redis-trib.rb:25
gem install redis --version 3.0.0 |
注意:gem install redis --version 3.0.0 失败的话,需要修改一下gem的源
gem sources --remove https://rubygems.org/
gem sources -a https://ruby.taobao.org/
6.4 再次执行第6步的命令,正常执行
输入yes,然后配置完成。
至此redis集群即搭建成功!
7:使用redis-cli命令进入集群环境
redis-cli -c -p 7000 |