moosefs 解决 mfsmaster单点故障keepalived 辅助
一 服务器IP及应用角色
二 原理及思路
三 moosefs 安装 配置 管理
四 metadata.mfs.bak的定时备份
五 keepalived 安装 配置 管理
六 故障切换测试
七 参考资料联系方式
八 本教程相关资料下载
一 服务器IP及应用角色
192.168.20.237 keepalived MASTER mfsmaster mfschuckserver mfsclient
192.168.20.38 keepalived Slave mfsmetalogger mfschuckserver mfsclient mfsmaster(备份)
192.168.20.233 虚拟IP
二 原理及思路
1 mfsmaster的故障恢复在1.6.5版本后可以由mfsmetalogger产生的日志文件 changelog_ml.*.mfs和metadata.mfs.back由命令mfsmetarestore恢复
2 定时从mfsmaster 获取 metadata.mfs.back 文件用于master恢复
3 192.168.11.237 keepalived MASTER run检测到 mfsmaster 故障的时候 停止 keepalived
4 192.168.20.38 keepalived 状态从backup转到 master 并触发 notify_master事件执行恢复并启动mfsmaster脚本
5 整个切换在2~5秒内完成 根据检测时间间隔。
6 nagios 监控mfs状态可见参考资料
三 moosefs 安装 配置 管理
mfsmaster mfsmetalogger mfschuckserver mfsclient 四个角色的安装 如下 只是使用的配置文件不同
-
useradd -s /sbin/nologin mfs
-
mkdir -p /opt/software
-
cd /opt/software
-
wget http://sourceforge.net/projects/fuse/files/fuse-2.X/2.8.5/fuse-2.8.5.tar.gz
-
wget http://sourceforge.net/projects/moosefs/files/moosefs/1.6.20/mfs-1.6.20-2.tar.gz
-
-
tar zxvf fuse-2.8.5.tar.gz
-
cd fuse-2.8.5
-
./configure --prefix=/usr
-
make && make install
-
-
cd /opt/software
-
tar zxvf mfs-1.6.20-2.tar.gz
-
cd mfs-1.6.20-2
-
./configure --prefix=/usr/local/mfs --with-default-user=mfs --with-default-group=mfs
-
make && make install
-
cd ..
复制代码
MFS相关的配置文件有 mfsmaster.cfg mfsexports.cfg
#192.168.20.237和192.168.20.38 mfsmaster 配置文件如下
vi /usr/local/mfs/etc/mfsmaster.cfg
-
# WORKING_USER = mfs
复制代码
#192.168.20.237 192.168.20.38 mfschunkserver 使用相同的mfschunkserver.cfg mfshdd.cfg配置文件如下
vi /usr/local/mfs/etc/mfschunkserver.cfg
-
# WORKING_USER = mfs
-
# WORKING_GROUP = mfs
-
# SYSLOG_IDENT = mfschunkserver
-
# LOCK_MEMORY = 0
-
# NICE_LEVEL = -19
-
# DATA_PATH = /usr/local/mfs/var/mfs
-
# MASTER_RECONNECTION_DELAY = 5
-
# BIND_HOST = *
-
MASTER_HOST = 192.168.20.233
-
# MASTER_PORT = 9420
-
# MASTER_TIMEOUT = 60
-
# CSSERV_LISTEN_HOST = *
-
# CSSERV_LISTEN_PORT = 9422
-
# HDD_CONF_FILENAME = /usr/local/mfs/etc/mfshdd.cfg
-
# HDD_TEST_FREQ = 10
-
# deprecated, to be removed in MooseFS 1.7
-
# LOCK_FILE = /var/run/mfs/mfschunkserver.lock
-
# BACK_LOGS = 50
-
# CSSERV_TIMEOUT = 5
复制代码
我这边mfschunk存储数据的目录是 /data/9ai9.net
vim /usr/local/mfs/etc/mfshdd.cfg
-
/data/9ai9.net
复制代码
mfs各角色的管理命令如下 更详细的说明参见参考资料
/usr/local/mfs/sbin/mfsmaster start|stop|restart|reload
/usr/local/mfs/sbin/mfsmetalogger start|stop|restart|reload
/usr/local/mfs/sbin/mfschunkserver start|stop|restart|reload
客户端将mfschunk共享挂载到/home/lixy命令
modprobe fuse
/usr/local/mfs/bin/mfsmount /home/lixy -H 192.168.20.233
四 metadata.mfs.bak的定时备份
将主MFSMASTER的metadata.mfs.bak定时备份到 备用的MFSmaster服务器上
这里scp方法每5分钟从主服务器取metadata.mfs.bak备份一次 用于启动
在crontab -e里添加以下内容
-
*/5 * * * * scp 192.168.20.237:/usr/local/mfs/var/mfs/metadata.mfs.back /usr/local/mfs/var/mfs
复制代码
五 keepalived 安装 配置 管理
-
wget http://www.keepalived.org/software/keepalived-1.2.2.tar.gz
-
tar zxvf keepalived-1.2.2.tar.gz
-
cd keepalived-1.2.2
-
./configure --prefix=/
-
make
-
make install
复制代码
#配置说明
1 KEEPALIVED MASTER 每2秒检测一次 mfsmaster状态
2 KEEPAVLIED BACKUP 不检测MFS状态 只等待KEEPALIVED事情触发恢复
192.168.20.237 keepavled MASTER主配置文件
-
! Configuration File for keepalived
-
global_defs {
-
router_id 9ai9.net
-
}
-
vrrp_script check_run {
-
script "/root/keepalived_check_mfsmaster.sh"
-
interval 2
-
}
-
vrrp_sync_group VG1 {
-
group {
-
VI_1
-
}
-
}
-
vrrp_instance VI_1 {
-
state MASTER
-
interface eth0
-
virtual_router_id 88
-
priority 100
-
advert_int 1
-
nopreempt
-
authentication {
-
auth_type PASS
-
auth_pass 9ai9.net
-
}
-
track_script {
-
check_run
-
}
-
virtual_ipaddress {
-
192.168.20.233
-
}
-
}
复制代码
vim /root/keepalived_check_mfsmaster.sh
注:这里 MFSMASTER_HOST的IP地址要是192.168.20.237 非VIP
-
#!/bin/bash
-
#write by qhappy QQ 87179505
-
#20110310
-
-
MFSMASTER_HOST=192.168.20.237
-
MFSMASTER_PORT=9420
-
if [ `uname -i` = x86_64 ];then
-
CHECK_MASTER=/root/qhappy_check_tcp_64
-
else
-
CHECK_MASTER=/root/qhappy_check_tcp_32
-
fi
-
CHECK_TIME=2
-
#mfsmaster is working MFS_OK is 1 , mfsmaster down MFS_OK is 0
-
MFS_OK=1
-
function check_mfsmaster (){
-
$CHECK_MASTER -H $MFSMASTER_HOST -p $MFSMASTER_PORT >/dev/null 2>&1
-
if [ $? = 0 ] ;then
-
MFS_OK=1
-
else
-
MFS_OK=0
-
fi
-
return $MFS_OK
-
}
-
while [ $CHECK_TIME -ne 0 ]
-
do
-
let "CHECK_TIME -= 1"
-
check_mfsmaster
-
if [ $MFS_OK = 1 ] ; then
-
CHECK_TIME=0
-
exit 0
-
fi
-
if [ $MFS_OK -eq 0 ] && [ $CHECK_TIME -eq 0 ]
-
then
-
/etc/init.d/keepalived stop
-
exit 1
-
fi
-
done
复制代码
chmod 755 /root/keepalived_check_mfsmaster.sh
192.168.20.38 keepalived BACKUP的 配置
vim /etc/keepalived/keepalived.conf
-
! Configuration File for keepalived
-
global_defs {
-
router_id 9ai9.net
-
}
-
-
vrrp_sync_group VG1 {
-
group {
-
VI_1
-
}
-
notify_master "/root/keepalived_notify.sh master"
-
notify_backup "/root/keepalived_notify.sh backup"
-
}
-
vrrp_instance VI_1 {
-
state BACKUP
-
interface eth0
-
virtual_router_id 88
-
priority 80
-
advert_int 1
-
authentication {
-
auth_type PASS
-
auth_pass 9ai9.net
-
}
-
-
virtual_ipaddress {
-
192.168.20.233
-
}
-
}
复制代码
vim /root/keepalived_notify.sh
-
#!/bin/bash
-
#write by qhappy QQ 87179505
-
#20110310
-
-
MFS_HOME=/usr/local/mfs
-
MFSMARSTER=${MFS_HOME}/sbin/mfsmaster
-
MFSMETARESTORE=${MFS_HOME}/sbin/mfsmetarestore
-
MFS_DATA_PATH=${MFS_HOME}/var/mfs
-
-
function backup2master(){
-
$MFSMETARESTORE -m ${MFS_DATA_PATH}/metadata.mfs.bak -o ${MFS_DATA_PATH}/metadata.mfs $MFS_DATA_PATH/changelog_ml*.mfs
-
#$MFSMETARESTORE -a
-
$MFSMARSTER start
-
}
-
-
function master2backup(){
-
$MFSMARSTER stop
-
}
-
-
function ERROR(){
-
echo "USAGE: keepalived_notify.sh master|backup "
-
}
-
-
case $1 in
-
master)
-
backup2master
-
;;
-
backup)
-
master2backup
-
;;
-
*)
-
ERROR
-
;;
-
esac
复制代码
chomd 755 /root/keepalived_notify.sh
各服务角色 启动顺序
1 mfsmaster
2 keepalived
3 mfsmetalogger
4 mfschunckserver
5 客户端挂载
关闭顺序和上面的相反
五 故障切换测试
192.168.20.237 执行
/usr/local/mfs/sbin/mfsmaster stop
192.168.20.237日志截图
192.168.20.38 日志截图
续上图
六 参考资料
moosefs官网 http://www.moosefs.org/
ChinaUnix http://bbs.chinaunix.net/thread-1644309-1-1.html
笔者 QQ 87179505
七 资料下载
下载的内容为keepalived 检测mfsmaster 状态的文件 脚本中定义的使用路径是 /root 也请放在这个路径下
本文转自ljianbing51CTO博客,原文链接:http://blog.51cto.com/ljianbing/1617279 ,如需转载请自行联系原作者