Hadoop 高可用集群搭建(虚拟机)
-
准备5台centos7
ip: 192.168.136.128 s1 192.168.136.129 s2 192.168.136.130 s3 192.168.136.131 s4 192.168.136.131 s5
关闭防火墙:
使用命令service iptables stop
chkconfig iptables off (设置自动启动为关闭)
chkconfig --del iptables (移除开机自动启动)
server iptables status(查看状态)
-
集群规划:
ip
主机名
安装软件
运行进程
192.168.136.128
s1
Hadoop jdk
nameNode DFSZKFailoverController resourceManager
192.168.136.129
s2
Hadoop jdk
nameNode DFSZKFailoverController resourceManager
192.168.136.130
s3
Hadoop jdk zookeeper
DataNode、NodeManager、JournalNode、QuorumPeerMain
192.168.136.131
s4
Hadoop jdk zookeeper
DataNode、NodeManager、JournalNode、QuorumPeerMain
192.168.136.132
s5
Hadoop jdk zookeeper
DataNode、NodeManager、JournalNode、QuorumPeerMain
-
免密配置
1.分别在各个集群上执行:ssh-****** -t dsa -P '' -f ~/.ssh/id_dsa
2.将各个节点(机器)上的/root/.ssh/id_dsa.pub文件集中到一台机器上(在此将所有集群上公钥发送到s1上):
scp -r /root/.ssh/id_dsa.pub namenode01:/root/.ssh/id_dsa.pub.x
3.将所有公钥配置到authorized_keys中: cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
4.用ssh把各台机器访问一次!
5分配给其他机器.
scp -r /root/.ssh/authorized_keys /root/.ssh/known_hosts 其他机器ip:/root/.ssh/
-
配置jdk
1.将jdk8的tar文件上传至s1 的/opt/java 解压
2.配置/etc/profile 文件
在底部添加
export JAVA_HOME=/opt/java/jdk1.8.0_60
export CLASSPATH=$:CLASSPATH:$JAVA_HOME/lib/
export PATH=$PATH:$JAVA_HOME/bin
3.执行 resource /etc/profile
4.将配置好的jdk分发到各个节点
scp -r /opt/java/ s2:/opt/java/
scp -r /opt/java/ s3:/opt/java/
scp -r /opt/java/ s4:/opt/java/
scp -r /opt/java/ s5:/opt/java/
-
配置zookeeper
在s3 、s4、s5节点上配置
- 上传zookeeper 3.4
- 解压到 :/opt/下
- 修改文件名为zookeeper
- cd /opt/zookeeper/conf
- mv zoo_sample.cfg zoo.cfg
- vim zoo.cfg
-
- 创建文件夹: mkdir -p /data/zookeeper
-
创建文件夹: mkdir -p /data/zookeeper-datelog
进入:cd /data/zookeeper
创建文件:vim myid
添加内容:4 ,各个zookeeper节点的数字要不一样,和上面的server.x对应。
保存
- 分发zookeeper文件到各个节点上
-
scp -r /opt/zookeeper/ s4:/opt/
scp -r /opt/zookeeper/ s5:/opt/
- 配置环境变量:
- 在各个节点启动zookeeper
-
启动: zkServer.sh start
- 在各个节点查看zookeeper状态
-
查看状态: zkServer.sh status
一个leader 两个follower
vim /etc/profile
追加内容:
export ZK_HOME=/opt/zookeeper
export PATH=$PATH:$ZK_HOME/bin
保存
source /etc/profile
-
配置hadoop
-
上传至s1并解压
解压:tar -zxvf hadoop-2.8.0.tar.gz
重命名 mv hadoop-2.8.0/ hadoop/
进入:cd hadoop/
除杂:rm -rf NOTICE.txt README.txt LICENSE.txt
-
配置
-
vim core-site.xm
<configuration> <!-- 指定hdfs的nameservice为ns1 --> <property> <name>fs.defaultFS</name> <value>hdfs://ns1</value> </property> <!-- 指定hadoop临时目录 --> <property> <name>hadoop.tmp.dir</name> <value>/opt/hadoop/tmp</value> </property> <!-- 指定zookeeper地址 --> <property> <name>ha.zookeeper.quorum</name> <value>192.168.136.130:2181,192.168.136.131:2181,192.168.136.132:2181</value> </property> </configuration>
-
vim hdfs-site.xml
<configuration> <!--指定hdfs的nameservice为ns1,需要和core-site.xml中的保持一致 --> <property> <name>dfs.nameservices</name> <value>ns1</value> </property> <!-- ns1下面有两个NameNode,分别是nn1,nn2 --> <property> <name>dfs.ha.namenodes.ns1</name> <value>nn1,nn2</value> </property> <!-- nn1的RPC通信地址 --> <property> <name>dfs.namenode.rpc-address.ns1.nn1</name> <value>192.168.136.128:9000</value> </property> <!-- nn1的http通信地址 --> <property> <name>dfs.namenode.http-address.ns1.nn1</name> <value>192.168.136.128:50070</value> </property> <!-- nn2的RPC通信地址 --> <property> <name>dfs.namenode.rpc-address.ns1.nn2</name> <value>192.168.136.129:9000</value> </property> <!-- nn2的http通信地址 --> <property> <name>dfs.namenode.http-address.ns1.nn2</name> <value>192.168.136.129:50070</value> </property> <!-- 指定NameNode的元数据在JournalNode上的存放位置 --> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://192.168.136.130:8485;192.168.136.131:8485;192.168.136.132:8485/ns1</value> </property> <!-- 指定JournalNode在本地磁盘存放数据的位置 --> <property> <name>dfs.journalnode.edits.dir</name> <value>/hadoop/journal</value> </property> <!-- 开启NameNode失败自动切换 --> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property> <!-- 配置失败自动切换实现方式 --> <property> <name>dfs.client.failover.proxy.provider.ns1</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <!-- 配置隔离机制方法,多个机制用换行分割,即每个机制暂用一行--> <property> <name>dfs.ha.fencing.methods</name> <value> sshfence shell(/bin/true) </value> </property> <!-- 使用sshfence隔离机制时需要ssh免登陆 --> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>~/.ssh/id_rsa</value> </property> <!-- 配置sshfence隔离机制超时时间 --> <property> <name>dfs.ha.fencing.ssh.connect-timeout</name> <value>30000</value> </property> </configuration>
-
vim mapred-site.xml
<configuration> <!-- 指定mr框架为yarn方式 --> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>
-
vim yarn-site.xml
<configuration> <!-- Site specific YARN configuration properties --> <!-- 开启RM高可靠 --> <property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property> <!-- 指定RM的cluster id --> <property> <name>yarn.resourcemanager.cluster-id</name> <value>yrc</value> </property> <!-- 指定RM的名字 --> <property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>rm1,rm2</value> </property> <!-- 分别指定RM的地址 --> <property> <name>yarn.resourcemanager.hostname.rm1</name> <value>192.168.136.128</value> </property> <property> <name>yarn.resourcemanager.hostname.rm2</name> <value>192.168.136.129</value> </property> <!-- 指定zk集群地址 --> <property> <name>yarn.resourcemanager.zk-address</name> <value>192.168.138.130:2181,192.168.136.131:2181,192.168.136.132:2181</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <description>The address of the applications manager interface in the RM.</description> <name>yarn.resourcemanager.address</name> <value>${yarn.resourcemanager.hostname.rm1}:8032</value> </property> <property> <description>The address of the scheduler interface.</description> <name>yarn.resourcemanager.scheduler.address</name> <value>${yarn.resourcemanager.hostname.rm1}:8030</value> </property> <property> <description>The http address of the RM web application.</description> <name>yarn.resourcemanager.webapp.address</name> <value>${yarn.resourcemanager.hostname.rm1}:8088</value> </property> <property> <description>The https adddress of the RM web application.</description> <name>yarn.resourcemanager.webapp.https.address</name> <value>${yarn.resourcemanager.hostname.rm1}:8090</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>${yarn.resourcemanager.hostname.rm1}:8031</value> </property> <property> <description>The address of the RM admin interface.</description> <name>yarn.resourcemanager.admin.address</name> <value>${yarn.resourcemanager.hostname.rm1}:8033</value> </property> </configuration>
-
vim hadoop-env.sh
- 修改slaves(slaves是指定子节点的位置,因为要在192.168.136.128启动hdfs(slaves文件指定的是datanode的位置) 启动yarn(slaves文件指定的是nodemanager位置))
-
将配置好的hadoop分配给其余节点
scp -r /opt/hadoop/ s2:/opt/
scp -r /opt/hadoop/ s3:/opt/
scp -r /opt/hadoop/ s4:/opt/
scp -r /opt/hadoop/ s5:/opt/
-
配置环境变量并分发
vim /etc/profile
追加:
HADOOP_HOME=/opt/hadoop
PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
保存
source /etc/profile
-
-
-
启动hadoop集群
保证 s3、s4、s5节点上的zookeep启动中 #查看状态:一个leader,两个follower
并且 启动 journalnode 命令:./hadoop-daemon.sh start journalnode
#运行jps命令检验,s3、s4、s5上多了JournalNode进程
格式化HDFS #在s1上执行命令
hdfs namenode -format
#格式化后会在根据core-site.xml中的hadoop.tmp.dir配置生成个文件,这里我配置的是/opt/hadoop/tmp,然后将/opt/hadoop/tmp拷贝到s2的/hadoop/下。
scp -r tmp/ s2: /opt/hadoop/
格式化ZK s1节点上
hdfs zkfc -formatZK
启动HDFS s1节点上
start-dfs.sh
启动YARN s1节点上 start-yarn.sh
#####注意#####:,因为namenode和resourcemanager他们都要占用大量资源,所以有条件时把他们分开,就要分别在不同的机器上启动
9.测试
1、可以到浏览器访问:
http://192.168.136.128:50070
NameNode s1:9000' (active)
http://192.168. 36.129:50070
NameNode s2:9000' (standby)
2、验证HDFS HA
首先向hdfs上传一个文件
hadoop fs -put /etc/profile /profile
hadoop fs -ls /
然后再kill掉active的NameNode(192.168.136.128)
kill -9 <pid of NN>
通过浏览器访问:http://192.168.136.129:50070
NameNode 'linux02:9000' (active)
这个时候s2上的NameNode变成了active
在执行命令:
hadoop fs -ls /
刚才上传的文件依然存在!!!
手动启动那个挂掉的NameNode
hadoop-daemon.sh start namenode
通过浏览器访问:http://192.168.136.128:50070
NameNode s1:9000' (standby)
验证YARN:
运行一下hadoop提供的demo中的WordCount程序:
Hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.0.jar wordcount /profile /out
完成任务则YARN运行无误
OK,大功告成!!!