storm1.0.0集群安装
1.storm集群组成
storm集群类似于Hadoop(1.x)集群,在Hadoop(1.x)集群中任务叫"MapReduce jobs"而在storm集群中任务叫"topologies","MapReduce jobs"和"topologies"的最大区别是"MapReduce jobs"执行后就结束了,但是"topologies"一直执行直到你强制关闭它。storm集群和Hadoop(1.x)集群对比如下:
Hadoop(1.x) | Storm | |
集主节点进程 | JobTracker | Nimbus |
工作节点进程 | TaskTracker | Supervisor |
应用名称 | Job | Topology |
API接口 | Mapper/Reducer | Spout/Bolt |
使用场景 | 离线数据分析处理 | 实时数据分析处理 |
storm集群中的节点分为如下三类:
- master nodes :主节点上运行的进程叫Nimbus,Nimbus主要负责分发客户端提交的代码到集群中,负责分配任务以及监控任务的执行。
- worker nodes :工作节点上运行的进程叫Supervisor,Supervisor监听分发给自己的任务代码,启动和关闭工作进程来执行Nimbus分发的任务,每个工作进程执行有很多个topology组成的任务,一个被执行的topology由许多分布于集群机器上的工作进程组成。
- zookeeper nodes: Nimbus和Supervisor节点之间所有的协调工作是通过Zookeeper集群来实现的。此外,Nimbus和Supervisor进程都是快速失败(fail-fast)和无状态(stateless)的;Storm集群所有的状态要么在Zookeeper集群中,要么存储在本地磁盘上。这意味着你可以用kill -9来杀死Nimbus和Supervisor进程,它们在重启后可以继续工作。这个设计使得Storm集群拥有不可思议的稳定性。
2.storm集群搭建
- 搭建Zookeeper集群
- 安装Storm依赖库
- 下载并解压Storm发布版本
- 修改storm.yaml配置文件
- 启动Storm各个后台进程
2.1 搭建Zookeeper集群
1) 下载并解压zookeeper3.4.6
- #下载zookeeper-3.4.6.tar.gz到/opt并解压解压
- cd /opt
- tar -zxvf zookeeper-3.4.6.tar.gz
2) 在集群中各个节点中配置/etc/hosts,内容如下:
- 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
- ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
- 192.168.202.131 master
- 192.168.202.132 slavery01
- 192.168.202.133 slavery02
3) 在集群中各个节点中创建zookeeper数据文件
- sudo rm -r /home/hadoop/zookeeper
- cd /home/hadoop
- mkdir zookeeper
4) 在hostname=master机器上配置zoo.cfg,将/opt/zookeeper-3.3.4/conf目录下面的 zoo_sample.cfg修改为zoo.cfg,配置文件内容如下所示:
- initLimit=10
- syncLimit=5
- dataDir=/home/hadoopmanage/zookeeper
- clientPort=2181
- server.1=master:2888:3888
- server.2=slavery01:2888:3888
- server.3=slavery02:2888:3888
5) 远程复制分发安装文件
- scp -r /opt/zookeeper-3.3.4 hadoop@slavery01:/opt/
- scp -r /opt/zookeeper-3.3.4 hadoop@slavery02:/opt/
6) 在集群中各个节点设置myid必须为数字
- #使用ssh是首先保证你集群中节点都做了免秘钥登录
- ssh master
- echo "1" > /home/hadoop/zookeeper/myid
- ssh slavery01
- echo "2" > /home/hadoop/zookeeper/myid
- ssh slavery02
- echo "3" > /home/hadoop/zookeeper/myid
7) 启动ZooKeeper集群
- cd /opt/zookeeper-3.4.6
- bin/zkServer.sh start
8) 查看单机ZooKeeper是leader还是follower
- cd /opt/zookeeper-3.4.6
- bin/zkServer.sh status
9) 停止ZooKeeper集群
- cd /opt/zookeeper-3.4.6
- bin/zkServer.sh stop
10) 用客户端查看zookeeper上数据
- cd /opt/zookeeper-3.4.6/
- bin/zkCli.sh -server master:2181,slavery01:2181,slavery02:2181
- [hadoop@master storm-1.0.0]$ cd /opt/zookeeper-3.4.6/
- [hadoop@master zookeeper-3.4.6]$ bin/zkCli.sh -server master:2181,slavery01:2181,slavery02:2181
- Connecting to master:2181,slavery01:2181,slavery02:2181
- 2016-05-02 16:39:29,880 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
- 2016-05-02 16:39:29,889 [myid:] - INFO [main:Environment@100] - Client environment:host.name=master
- 2016-05-02 16:39:29,889 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.7.0_65
- 2016-05-02 16:39:29,902 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
- 2016-05-02 16:39:29,903 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/opt/jdk1.7.0_65/jre
- 2016-05-02 16:39:29,903 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/opt/zookeeper-3.4.6/bin/../build/classes:/opt/zookeeper-3.4.6/bin/../build/lib/*.jar:/opt/zookeeper-3.4.6/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper-3.4.6/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper-3.4.6/bin/../lib/netty-3.7.0.Final.jar:/opt/zookeeper-3.4.6/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper-3.4.6/bin/../lib/jline-0.9.94.jar:/opt/zookeeper-3.4.6/bin/../zookeeper-3.4.6.jar:/opt/zookeeper-3.4.6/bin/../src/java/lib/*.jar:/opt/zookeeper-3.4.6/bin/../conf:.:/opt/jdk1.7.0_65/lib/dt.jar:/opt/jdk1.7.0_65/lib/tools.jar
- 2016-05-02 16:39:29,903 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/opt/hadoop-2.7.1/lib/native/:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
- 2016-05-02 16:39:29,904 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
- 2016-05-02 16:39:29,904 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
- 2016-05-02 16:39:29,904 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
- 2016-05-02 16:39:29,904 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
- 2016-05-02 16:39:29,905 [myid:] - INFO [main:Environment@100] - Client environment:os.version=2.6.32-358.el6.x86_64
- 2016-05-02 16:39:29,905 [myid:] - INFO [main:Environment@100] - Client environment:user.name=hadoop
- 2016-05-02 16:39:29,905 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/home/hadoop
- 2016-05-02 16:39:29,906 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/opt/zookeeper-3.4.6
- 2016-05-02 16:39:29,909 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=master:2181,slavery01:2181,slavery02:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@8afbefd
- Welcome to ZooKeeper!
- 2016-05-02 16:39:30,290 [myid:] - INFO [main-SendThread(master:2181):ClientCnxn$SendThread@975] - Opening socket connection to server master/192.168.202.131:2181. Will not attempt to authenticate using SASL (unknown error)
- 2016-05-02 16:39:30,350 [myid:] - INFO [main-SendThread(master:2181):ClientCnxn$SendThread@852] - Socket connection established to master/192.168.202.131:2181, initiating session
- JLine support is enabled
- 2016-05-02 16:39:31,469 [myid:] - INFO [main-SendThread(master:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server master/192.168.202.131:2181, sessionid = 0x154701cef030003, negotiated timeout = 30000
- WATCHER::
- WatchedEvent state:SyncConnected type:None path:null
- [zk: master:2181,slavery01:2181,slavery02:2181(CONNECTED) 0]
查看zookeeper数据根目/录和数据目录/storm下数据:
- [zk: master:2181,slavery01:2181,slavery02:2181(CONNECTED) 0] ls /
- [storm, hbase, zookeeper]
- [zk: master:2181,slavery01:2181,slavery02:2181(CONNECTED) 1] ls /storm
- [backpressure, workerbeats, nimbuses, supervisors, errors, logconfigs, storms, assignments, leader-lock, blobstore]
- [zk: master:2181,slavery01:2181,slavery02:2181(CONNECTED) 2]
2.2 安装Storm依赖库
1) jdk安装(官网要求是1.6或者以上版本,这里安装1.7)
- 1.卸载linux自带的jdk环境
- 1)首先用命令java -version 查看系统中原有的java版本
- 2)然后用用 rpm -qa | gcj 命令查看具体的信息
- 3)最后用 rpm -e --nodeps java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64卸载
- 2.安装jdk-7u65-linux-x64.gz
- 1)下载jdk-7u65-linux-x64.gz放置于/opt/java/jdk-7u65-linux-x64.gz
- 2)解压,输入命令tar -zxvf jdk-7u65-linux-x64.gz
- 3)编辑vi /etc/profile,在文件末尾追加如下内容
- export JAVA_HOME=/opt/java/jdk1.7.0_65
- export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
- export PATH=$PATH:$JAVA_HOME/bin
- 4)使配置生效,输入命令,source /etc/profile
- 5)输入命令java -version,检查JDK环境是否配置成功
- 2.2.2 Python安装
2) Python安装(官网要求2.6.6以上)
- #安装python2.7.10
- #1)在https://www.python.org/ftp/python/2.7.10/Python-2.7.10.tgz下载Python-2.7.10.tgz放置于/opt下
- #2)解压tar -xzf Python-2.7.10.tgz到/opt下
- cd /opt
- tar -xzf Python-2.7.10.tgz
- #3)编译安装Python
- cd /opt/Python-2.7.10
- ./configue,
- make
- make install
- #4)如果不安装或者python版本过老,在后面会报错如下:
- #No module named argparse
- #5)Python版本查看
- python -V
2.3 下载并解压Storm发布版本
1) 在hostname=master机器上下载Storm发行版本
- cd /opt
- wget http://www.apache.org/dyn/closer.lua/storm/apache-storm-1.0.0/apache-storm-1.0.0.tar.gz
2) 在hostname=master机器上解压到目录/opt下
- cd /opt
- tar -zxvf apache-storm-1.0.0.tar.gz
- mv apache-storm-1.0.0 storm-1.0.0
3) 在hostname=master机器上修改/opt/storm-1.0.0/conf/storm.yaml配置文件
- storm.zookeeper.servers:
- - "master"
- - "slavery01"
- - "slavery02"
- nimbus.seeds: ["master"]
- supervisor.slots.ports:
- - 6700
- - 6701
- - 6702
- - 6703
- storm.local.dir: "/home/hadoopmanage/storm/localdir/"
注意:上面配置参数前的空格符、TAB符不要去掉,否则会报如下错误信息:
- at org.apache.storm.shade.org.yaml.snakeyaml.scanner.ScannerImpl.stalePossibleSimpleKeys(ScannerImpl.java:460)
- at org.apache.storm.shade.org.yaml.snakeyaml.scanner.ScannerImpl.needMoreTokens(ScannerImpl.java:280)
- at org.apache.storm.shade.org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:225)
- at org.apache.storm.shade.org.yaml.snakeyaml.parser.ParserImpl$ParseIndentlessSequenceEntry.produce(ParserImpl.java:532)
- at org.apache.storm.shade.org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
- at org.apache.storm.shade.org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:143)
- at org.apache.storm.shade.org.yaml.snakeyaml.composer.Composer.composeSequenceNode(Composer.java:203)
- at org.apache.storm.shade.org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:157)
- at org.apache.storm.shade.org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:237)
- at org.apache.storm.shade.org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:159)
- at org.apache.storm.shade.org.yaml.snakeyaml.composer.Composer.composeDocument(Composer.java:122)
- at org.apache.storm.shade.org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:105)
- at org.apache.storm.shade.org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:120)
- at org.apache.storm.shade.org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:481)
- at org.apache.storm.shade.org.yaml.snakeyaml.Yaml.load(Yaml.java:424)
- at org.apache.storm.utils.Utils.findAndReadConfigFile(Utils.java:290)
- at org.apache.storm.utils.Utils.readStormConfig(Utils.java:391)
- at org.apache.storm.utils.Utils.<clinit>(Utils.java:119)
- ... 39 more
4) 在hostname=master机器上分发安装文件到其他节点
- cd /opt
- scp -r storm-1.0.0 hadoop@slavery01:/opt
- scp -r storm-1.0.0 hadoop@slavery02:/opt
5) 在各个节点上增加storm集群本地存储文件,该目录被Nimbus和Supervisor进程用于存储少量状态,如jars、confs等的本地磁盘目录,需要提前创建该目录并给以足够的访问权限。然后在storm.yaml中配置该目录
- mkdir -p /home/hadoopmanage/storm/localdir/
2.4 启动Storm各个后台进程
1) 在hostname=master节点启动Nimbus进程服务,放置于后台运行
- cd /opt/storm-1.0.0/
- bin/storm nimbus >/dev/null 2>&1 &
2) 在各个hostname-slavery0*节点启动Supervisor
- cd /opt/storm-1.0.0/
- bin/storm supervisor >/dev/null 2>&1 &
3)在hostname=master节点启动UI进程服务,放置于后台运行
- cd /opt/storm-1.0.0/
- bin/storm ui >/dev/null 2>&1 &
启动后打开浏览器,访问http://master:8080/index.html 或者http://192.168.202.131:8080/index.html,打开的storm ui界面如下:
3.向storm集群中提交任务
3.1 启动Storm Topology
- storm jar mycode.jar com.test.MyTopology arg1 arg2 arg3
其中mycode.jar是包含Topology实现代码的jar包,com.test.MyTopology的main方法是Topology的入口,arg1、arg2和arg3为org.me.MyTopology执行时需要传入的参数。
3.2 停止Storm Topology
- storm kill {toponame}
其中,{toponame}为Topology提交到Storm集群时指定的Topology任务名称。