在风暴0.10.0 2工作进程启动,即使当我设置工人= 1,和UI报告工人= 1

问题描述:

我有一个风暴拓扑我为此:setNumWorkers(1);在风暴0.10.0 2工作进程启动,即使当我设置工人= 1,和UI报告工人= 1

当我查看关于此运行拓扑的风暴UI报告时,我看到Num工作者设置为“1”。

但是,当我登录到运行supervisor的节点时,我看到两个进程具有相同的-Dworker.id和-Dworker.port设置。
我包括'ps'的输出显示了我在下面的这两个过程。我的问题是:为什么有两个进程似乎配置为工作进程,如果我只请求一个(注意:Storm UI确认我只有一个工人)。

这对我很重要,因为当我做任何分析或分析哪些资源正在被我的拓扑消耗,我想知道哪个过程要归零。

ps的输出

root  787 20.0 0.6 5858228 78388 ?  Sl 05:04 0:00 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp /opt/apache-storm-0.10.0/lib/log4j-slf4j-impl-2.1.jar:/opt/apache-storm-0.10.0/lib/servlet-api-2.5.jar:/opt/apache-storm-0.10.0/lib/clojure-1.6.0.jar:/opt/apache-storm-0.10.0/lib/slf4j-api-1.7.7.jar:/opt/apache-storm-0.10.0/lib/hadoop-auth-2.4.0.jar:/opt/apache-storm-0.10.0/lib/log4j-api-2.1.jar:/opt/apache-storm-0.10.0/lib/disruptor-2.10.4.jar:/opt/apache-storm-0.10.0/lib/storm-core-0.10.0.jar:/opt/apache-storm-0.10.0/lib/log4j-over-slf4j-1.6.6.jar:/opt/apache-storm-0.10.0/lib/log4j-core-2.1.jar:/opt/apache-storm-0.10.0/lib/asm-4.0.jar:/opt/apache-storm-0.10.0/lib/kryo-2.21.jar:/opt/apache-storm-0.10.0/lib/reflectasm-1.07-shaded.jar:/opt/apache-storm-0.10.0/lib/minlog-1.2.jar:/opt/apache-storm-0.10.0/conf:/opt/apache-storm-0.10.0/storm-local/supervisor/stormdist/big-storm-job-1-1487739502/stormjar.jar -Dlogfile.name=big-storm-job-1-1487739502-worker-6700.log -Dstorm.home=/opt/apache-storm-0.10.0 -Dstorm.id=big-storm-job-1-1487739502 -Dworker.id=e8e03e95-1fcc-492a-b5e4-51ef7b8db2ee -Dworker.port=6700 -Dstorm.log.dir=/opt/apache-storm-0.10.0/logs -Dlog4j.configurationFile=/opt/apache-storm-0.10.0/log4j2/worker.xml backtype.storm.LogWriter /usr/lib/jvm/java-8-openjdk-amd64/bin/java -server -Xmx768m -Djava.library.path=/opt/apache-storm-0.10.0/storm-local/supervisor/stormdist/big-storm-job-1-1487739502/resources/Linux-amd64:/opt/apache-storm-0.10.0/storm-local/supervisor/stormdist/big-storm-job-1-1487739502/resources:/usr/local/lib:/opt/local/lib:/usr/lib -Dlogfile.name=big-storm-job-1-1487739502-worker-6700.log -Dstorm.home=/opt/apache-storm-0.10.0 -Dstorm.conf.file= -Dstorm.options= -Dstorm.log.dir=/opt/apache-storm-0.10.0/logs -Dlogging.sensitivity=S3 -Dlog4j.configurationFile=/opt/apache-storm-0.10.0/log4j2/worker.xml -Dstorm.id=big-storm-job-1-1487739502 -Dworker.id=e8e03e95-1fcc-492a-b5e4-51ef7b8db2ee -Dworker.port=6700 -cp /opt/apache-storm-0.10.0/lib/log4j-slf4j-impl-2.1.jar:/opt/apache-storm-0.10.0/lib/servlet-api-2.5.jar:/opt/apache-storm-0.10.0/lib/clojure-1.6.0.jar:/opt/apache-storm-0.10.0/lib/slf4j-api-1.7.7.jar:/opt/apache-storm-0.10.0/lib/hadoop-auth-2.4.0.jar:/opt/apache-storm-0.10.0/lib/log4j-api-2.1.jar:/opt/apache-storm-0.10.0/lib/disruptor-2.10.4.jar:/opt/apache-storm-0.10.0/lib/storm-core-0.10.0.jar:/opt/apache-storm-0.10.0/lib/log4j-over-slf4j-1.6.6.jar:/opt/apache-storm-0.10.0/lib/log4j-core-2.1.jar:/opt/apache-storm-0.10.0/lib/asm-4.0.jar:/opt/apache-storm-0.10.0/lib/kryo-2.21.jar:/opt/apache-storm-0.10.0/lib/reflectasm-1.07-shaded.jar:/opt/apache-storm-0.10.0/lib/minlog-1.2.jar:/opt/apache-storm-0.10.0/conf:/opt/apache-storm-0.10.0/storm-local/supervisor/stormdist/big-storm-job-1-1487739502/stormjar.jar backtype.storm.daemon.worker big-storm-job-1-1487739502 8fde2226-4b32-406d-8809-81ed88e5ae1f 6700 e8e03e95-1fcc-492a-b5e4-51ef7b8db2ee 

root  805 203 2.0 4308648 255336 ?  Sl 05:04 0:06 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -server -Xmx768m -Djava.library.path=/opt/apache-storm-0.10.0/storm-local/supervisor/stormdist/big-storm-job-1-1487739502/resources/Linux-amd64:/opt/apache-storm-0.10.0/storm-local/supervisor/stormdist/big-storm-job-1-1487739502/resources:/usr/local/lib:/opt/local/lib:/usr/lib -Dlogfile.name=big-storm-job-1-1487739502-worker-6700.log -Dstorm.home=/opt/apache-storm-0.10.0 -Dstorm.conf.file= -Dstorm.options= -Dstorm.log.dir=/opt/apache-storm-0.10.0/logs -Dlogging.sensitivity=S3 -Dlog4j.configurationFile=/opt/apache-storm-0.10.0/log4j2/worker.xml -Dstorm.id=big-storm-job-1-1487739502 -Dworker.id=e8e03e95-1fcc-492a-b5e4-51ef7b8db2ee -Dworker.port=6700 -cp /opt/apache-storm-0.10.0/lib/log4j-slf4j-impl-2.1.jar:/opt/apache-storm-0.10.0/lib/servlet-api-2.5.jar:/opt/apache-storm-0.10.0/lib/clojure-1.6.0.jar:/opt/apache-storm-0.10.0/lib/slf4j-api-1.7.7.jar:/opt/apache-storm-0.10.0/lib/hadoop-auth-2.4.0.jar:/opt/apache-storm-0.10.0/lib/log4j-api-2.1.jar:/opt/apache-storm-0.10.0/lib/disruptor-2.10.4.jar:/opt/apache-storm-0.10.0/lib/storm-core-0.10.0.jar:/opt/apache-storm-0.10.0/lib/log4j-over-slf4j-1.6.6.jar:/opt/apache-storm-0.10.0/lib/log4j-core-2.1.jar:/opt/apache-storm-0.10.0/lib/asm-4.0.jar:/opt/apache-storm-0.10.0/lib/kryo-2.21.jar:/opt/apache-storm-0.10.0/lib/reflectasm-1.07-shaded.jar:/opt/apache-storm-0.10.0/lib/minlog-1.2.jar:/opt/apache-storm-0.10.0/conf:/opt/apache-storm-0.10.0/storm-local/supervisor/stormdist/big-storm-job-1-1487739502/stormjar.jar backtype.storm.daemon.worker big-storm-job-1-1487739502 8fde2226-4b32-406d-8809-81ed88e5ae1f 6700 e8e03e95-1fcc-492a-b5e4-51ef7b8db2ee 

以防万一它是有人读到此试图让我的环境中更好的画面有所帮助,这里是风暴(和其他的东西)我泊坞窗的配置。希望有帮助。

version: '2' 
services: 
    zookeeper: 
    image: wurstmeister/zookeeper 
    container_name: zk 
    hostname: zk 
    ports: 
     - "2181:2181" 
    networks: 
     storm: 
    kafka: 
    image: wurstmeister/kafka:0.8.2.2-1 
    container_name: kafka 
    hostname: kafka 
    ports: 
     - "9092:9092" 
    environment: 
     KAFKA_BROKER_ID: 1 
     KAFKA_ADVERTISED_HOST_NAME: 10.211.55.4 
     KAFKA_ZOOKEEPER_CONNECT: 10.211.55.4 
    volumes: 
     - /var/run/docker.sock:/var/run/docker.sock 
    nimbus: 
    image: sunside/storm-nimbus 
    container_name: storm-nimbus 
    hostname: storm-nimbus 
    ports: 
     - "49773:49772" 
     - "49772:49773" 
     - "49627:49627" 
    environment: 
     - "LOCAL_HOSTNAME=nimbus" 
     - "ZOOKEEPER_ADDRESS=zk" 
     - "ZOOKEEPER_PORT=2181" 
     - "NIMBUS_ADDRESS=nimbus" 
     - "NIMBUS_THRIFT_PORT=49627" 
     - "DRPC_PORT=49772" 
     - "DRPCI_PORT=49773" 
    volumes: 
     - /media/psf/Home/dev/storm-pipeline:/pipeline 
    networks: 
     storm: 
    supervisor: 
    image: sunside/storm-supervisor 
    container_name: storm-supervisor 
    hostname: storm-supervisor 
    ports: 
     - "8000:8000" 
    environment: 
     - "LOCAL_HOSTNAME=supervisor" 
     - "NIMBUS_ADDRESS=nimbus" 
     - "NIMBUS_THRIFT_PORT=49627" 
     - "DRPC_PORT=49772" 
     - "DRPCI_PORT=49773" 
     - "ZOOKEEPER_ADDRESS=zk" 
     - "ZOOKEEPER_PORT=2181" 
    networks: 
     storm: 
    ui: 
    image: sunside/storm-ui 
    container_name: storm-ui 
    hostname: storm-ui 
    ports: 
     - "8888:8080" 
    environment: 
     - "LOCAL_HOSTNAME=ui" 
     - "NIMBUS_ADDRESS=nimbus" 
     - "NIMBUS_THRIFT_PORT=49627" 
     - "DRPC_PORT=49772" 
     - "DPRCI_PORT=49773" 
     - "ZOOKEEPER_ADDRESS=zk" 
     - "ZOOKEEPER_PORT=2181" 
    networks: 
     storm: 
    elasticsearch: 
    image: elasticsearch:2.3 
    container_name: elasticsearch 
    hostname: elasticsearch 
    ports: 
     - "9200:9200" 
    networks: 
     storm: 
networks: 
    storm: 
    external: true 

答案的谜团是,这两个过程中的一个是真正的“工人”进程(即正在执行的类backtype.storm.daemon.worker)......其他进程在响应'ps'命令时输出的是日志写入器进程,由类backtype.storm.LogWriter执行。

我应该注意到这两个过程的输出线。哦,现在我们知道了!

+0

感谢来自Horton的Deepna Bains支持发现所期望的与我们所得到的不一致的原因。 –