在mesos上运行kafka发布消息的问题DCOS
我有一个运行DCOS的小型集群。我可以在this指南中成功安装kafka。运行在mesos上运行kafka发布消息的问题DCOS
$ dcos kafka connection
给
{
"address": [
"10.131.17.126:9475",
"10.131.24.6:9655",
"10.131.14.192:9181"
],
"zookeeper": "master.mesos:2181/dcos-service-kafka",
"dns": [
"broker-0.kafka.mesos:9475",
"broker-1.kafka.mesos:9655",
"broker-2.kafka.mesos:9181"
]
}
我可以创建主题,我已经研究饲养员用CLI工具和状态似乎不错
get /dcos-service-kafka/brokers/ids/0
{"jmx_port":-1,"timestamp":"1474206074029","endpoints":["PLAINTEXT://10.131.17.126:9475"],"host":"10.131.17.126","version":3,"port":9475}
get /dcos-service-kafka/brokers/ids/1
{"jmx_port":-1,"timestamp":"1474206120002","endpoints":["PLAINTEXT://10.131.24.6:9655"],"host":"10.131.24.6","version":3,"port":9655}
get /dcos-service-kafka/brokers/ids/2
{"jmx_port":-1,"timestamp":"1474206122985","endpoints":["PLAINTEXT://10.131.14.192:9181"],"host":"10.131.14.192","version":3,"port":9181}
然而,当我尝试发布
echo "Hello, World." | ./kafka-console-producer.sh --broker-list 10.131.17.126:9475, 10.131.24.6:9655, 10.131.14.192:9181 --topic topic1
我得到
[2016-09-18 18:49:32,909] ERROR Error when sending message to topic topic1 with key: null, value: 13 bytes with error: Failed to update metadata after 60000 ms. (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
我怀疑这可能是与私人与公共IP地址,也许host.name在server.properties。
任何人都可以提出一些建议,我可能会调试(并希望修复!)的问题,所以我可以成功地发布消息?
感谢
AJ
编辑:对于任何人都希望在未来。 This was a problem in /etc/hosts caused by a terraform script。
您的怀疑是正确的。这些是私人IP地址,不能从群集外寻址。为了与Kafka进行通信,您需要设置一个VPN,以使这些IP地址变得可访问,或者在集群中的某台计算机上运行发布命令。
另外,它看起来像是运行在1.8以前的DC/OS版本上。如果使用1.8,则无论动态分配的IP地址如何,您都可以使用更简单的代理端点。您可以使用命名的VIP broker.kafka.l4lb.thisdcos.directory:9092
,但是这只能从群集中的机器寻址。
将haproxy或nginx设置为指向命名VIP也是一种轻松外部访问在DC/OS集群上运行的服务(本例中为Kafka)的方法。您需要确保这些代理在公共代理上运行。 See here for more details。
这里是安装,生产和从默认安装卡夫卡耗费的例子:
~ $ dcos package install kafka
Installing Marathon app for package [kafka] version [1.1.11-0.10.0.0]
Installing CLI subcommand for package [kafka] version [1.1.11-0.10.0.0]
New command available: dcos kafka
DC/OS Kafka Service is being installed.
Documentation: https://docs.mesosphere.com/usage/services/kafka/
Issues: https://docs.mesosphere.com/support/
~ $ dcos kafka connection
{
"address": [
"10.0.3.64:9951",
"10.0.3.68:9795",
"10.0.3.66:9531"
],
"zookeeper": "master.mesos:2181/dcos-service-kafka",
"dns": [
"broker-0.kafka.mesos:9951",
"broker-1.kafka.mesos:9795",
"broker-2.kafka.mesos:9531"
],
"vip": "broker.kafka.l4lb.thisdcos.directory:9092"
}
~ $ dcos kafka topic create topic0
{
"message": "Output: Created topic \"topic0\".\n"
}
~ $ dcos node ssh --master-proxy --leader
[email protected] ~ $ wget http://download.nextag.com/apache/kafka/0.10.0.1/kafka_2.11-0.10.0.1.tgz
[email protected] ~ $ tar xf kafka_2.11-0.10.0.1.tgz
[email protected] ~ $ cd kafka_2.11-0.10.0.1
[email protected] ~/kafka_2.11-0.10.0.1 $ bin/kafka-console-producer.sh --broker-list broker.kafka.l4lb.thisdcos.directory:9092 --topic topic0
This is a message
This is another message
^[email protected] ~/kafka_2.11-0.10.0.1 $ bin/kafka-console-consumer.sh --zookeeper master.mesos:2181/dcos-service-kafka --topic topic0 --from-beginning
This is a message
This is another message
^CProcessed a total of 2 messages
$ bin/kafka-console-producer.sh --broker-list 10.0.3.64:9951,10.0.3.68:9795,10.0.3.66:9531 --topic topic0
foo
bar
baz
^[email protected] ~/kafka_2.11-0.10.0.1 $ bin/kafka-console-consumer.sh --zookeeper master.mesos:2181/dcos-service-kafka --topic topic0 --from-beginning
This is a message
This is another message
foo
bar
baz
^CProcessed a total of 5 messages
更新 - 这确实出现了已被丢失在/ etc/hosts中的条目引起的。我已经更新了我的terraform脚本以在安装过程中编写这些脚本,上面的示例现在按预期工作。
感谢您的帮助
太棒了!很高兴你达成了一个解决方案。对不起,我无法提供更多帮助。 – Gabriel
我已经更新到1.8了,谢谢。我按照用户指南'dcos node ssh --master-proxy --leader' ssh'ing到其中一个代理节点。我可以从那里和从码头集装箱内ping所有内部端口 –
应该添加它仍然不工作。不确定使用VPN会解决这个问题,如果我错了,请纠正我。 –
感谢您的帮助。不幸的是,这仍然不适合我。如果我遵循上面的步骤,我会得到: –