elasticsearch：EC2发现：主节点工作数据节点失败

问题描述：

我的目标是在EC2中的三个实例上运行6节点集群。我在每个实例上放置一个只有一个主节点和一个只有数据的节点（使用有弹性的剧本）。elasticsearch：EC2发现：主节点工作数据节点失败

来自三个实例中每一个实例的主节点都使用EC2发现找到彼此没有问题，并形成三个群集并选出一个主节点。来自相同实例的数据节点在启动时失败，并显示以下错误。

有什么我试图
- 交换数据节点通过主机名明确zen.unicast发现的工作
- 我可以从实例端口9301的telnet A-> B没有问题

参考：
Java版本 - OpenJDK的运行时环境（2.5.6的IcedTea）（7u79-2.5.6-0ubuntu1.14.04.1） ES版本 - 2.1.0

数据节点elasticseach.y毫升

bootstrap.mlockall: false 
cloud.aws.region: us-east 
cluster.name: my-cluster 
discovery.ec2.groups: stage-elasticsearch 
discovery.ec2.host_type: private_dns 
discovery.ec2.ping_timeout: 30s 
discovery.type: ec2 
discovery.zen.minimum_master_nodes: 2 
discovery.zen.ping.multicast.enabled: false 
gateway.expected_nodes: 4 
http.port: 9201 
network.host: _ec2:privateDns_ 
node.data: true 
node.master: false 
transport.tcp.port: 9301 
node.name: ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1

主节点elasticsearch.yml

bootstrap.mlockall: false 
cloud.aws.region: us-east 
cluster.name: my-cluster 
discovery.ec2.groups: stage-elasticsearch 
discovery.ec2.host_type: private_dns 
discovery.ec2.ping_timeout: 30s 
discovery.type: ec2 
discovery.zen.minimum_master_nodes: 2 
discovery.zen.ping.multicast.enabled: false 
gateway.expected_nodes: 4 
http.port: 9200 
network.host: _ec2:privateDns_ 
node.data: false 
node.master: true 
transport.tcp.port: 9300 
node.name: ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-master

错误从数据节点启动：

[2016-03-02 15:45:06,246][INFO ][node      ] [ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1] initializing ... 
[2016-03-02 15:45:06,679][INFO ][plugins     ] [ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1] loaded [cloud-aws], sites [head] 
[2016-03-02 15:45:06,710][INFO ][env      ] [ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [11.5gb], net total_space [14.6gb], spins? [no], types [ext4] 
[2016-03-02 15:45:09,597][INFO ][node      ] [ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1] initialized 
[2016-03-02 15:45:09,597][INFO ][node      ] [ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1] starting ... 
[2016-03-02 15:45:09,678][INFO ][transport    ] [ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1] publish_address {ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1/xxx-xxx-xx-xxx:9301}, bound_addresses {xxx-xxx-xx-xxx:9301} 
[2016-03-02 15:45:09,687][INFO ][discovery    ] [ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1] my-cluster/PNI6WAmzSYGgZcX2HsqenA 
[2016-03-02 15:45:09,701][WARN ][com.amazonaws.jmx.SdkMBeanRegistrySupport] 
java.security.AccessControlException: access denied ("javax.management.MBeanServerPermission" "findMBeanServer") 
    at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372) 
    at java.security.AccessController.checkPermission(AccessController.java:559) 
    at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) 
    at javax.management.MBeanServerFactory.checkPermission(MBeanServerFactory.java:413) 
    at javax.management.MBeanServerFactory.findMBeanServer(MBeanServerFactory.java:361) 
    at com.amazonaws.jmx.MBeans.getMBeanServer(MBeans.java:111) 
    at com.amazonaws.jmx.MBeans.registerMBean(MBeans.java:50) 
    at com.amazonaws.jmx.SdkMBeanRegistrySupport.registerMetricAdminMBean(SdkMBeanRegistrySupport.java:27) 
    at com.amazonaws.metrics.AwsSdkMetrics.registerMetricAdminMBean(AwsSdkMetrics.java:355) 
    at com.amazonaws.metrics.AwsSdkMetrics.<clinit>(AwsSdkMetrics.java:316) 
    at com.amazonaws.AmazonWebServiceClient.requestMetricCollector(AmazonWebServiceClient.java:563) 
    at com.amazonaws.AmazonWebServiceClient.isRMCEnabledAtClientOrSdkLevel(AmazonWebServiceClient.java:504) 
    at com.amazonaws.AmazonWebServiceClient.isRequestMetricsEnabled(AmazonWebServiceClient.java:496) 
    at com.amazonaws.AmazonWebServiceClient.createExecutionContext(AmazonWebServiceClient.java:457) 
    at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5924) 
    at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.fetchDynamicNodes(AwsEc2UnicastHostsProvider.java:118) 
    at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:230) 
    at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:215) 
    at org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:55) 
    at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.buildDynamicNodes(AwsEc2UnicastHostsProvider.java:104) 
    at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.sendPings(UnicastZenPing.java:335) 
    at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.ping(UnicastZenPing.java:240) 
    at org.elasticsearch.discovery.zen.ping.ZenPingService.ping(ZenPingService.java:106) 
    at org.elasticsearch.discovery.zen.ping.ZenPingService.pingAndWait(ZenPingService.java:84) 
    at org.elasticsearch.discovery.zen.ZenDiscovery.findMaster(ZenDiscovery.java:879) 
    at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335) 
    at org.elasticsearch.discovery.zen.ZenDiscovery.access$5000(ZenDiscovery.java:75) 
    at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1236) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
    at java.lang.Thread.run(Thread.java:745) 
[2016-03-02 15:45:09,703][WARN ][com.amazonaws.metrics.AwsSdkMetrics] 
java.security.AccessControlException: access denied ("javax.management.MBeanServerPermission" "findMBeanServer") 
    at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372) 
    at java.security.AccessController.checkPermission(AccessController.java:559) 
    at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) 
    at javax.management.MBeanServerFactory.checkPermission(MBeanServerFactory.java:413) 
    at javax.management.MBeanServerFactory.findMBeanServer(MBeanServerFactory.java:361) 
    at com.amazonaws.jmx.MBeans.getMBeanServer(MBeans.java:111) 
    at com.amazonaws.jmx.MBeans.isRegistered(MBeans.java:98) 
    at com.amazonaws.jmx.SdkMBeanRegistrySupport.isMBeanRegistered(SdkMBeanRegistrySupport.java:46) 
    at com.amazonaws.metrics.AwsSdkMetrics.registerMetricAdminMBean(AwsSdkMetrics.java:361) 
    at com.amazonaws.metrics.AwsSdkMetrics.<clinit>(AwsSdkMetrics.java:316) 
    at com.amazonaws.AmazonWebServiceClient.requestMetricCollector(AmazonWebServiceClient.java:563) 
    at com.amazonaws.AmazonWebServiceClient.isRMCEnabledAtClientOrSdkLevel(AmazonWebServiceClient.java:504) 
    at com.amazonaws.AmazonWebServiceClient.isRequestMetricsEnabled(AmazonWebServiceClient.java:496) 
    at com.amazonaws.AmazonWebServiceClient.createExecutionContext(AmazonWebServiceClient.java:457) 
    at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5924) 
    at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.fetchDynamicNodes(AwsEc2UnicastHostsProvider.java:118) 
    at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:230) 
    at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider$DiscoNodesCache.refresh(AwsEc2UnicastHostsProvider.java:215) 
    at org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:55) 
    at org.elasticsearch.discovery.ec2.AwsEc2UnicastHostsProvider.buildDynamicNodes(AwsEc2UnicastHostsProvider.java:104) 
    at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.sendPings(UnicastZenPing.java:335) 
    at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.ping(UnicastZenPing.java:240) 
    at org.elasticsearch.discovery.zen.ping.ZenPingService.ping(ZenPingService.java:106) 
    at org.elasticsearch.discovery.zen.ping.ZenPingService.pingAndWait(ZenPingService.java:84) 
    at org.elasticsearch.discovery.zen.ZenDiscovery.findMaster(ZenDiscovery.java:879) 
    at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335) 
    at org.elasticsearch.discovery.zen.ZenDiscovery.access$5000(ZenDiscovery.java:75) 
    at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1236) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
    at java.lang.Thread.run(Thread.java:745) 
[2016-03-02 15:45:39,688][WARN ][discovery    ] [ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1] waited for 30s and no initial state was set by the discovery 
[2016-03-02 15:45:39,698][INFO ][http      ] [ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1] publish_address {ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1/xxx-xxx-xx-xxx:9201}, bound_addresses {xxx-xxx-xx-xxx:9201} 
[2016-03-02 15:45:39,699][INFO ][node      ] [ip-xxx-xxx-xx-xxx.vpc.fakedomain.com-data1] started

答

我通过移除transport.tcp.port的明确设置固定的：9300和使用默认选择9300-9399范围内的任何端口。

来自AwsSdkMetrics的警告仍然存在，但不是Val所述的问题。

感谢您分享您的答案。 – Val

答

这是不实际的错误。

请参阅this issue这里已经报告过。看来插件太多了。

如果修改logging.yml配置文件作为这个这个问题的建议，那么你会被罚款：

# aws will try to do some sketchy JMX stuff, but its not needed. 
com.amazonaws.jmx.SdkMBeanRegistrySupport: ERROR 
com.amazonaws.metrics.AwsSdkMetrics: ERROR

不够公平，但似乎确实会影响ec2发现，因为除非我关闭ec2发现并明确设置zen.unicast.ping主机列表，否则这些数据节点似乎不能加入群集。 – mconlin

elasticsearch：EC2发现：主节点工作数据节点失败

相关推荐