您的位置: 首页  >  技术问答  >  星火没有启动远程实例

星火没有启动远程实例

分类: 技术问答 • 2022-03-12 14:26:20

星火没有启动远程实例

问题描述：

我远程启动火花实例与：星火没有启动远程实例

./spark-ec2 --key-pair=octavianKey4 --identity-file=octavianKey4.pem --region=eu-west-1 --zone=eu-west-1c launch my-instance-name --resume

然后，我确信我可以在端口7077 TCP连接，我试图用连接到它：

spark-shell --master spark://my-instance-name.eu-west-1.compute.amazonaws.com:7077

当我这样做，我得到：

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 
Setting default log level to "WARN". 
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 
16/11/11 20:09:52 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
16/11/11 20:09:53 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master my-instance-name.eu-west-1.compute.amazonaws.com:7077 
org.apache.spark.SparkException: Exception thrown in awaitResult 
     at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77) 
     at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75) 
     at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) 
     at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) 
     at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) 
     at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167) 
     at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83) 
     at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:100) 
     at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:108) 
     at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1$$anon$1.run(StandaloneAppClient.scala:106) 
     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
     at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
     at java.lang.Thread.run(Thread.java:745) 
Caused by: java.io.IOException: Failed to connect to my-instance-name.eu-west-1.compute.amazonaws.com/52.210.171.38:7077 
     at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:228) 
     at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:179) 
     at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:197) 
     at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:191) 
     at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:187) 
     ... 4 more 
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: my-instance-name.eu-west-1.compute.amazonaws.com/52.210.171.38:7077 
     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 
     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 
     at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:257) 
     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:291) 
     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:628) 
     at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:552) 
     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:466) 
     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:438) 
     at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) 
     at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) 
     ... 1 more 
16/11/11 20:10:12 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master my-instance-name.eu-west-1.compute.amazonaws.com:7077 
org.apache.spark.SparkException: Exception thrown in awaitResult 
     at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77) 
     at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75) 
     at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) 
     at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) 
     at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) 
     at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167) 
     at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83) 
     at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:100) 
     at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:108) 
     at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1$$anon$1.run(StandaloneAppClient.scala:106) 
     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
     at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
     at java.lang.Thread.run(Thread.java:745) 
Caused by: java.io.IOException: Failed to connect to my-instance-name.eu-west-1.compute.amazonaws.com/52.210.171.38:7077 
     at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:228) 
     at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:179) 
     at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:197) 
     at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:191) 
     at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:187) 
     ... 4 more 
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: my-instance-name.eu-west-1.compute.amazonaws.com/52.210.171.38:7077 
     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 
     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 
     at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:257) 
     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:291) 
     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:628) 
     at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:552) 
     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:466) 
     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:438) 
     at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) 
     at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) 
     ... 1 more 
16/11/11 20:10:32 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master my-instance-name.eu-west-1.compute.amazonaws.com:7077 
org.apache.spark.SparkException: Exception thrown in awaitResult 
     at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77) 
     at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75) 
     at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) 
     at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) 
     at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) 
     at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167) 
     at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83) 
     at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:100) 
     at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:108) 
     at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1$$anon$1.run(StandaloneAppClient.scala:106) 
     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
     at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
     at java.lang.Thread.run(Thread.java:745) 
Caused by: java.io.IOException: Failed to connect to my-instance-name.eu-west-1.compute.amazonaws.com/52.210.171.38:7077 
     at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:228) 
     at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:179) 
     at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:197) 
     at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:191) 
     at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:187) 
     ... 4 more 
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: my-instance-name.eu-west-1.compute.amazonaws.com/52.210.171.38:7077 
     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 
     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 
     at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:257) 
     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:291) 
     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:628) 
     at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:552) 
     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:466) 
     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:438) 
     at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) 
     at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) 
     ... 1 more 
16/11/11 20:10:52 ERROR StandaloneSchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up. 
16/11/11 20:10:52 WARN StandaloneSchedulerBackend: Application ID is not initialized yet. 
16/11/11 20:10:52 WARN StandaloneAppClient$ClientEndpoint: Drop UnregisterApplication(null) because has not yet connected to master 
16/11/11 20:10:53 ERROR SparkContext: Error initializing SparkContext. 
java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem 
     at scala.Predef$.require(Predef.scala:224) 
     at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91) 
     at org.apache.spark.SparkContext.<init>(SparkContext.scala:528) 
     at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2309) 
     at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:843) 
     at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:835) 
     at scala.Option.getOrElse(Option.scala:121) 
     at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:835) 
     at org.apache.spark.repl.Main$.createSparkSession(Main.scala:101) 
     at $line3.$read$$iw$$iw.<init>(<console>:15) 
     at $line3.$read$$iw.<init>(<console>:42) 
     at $line3.$read.<init>(<console>:44) 
     at $line3.$read$.<init>(<console>:48) 
     at $line3.$read$.<clinit>(<console>) 
     at $line3.$eval$.$print$lzycompute(<console>:7) 
     at $line3.$eval$.$print(<console>:6) 
     at $line3.$eval.$print(<console>) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
     at java.lang.reflect.Method.invoke(Method.java:497) 
     at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786) 
     at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047) 
     at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638) 
     at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637) 
     at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31) 
     at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19) 
     at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637) 
     at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569) 
     at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565) 
     at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807) 
     at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681) 
     at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395) 
     at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:38) 
     at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37) 
     at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37) 
     at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214) 
     at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:37) 
     at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:105) 
     at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920) 
     at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909) 
     at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909) 
     at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97) 
     at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909) 
     at org.apache.spark.repl.Main$.doMain(Main.scala:68) 
     at org.apache.spark.repl.Main$.main(Main.scala:51) 
     at org.apache.spark.repl.Main.main(Main.scala) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
     at java.lang.reflect.Method.invoke(Method.java:497) 
     at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) 
     at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) 
     at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) 
     at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) 
     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 
java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem 
    at scala.Predef$.require(Predef.scala:224) 
    at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91) 
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:528) 
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2309) 
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:843) 
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:835) 
    at scala.Option.getOrElse(Option.scala:121) 
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:835) 
    at org.apache.spark.repl.Main$.createSparkSession(Main.scala:101) 
    ... 47 elided 
<console>:14: error: not found: value spark 
     import spark.implicits._ 
      ^
<console>:14: error: not found: value spark 
     import spark.sql 
      ^
Welcome to 
     ____    __ 
    /__/__ ___ _____/ /__ 
    _\ \/ _ \/ _ `/ __/ '_/ 
    /___/ .__/\_,_/_/ /_/\_\ version 2.1.0-SNAPSHOT 
     /_/ 

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_60) 
Type in expressions to have them evaluated. 
Type :help for more information.

所以，这似乎很矛盾：

1.引发了一些例外情况，说我无法连接。

2.尽管如此，火花外壳确实启动。

3.However，在shell我不能用正常火花变量和方法，如sc

那么，究竟是怎么回事？

答

1）尽量netstat -at | grep 7077和检查主人是同一主机上运行

2）去激发用户界面默认端口8080和检查给出的--master一样

+0

呃，我在云端不同的机器上运行aws，所以主机不会在同一主机上运行 – octavian

+0

检查防火墙是否连接火星主机 –

+0

不，不是这样。 – octavian

答

您没有提供任何证据证明主人实际上运行正常：最有可能的不是。

尝试从自己的机器上运行

telnet my-instance-name.eu-west-1.compute.amazonaws.com 7077

。你也可以看看该服务器上的火花日志 - 明白为什么master未能正常启动

答

检查远程服务器上的火花版本，以及本地服务器从您所连接。

此错误主要是由于版本不匹配造成的。

+0

如果你确定，那可能就是答案。虽然一些细节将不胜感激。 –

相关推荐