spark shuffle服务开启

在提交了一个spark任务后，发现卡在了这一步，看意思是不打算分配资源给这个任务了。看字面意思是RM已经收到了请求，只是不给它而已，就有两种可能一是资源不足，而是没联系上AM。
然后下面告诉我们压根就没发现nodemanager
spark shuffle服务开启

在启动Hadoop之后，发现nodemanager节点并未启动起来，于是对Hadoop进行初始化并启动，但发现启动Hadoop的时候节点正常，nodemanager节点依然会挂掉。
spark shuffle服务开启

2018-09-19 15:15:57,961 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2299)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:121)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:245)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:261)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:495)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:543)

Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2267)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2291)
... 10 more

Caused by: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found

at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2171)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2265)
... 11 more

这里显示是spark使用spark_shuffle时出现了问题，那好下面就是解决办法。

<name>yarn.nodemanager.aux-services</name>
<value>spark_shuffle,mapreduce_shuffle</value>

<name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
<value>org.apache.spark.network.yarn.YarnShuffleService</value>

然后把spark目录下的yarn/spark-2.2.2-yarn-shuffle.jar hadoop/share/hadoop/yarn/lib/

然后重启就可以了。。。

spark shuffle服务开启

相关推荐