bug调试(spark部分)
在spark集群启动起来,编写的Scala语言:
sc.textFile("hdfs://node01:8080/spark/data/words").flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).collect()
执行的时候报错:
18/07/24 12:55:50 INFO spark.SparkContext: Created broadcast 0 from textFile at <console>:28
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ i
s not supported in state standby at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.jav
a:1727) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1352)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:4174)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:8
81) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(Cl
ientNamenodeProtocolServerSideTranslatorPB.java:821) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.c
allBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java
:619)
原因:
我的node01的状态是standby,而Scala语言要在active状态下运行。
解决办法:
将node01调整为active状态,具体做法可以杀死node02 对应的端口,让切换过去之后再开启;
或者,直接将目标虚拟机改为node02,另外这部分:
书写中,还要严格控制大小写:
sc.textFile("hdfs://node02:8080/spark/data/words").flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).collect()