hadoop集群测试mapreduce程序的各种坑
Error1:datanode未能启动,namenode和datanode的clusterID不一致
原因:datanode日志显示java.io.IOException: Incompatible clusterIDs in /opt/hadoop-2.7.3/tmp/dfs/data: namenode clusterID = CID-add6cc33-56f0-4d7c-8484-60740bf85c7c; datanode clusterID = CID-d616c28f-a2f3-4196-8b55-bfc77d08e678
解决办法:将datanode的VERSION(/opt/hadoop-2.7.3/tmp/dfs/data/current的VERSION)的clusterID手动修改成为 namenode的VERSION(/opt/hadoop-2.7.3/tmp/dfs/name/current的VERSION)的clusterID,再重新启动集群就OK。
Error2: Input path does not exist: hdfs://master:9000/user/root/input
原因:按照资料执行 hadoop jar /opt/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.4.jar wordcount input output,input应该为/input,output为/output,不然会找到/user/root/input目录下(之前上传的wordcount.txt到/input目录下,而不是/user/root/input目录下)
解决办法:命令中将input修改为/input
hadoop jar /opt/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.4.jar wordcount /input /output
Error3:mapreduce一直卡在 Running
job: job_1512019590518_0003
原因:mapreduce测试时给的输入给了文件夹而不是文件.hadoop jar /opt/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.4.jar wordcount /input /output.导致 (waiting
for AM container to be allocated)
解决办法:将输入改为文件。
hadoop jar /opt/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.4.jar wordcount /input/wordcount.txt /output
对你有用的话给个赞吧b( ̄▽ ̄)d