win10搭建hadoop和spark
一、Java环境变量配置
Windows x64:https://www.oracle.com/java/technologies/javase-jdk8-downloads.html
二、hadoop环境变量配置
hadoop-2.7.7.tar.gz:https://mirrors.aliyun.com/apache/hadoop/common/hadoop-2.7.7/hadoop-2.7.7.tar.gz
三、spark环境变量配置
spark-2.4.0-bin-hadoop2.7.tgz:https://mirrors.huaweicloud.com/apache/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz
四、hdfs配置
1、配置hadoop-env.cmd文件里的JAVA_HOME
修改 hadoop-2.7.7\etc\hadoop 下的 hadoop-env.cmd 文件
2、直接启动,发现报错
3、将报错的关键信息记录下来
- 20/03/17 18:23:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
- 20/03/17 18:23:31 WARN nodemanager.DirectoryCollection: Unable to create directory /tmp/hadoop-com/nm-local-dir error (null) entry in command string: null chmod 0755 C:\tmp, removing from the list of valid directories.
- 20/03/17 18:23:31 WARN nodemanager.DirectoryCollection: Unable to create directory D:/Hadoop/HDFS/hadoop-2.7.7/logs/userlogs error (null) entry in command string: null chmod 0755 D:\Hadoop\HDFS\hadoop-2.7.7\logs\userlogs, removing from the list of valid directories.
- 20/03/17 18:23:31 ERROR nodemanager.LocalDirsHandlerService: Most of the disks failed. 1/1 local-dirs are bad: /tmp/hadoop-com/nm-local-dir; 1/1 log-dirs are bad: D:/Hadoop/HDFS/hadoop-2.7.7/logs/userlogs
- 20/03/17 18:23:29 ERROR namenode.NameNode: Failed to start namenode.
- java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): file:/// has no authority.
- 20/03/17 18:23:38 WARN datanode.DataNode: Invalid dfs.datanode.data.dir \tmp\hadoop-com\dfs\data :
- java.io.IOException: (null) entry in command string: null chmod 0700 C:\tmp\hadoop-com\dfs\data
- java.io.IOException: All directories in dfs.datanode.data.dir are invalid: "/C:/tmp/hadoop-com/dfs/data"
4、问题描述与解决办法
问题描述:
1、无法加载hadoop library下的jar到Windows平台
2、无法根据命令创建目录 /tmp/hadoop-com/nm-local-dir
3、无法根据命令创建目录 D:/Hadoop/HDFS/hadoop-2.7.7/logs/userlogs
4、初始化的磁盘报错 local-dirs、log-dirs
5、namenode启动失败
6、namenode url 参数错误,fs.defaultFS
7、无效的配置 dfs.datanode.data.dir \tmp\hadoop-com\dfs\data
8、输入 chmod 0700 C:\tmp\hadoop-com\dfs\data 命令为空,需要手动创建文件夹或者配置dfs\data
9、DataNode的目录无效:dfs.datanode.data.dir are invalid: "/C:/tmp/hadoop-com/dfs/data"
解决办法:
1、上github上下载winutils.exe,放到hadoop/bin目录下 https://github.com/steveloughran/winutils
2、配置 /tmp/hadoop-com/nm-local-dir
3、配置 /logs/userlogs
4、手动创建 local-dirs、log-dirs 目录,或者配置 local-dirs、log-dirs
下班了,坐地铁中,未完待续。。。