Linux_Hadoop1.2.1 安装笔记
环境：Vmware Workstation 10，CentOS-7-x86_64-DVD-1511.iso，Xshell 4.0，Master：192.168.216.141，Slave1：192.168.216.142，Slave2：192.168.216.143.
给普通用户hadoop授予sudo权限
[[email protected] ~]$ su root
密码：
[[email protected] hadoop]# ll /etc/sudoers
-r–r—–. 1 root root 4188 7月 7 2015 /etc/sudoers
[[email protected] hadoop]# chmod u+w /etc/sudoers
[[email protected] hadoop]# ll /etc/sudoers
-rw-r—–. 1 root root 4188 7月 7 2015 /etc/sudoers
[[email protected] hadoop]# vim /etc/sudoers
root ALL=(ALL) ALL
hadoop ALL=(ALL) ALL
[[email protected] hadoop]# chmod u-w /etc/sudoers
[[email protected] hadoop]# ll /etc/sudoers
-r–r—–. 1 root root 4215 6月 20 17:56 /etc/sudoers
[[email protected] hadoop]# exit
exit
Master服务器设置ssh无密码登录
[[email protected] ~]$ ssh-****** -t rsa -P ”
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory ‘/home/hadoop/.ssh’.
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
[[email protected] ~]$ ll ~/.ssh
总用量 8
-rw——-. 1 hadoop hadoop 1675 6月 20 18:02 id_rsa
-rw-r–r–. 1 hadoop hadoop 398 6月 20 18:02 id_rsa.pub
[[email protected]localhost ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[[email protected] ~]$ chmod 600 ~/.ssh/authorized_keys
[[email protected] ~]$ ll ~/.ssh/
authorized_keys的权限必须是600
总用量 16
-rw——-. 1 hadoop hadoop 398 6月 20 18:03 authorized_keys
-rw——-. 1 hadoop hadoop 1675 6月 20 18:02 id_rsa
-rw-r–r–. 1 hadoop hadoop 398 6月 20 18:02 id_rsa.pub
[[email protected] ~]$ sudo vim /etc/ssh/sshd_config
AuthorizedKeysFile .ssh/authorized_keys
[[email protected] ~]$ ssh localhost
The authenticity of host ‘localhost (::1)’ can’t be established.
ECDSA key fingerprint is 1b:1d:06:bb:5c:dd:92:e6:22:ff:03:b3:a7:c0:e7:ed.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘localhost’ (ECDSA) to the list of known hosts.
[email protected]’s password:
Last login: Tue Jun 20 17:44:21 2017 from 192.168.216.1
[[email protected] ~]$ su
密码：
[[email protected] hadoop]# service sshd restart
Redirecting to /bin/systemctl restart sshd.service
[[email protected] hadoop]# exit
exit
[[email protected] ~]$ ssh localhost
The authenticity of host ‘localhost (::1)’ can’t be established.
ECDSA key fingerprint is 1b:1d:06:bb:5c:dd:92:e6:22:ff:03:b3:a7:c0:e7:ed.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘localhost’ (ECDSA) to the list of known hosts.
Last failed login: Tue Jun 20 18:09:37 CST 2017 from localhost on ssh:notty
There were 4 failed login attempts since the last successful login.
Last login: Tue Jun 20 18:06:00 2017 from localhost
将~/.ssh文件夹所有发送给Slave1和Slave2：
[[email protected] ~]$ scp -r ~/.ssh [email protected]:~
[email protected]’s password:
id_rsa 100% 1675 1.6KB/s 00:00
id_rsa.pub 100% 398 0.4KB/s 00:00
authorized_keys 100% 398 0.4KB/s 00:00
known_hosts 100% 348 0.3KB/s 00:00
[[email protected] ~]$ scp -r ~/.ssh [email protected]:~
The authenticity of host ‘192.168.216.143 (192.168.216.143)’ can’t be established.
ECDSA key fingerprint is 64:05:c3:ec:5e:cb:64:d8:41:74:d6:ee:cb:c3:84:59.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘192.168.216.143’ (ECDSA) to the list of known hosts.
[email protected]’s password:
id_rsa 100% 1675 1.6KB/s 00:00
id_rsa.pub 100% 398 0.4KB/s 00:00
authorized_keys 100% 398 0.4KB/s 00:00
known_hosts 100% 525 0.5KB/s 00:00
接下来可以测试Master，Slave1和Slave2相互免密码登录。
Master服务器
[[email protected] ~]$ ssh 192.168.216.142
Last login: Tue Jun 20 18:24:22 2017 from 192.168.216.142
[[email protected] ~]$ ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno16777736: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:9e:9f:16 brd ff:ff:ff:ff:ff:ff
inet 192.168.216.142/24 brd 192.168.216.255 scope global dynamic eno16777736
valid_lft 1148sec preferred_lft 1148sec
inet6 fe80::20c:29ff:fe9e:9f16/64 scope link
valid_lft forever preferred_lft forever
[[email protected] ~]$ exit
登出
Connection to 192.168.216.142 closed.
[[email protected] ~]$ ssh 192.168.216.143
Last login: Tue Jun 20 17:30:54 2017 from 192.168.216.1
[[email protected] ~]$ exit
登出
Connection to 192.168.216.143 closed.
同理从Slave1到Master, Slave2登录；Slave2到Master, Slave1登录。
至此准备阶段结束
Master服务器
[[email protected] ~]$ wget http://archive.apache.org/dist/hadoop/core/hadoop-1.2.1/hadoop-1.2.1.tar.gz
[[email protected] ~]$ sudo tar -zxvf hadoop-1.2.1.tar.gz -C /usr/local/
[[email protected] ~]$ sudo mv /usr/local/hadoop-1.2.1/ /usr/local/hadoop
[[email protected] ~]$ ll /usr/local/ | grep hadoop
drwxr-xr-x. 15 root root 4096 7月 23 2013 hadoop
[[email protected] ~]$ sudo chown -R hadoop:hadoop /usr/local/hadoop/
[[email protected] ~]$ ll /usr/local/ | grep hadoop
drwxr-xr-x. 15 hadoop hadoop 4096 7月 23 2013 hadoop
[[email protected] ~]$ sudo vim /etc/profile
[[email protected] ~]$ source /etc/profile
[[email protected] ~]$ hadoop
Usage: hadoop [–config confdir] COMMAND
配置文件
[[email protected] ~]$ vim /usr/local/hadoop/conf/hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_131
[[email protected] ~]$ vim /usr/local/hadoop/conf/core-site.xml
[[email protected] ~]$ vim /usr/local/hadoop/conf/hdfs-site.xml
[[email protected] ~]$ vim /usr/local/hadoop/conf/mapred-site.xml
[[email protected] ~]$ vim /usr/local/hadoop/conf/masters
192.168.216.141
[[email protected] ~]$ vim /usr/local/hadoop/conf/slaves
192.168.216.142
192.168.216.143
[[email protected] ~]$ cd /usr/local/hadoop/ && mkdir datanode tmp
[[email protected] ~]$ chmod 755 /usr/local/hadoop/tmp
[[email protected] ~]$ chmod 755 /usr/local/hadoop/datanode
注意：datanode和tmp目录的权限必须是755，否则datanode启动失败。
将/usr/local/hadoop整个文件夹发送到Slave1和Slave2：
[[email protected] ~]$ scp -r /usr/local/hadoop/ 192.168.216.142:~
[[email protected] ~]$ scp -r /usr/local/hadoop/ 192.168.216.143:~
Slave1和Slave2上操作
[[email protected] ~]$ sudo mv hadoop /usr/local/
[[email protected] ~]$ sudo vim /etc/profile
[[email protected] ~]$ source /etc/profile
[[email protected] ~]$ hadoop
Usage: hadoop [–config confdir] COMMAND
至此hadoop一主两从部署完毕
Master，Slave1和Slave2
关闭防火墙
注意：下面命令中“–”是两个“-”，****显示有误。
[[email protected] hadoop]$ firewall-cmd –state
running
[[email protected] hadoop]$ sudo systemctl stop firewalld
[sudo] password for hadoop:
[[email protected] hadoop]$ firewall-cmd –state
not running
格式化namenode文件系统
[[email protected] hadoop]$ hadoop namenode -format
17/06/22 14:16:49 INFO namenode.NameNode: STARTUP_MSG:
/**************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = localhost/127.0.0.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by ‘mattf’ on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.8.0_131
**************************************************/
17/06/22 14:16:49 INFO util.GSet: Computing capacity for map BlocksMap
17/06/22 14:16:49 INFO util.GSet: VM type = 64-bit
17/06/22 14:16:49 INFO util.GSet: 2.0% max memory = 1013645312
17/06/22 14:16:49 INFO util.GSet: capacity = 2^21 = 2097152 entries
17/06/22 14:16:49 INFO util.GSet: recommended=2097152, actual=2097152
17/06/22 14:16:49 INFO namenode.FSNamesystem: fsOwner=hadoop
17/06/22 14:16:49 INFO namenode.FSNamesystem: supergroup=supergroup
17/06/22 14:16:49 INFO namenode.FSNamesystem: isPermissionEnabled=false
17/06/22 14:16:49 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
17/06/22 14:16:49 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
17/06/22 14:16:49 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
17/06/22 14:16:49 INFO namenode.NameNode: Caching file names occuring more than 10 times
17/06/22 14:16:50 INFO common.Storage: Image file /usr/local/hadoop/name/current/fsimage of size 112 bytes saved in 0 seconds.
17/06/22 14:16:50 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/usr/local/hadoop/name/current/edits
17/06/22 14:16:50 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/usr/local/hadoop/name/current/edits
17/06/22 14:16:50 INFO common.Storage: Storage directory /usr/local/hadoop/name has been successfully formatted.
17/06/22 14:16:50 INFO namenode.NameNode: SHUTDOWN_MSG:
/**************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1
**************************************************/
Master
[[email protected] hadoop]$ start-all.sh
starting namenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-namenode-localhost.out
192.168.216.143: datanode running as process 12402. Stop it first.
192.168.216.142: datanode running as process 12458. Stop it first.
The authenticity of host ‘192.168.216.141 (192.168.216.141)’ can’t be established.
ECDSA key fingerprint is 45:3b:37:83:b6:7f:20:55:47:10:b7:ad:63:ae:96:33.
Are you sure you want to continue connecting (yes/no)? yesw^H
192.168.216.141: Warning: Permanently added ‘192.168.216.141’ (ECDSA) to the list of known hosts.
192.168.216.141: secondarynamenode running as process 12610. Stop it first.
starting jobtracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-jobtracker-localhost.out
192.168.216.143: tasktracker running as process 12673. Stop it first.
192.168.216.142: tasktracker running as process 12731. Stop it first.
[[email protected] hadoop]$ jps
12610 SecondaryNameNode
12866 Jps
12758 JobTracker
12442 NameNode
[[email protected] hadoop]$ hadoop dfsadmin -report
Configured Capacity: 37492883456 (34.92 GB)
Present Capacity: 34187776030 (31.84 GB)
DFS Remaining: 34187759616 (31.84 GB)
DFS Used: 16414 (16.03 KB)
DFS Used%: 0%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Datanodes available: 2 (2 total, 0 dead)
Name: 192.168.216.142:50010
Decommission Status : Normal
Configured Capacity: 18746441728 (17.46 GB)
DFS Used: 8207 (8.01 KB)
Non DFS Used: 1652527089 (1.54 GB)
DFS Remaining: 17093906432(15.92 GB)
DFS Used%: 0%
DFS Remaining%: 91.18%
Last contact: Thu Jun 22 14:19:56 CST 2017
Name: 192.168.216.143:50010
Decommission Status : Normal
Configured Capacity: 18746441728 (17.46 GB)
DFS Used: 8207 (8.01 KB)
Non DFS Used: 1652580337 (1.54 GB)
DFS Remaining: 17093853184(15.92 GB)
DFS Used%: 0%
DFS Remaining%: 91.18%
Last contact: Thu Jun 22 14:19:56 CST 2017
Slave1
[[email protected] hadoop]$ start-all.sh
starting namenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-namenode-localhost.out
192.168.216.143: starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-datanode-localhost.out
192.168.216.142: starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-datanode-localhost.out
192.168.216.141: secondarynamenode running as process 12610. Stop it first.
starting jobtracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-jobtracker-localhost.out
192.168.216.142: tasktracker running as process 12731. Stop it first.
192.168.216.143: tasktracker running as process 12673. Stop it first.
[[email protected] hadoop]$ jps
12458 DataNode
12731 TaskTracker
12847 Jps
Slave2
[[email protected] hadoop]$ start-all.sh
starting namenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-namenode-localhost.out
192.168.216.143: datanode running as process 12402. Stop it first.
192.168.216.142: datanode running as process 12458. Stop it first.
192.168.216.141: starting secondarynamenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-secondarynamenode-localhost.out
starting jobtracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-jobtracker-localhost.out
192.168.216.142: starting tasktracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-localhost.out
192.168.216.143: starting tasktracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-localhost.out
[[email protected] hadoop]$ jps
12673 TaskTracker
12402 DataNode
12793 Jps