CHD 5.10 离线安装
安装虚拟机，我用的是centos6.5的版本。
选择回车
重启机器
设置虚拟机网络
1.集群节点规划,安装三台机器，每台机器都执行以下操作，切记！
[[email protected] ~]# vi /etc/sysconfig/networkNETWORKING=yes
HOSTNAME=hadoop-01 # 机器名称
GATEWAY=192.168.137.2 # 网关
[[email protected] ~]# vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.137.130 hadoop-cdh1
192.168.137.131 hadoop-cdh2
192.168.137.132 hadoop-cdh3
[[email protected] ~]# vi /etc/resolv.conf
search localdomain
nameserver 192.168.137.2
nameserver 8.8.8.8 #ping 外网
nameserver 8.8.4.4
[[email protected] ~]# vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
IPADDR=192.168.137.130 #ip
NETMASK=225.225.225.0
GATEWAY=192.168.137.2
/etc/udev/rules.d/70-persistent-net.rules
1) Linux操作系统中永久性生效，重启后不会复原
开启： chkconfig iptables on
关闭： chkconfig iptables off
2) 即时生效，重启后复原
开启： service iptables start
关闭： service iptables stop
service network restart 重启网卡
service iptables status可以查看到iptables服务的当前状态。
yum install -y lrzsz
yum -y install wget
yum -y install vim
三台机器设置完成后，reboot重启三台机器，检查机器配置。
hostname 是否正确。
hostname -i是否正确。
ping www .baidu.com是否可以ping通。
安装mysql
https://blog.csdn.net/nszkadrgg/article/details/78666628
配置hadoop用户的ssh信任关系(三台机器都要安装)
安装无密码登录(三台机器都要安装)
[[email protected] ~]# yum -y install openssh-clients
[[email protected] java]# ssh-****** -t rsa -P '' -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Created directory '/root/.ssh'.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
02:97:cf:e5:85:f4:3d:77:4b:24:18:23:9d:47:0e:1b [email protected]
The key's randomart image is:
+--[ RSA 2048]----+
| .oE=o . |
| . ..**oo |
| . o o.+.o.o|
| o o o . .oo|
| . S . . |
| . |
| |
| |
| |
+-----------------+
[[email protected] java]# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[[email protected] java]# chmod 0600 ~/.ssh/authorized_keys
测试三台机器ssh登录(每台都要测试)
[[email protected] java]# ssh hadoop-cdh2
The authenticity of host 'hadoop-cdh2 (192.168.137.131)' can't be established.
RSA key fingerprint is 69:99:33:84:fe:91:f6:43:df:4b:0c:63:ba:9b:c1:fc.
Are you sure you want to continue connecting (yes/no)?
Host key verification failed.
[[email protected] java]# ssh hadoop-cdh2
The authenticity of host 'hadoop-cdh2 (192.168.137.131)' can't be established.
RSA key fingerprint is 69:99:33:84:fe:91:f6:43:df:4b:0c:63:ba:9b:c1:fc.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hadoop-cdh2,192.168.137.131' (RSA) to the list of known hosts.
[email protected]'s password:
Last login: Sat Apr 21 03:54:12 2018 from hadoop-cdh3
检查系统版本是否定一致 (每台都要做)
[[email protected] ~]# cat /etc/redhat-release
CentOS release 6.5 (Final)
设置时钟同步(每台都要做)
[[email protected] ~]# grep ZONE /etc/sysconfig/clock
ZONE="Asia/Shanghai"
修改机器的时间
[[email protected] ~]# date
Sat Apr 21 04:07:25 CST 2018
[[email protected] ~]# date -s 04/20/2018
Fri Apr 20 00:00:00 CST 2018
[[email protected] ~]# date -s 20:32:00
Fri Apr 20 20:32:00 CST 2018
[[email protected] ~]# dare
-bash: dare: command not found
[[email protected] ~]# date
Fri Apr 20 20:32:06 CST 2018
安装ntpdate(每台机器都要安装)
[[email protected] ~]# yum -y install ntpdate
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.zju.edu.cn
* extras: mirrors.aliyun.com
* updates: mirrors.njupt.edu.cn
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package ntpdate.x86_64 0:4.2.6p5-12.el6.centos.2 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
=======================================================================================================================================================================================================================
Package Arch Version Repository Size
=======================================================================================================================================================================================================================
Installing:
ntpdate x86_64 4.2.6p5-12.el6.centos.2 updates 79 k
Transaction Summary
=======================================================================================================================================================================================================================
Install 1 Package(s)
Total download size: 79 k
Installed size: 123 k
Downloading Packages:
ntpdate-4.2.6p5-12.el6.centos.2.x86_64.rpm | 79 kB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : ntpdate-4.2.6p5-12.el6.centos.2.x86_64 1/1
Verifying : ntpdate-4.2.6p5-12.el6.centos.2.x86_64 1/1
Installed:
ntpdate.x86_64 0:4.2.6p5-12.el6.centos.2
Complete!
安装ntp服务(三台都要安装)
yum -y install ntp
启动ntpd服务(三台机器都要启动)
[[email protected] ~]# service ntpd start
Starting ntpd: [ OK ]
时间设置同步
[[email protected] ~]# ntpdate hadoop-cdh2
20 Apr 20:41:06 ntpdate[2676]: no server suitable for synchronization found
[[email protected] ~]# ntpdate hadoop-cdh3
20 Apr 20:41:06 ntpdate[2676]: no server suitable for synchronization found
安装和启动http服务
[[email protected] ~]# rpm -qa|grep httpd
httpd-tools-2.2.15-60.el6.centos.6.x86_64
httpd-2.2.15-60.el6.centos.6.x86_64
[[email protected] ~]#
其他两台没有安装，安装hadoop-cdh2，hadoop-cdh3两台的http服务，安装 yum install -y httpd
下面的三步三台都做
[[email protected] ~]# chkconfig --list|grep httpd
httpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
[[email protected] ~]# chkconfig httpd on
[[email protected] ~]# chkconfig --list|grep httpd
httpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
[[email protected] ~]# service httpd start
Starting httpd:
安装jdk，一定要用oracle版本的jdk，官网上可以下载，不要安装linux自带的openJDK。
[[email protected] ~]# rz #上传jdk-8u45-linux-x64.gz[[email protected] java]# mkdir -p /usr/java
[[email protected] java]# mv jdk-8u45-linux-x64.gz /usr/java
[[email protected] java]# tar -xzvf jdk-8u45-linux-x64.gz
解压后的jdk复制到hadoop-cdh2，hadoop-cdh3
其他的两台创建mkdir -p /usr/java目录
scp到其他的两台机器
scp -r /usr/java/jdk1.8.0_45/ hadoop-cdh2:/usr/java
scp -r /usr/java/jdk1.8.0_45/ hadoop-cdh3:/usr/java
配置jdk的环境变量
[[email protected] java]# vim /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_45
export PATH=$JAVA_HOME/bin:$PATH
[[email protected] java]# source /etc/profile
[[email protected] java]# java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
scp 到其他的两台机器
[[email protected] java]# scp -r /etc/profile/ hadoop-cdh2:/etc/profile
[[email protected] java]# scp -r /etc/profile/ hadoop-cdh3:/etc/profile
其他的两台机器source /ect/profile
接下来安装CDH
http://archive.cloudera.com/cdh5/parcels/5.10.0/
到CDH官网下载对应版本的parcels文件
我们存在parcel的位置是 /var/www/html
[[email protected] html]# mkdir parcels
[[email protected] html]# cd parcels/
[[email protected] parcels]# ll
total 1466572
-rw-r--r--. 1 root root 1501694035 Dec 10 19:14 CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel
-rw-r--r--. 1 root root 41 Dec 10 18:51 CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel.sha
-rw-r--r--. 1 root root 64807 Dec 10 19:15 manifest.json
[[email protected] parcels]# pwd
/var/www/html/parcels
CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel.sha1 后面的1去掉，之前我去掉了
[[email protected] parcels]# mv CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel.sha1 CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel.sha
校验你下载的文件是不是全的，如果不全也是不会安装成功的，所以一定要校验。
[[email protected] parcels]# sha1sum CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel
52f95da433f203a05c2fd33eb0f144e6a5c9d558 CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel
[[email protected] parcels]# cat CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel.sha
52f95da433f203a05c2fd33eb0f144e6a5c9d558
下载cm
http://archive.cloudera.com/cm5/repo-as-tarball/5.10.0/
校验下载的文件
解压
[[email protected] html]# tar -zxvf cm5.10.0-centos6.tar.gz
然后创建和官网一样的路径
修改文件名称
配置本地的yum源，集群自安装的时候就会从本地down包，不会从官网down(三台机器都要配置)
vim /etc/yum.repos.d/cloudrea-manager.repo 配置的内容一样，没有cloudrea-manager.repo文件就创建
在浏览器上查看是否配置成功，能打开说明配置成功。
官网参考链接
http://archive.cloudera.com/cdh5/parcels/5.10/
http://archive.cloudera.com/cm5/repo-as-tarball/5.10.0/
http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.10/
install server rpm in cm instance
install server
mysql创建元数据库和用户，两个都创建
修改service和mysql的配置
vim /etc/cloudera-scm-server/db.properties 修改后保存
添加mysql的驱动包
service 启动的日志
用户名密码默认admin
下一步
下一步
下一步
下一步
下一步
下一步
下一步
下一步
这一步很重要，点击下一步
开始分发到每台机器
下一步
关闭大页面
设置swap，三台机器都要设置
重新Run Again
没有黄色的提示了
现在安装hadoop的三个核心组件 ，hdfs, yarn,zookeeper
下面设置机器的角色分配
角色分配:
192.168.137.130 hadoop-cdh1
192.168.137.131 hadoop-cdh2
192.168.137.132 hadoop-cdh3
第一阶段角色:
hadoop-cdh1: mysql cm-server cm-agent
hadoop-cdh2: cm-agent
hadoop-cdh3: cm-agent
cm(5个服务):hadoop-cdh1
第二阶段角色:
hdfs 、yarn 、zk
hadoop-cdh1: NN DN RM NM ZK
hadoop-cdh2: DN SNN NM ZK
hadoop-cdh3: DN NM ZK
选择下一步，这里一定要有mysql 的驱动包，目录就按照下面的目录放置，别的目录有时候会找不到
[[email protected] cloudera-manager]# cd /usr/share/java/
[[email protected] java]# ll
total 964
-rw-r--r--. 1 root root 985603 Oct 22 2017 mysql-connector-java.jar
[[email protected] java]#
点击下一步，所有的都不要变
点击下一步，安装服务
安装完成，点击完成。
设置namenode最小1G。
查找heap，点击菜单的namenode，设置为1G，点击保存。
返回主菜单
重启
下一步
点击Finish
点击Finish
其他的黄色和红色的信息，根据你自己集群的情况，如果资源充足就设置，不充足就抑制。
service cloudera-scm-agent status 查看agent的状态(每台)
service cloudera-scm-service status 查看server的状态
正确的关机顺序
service cloudera-scm-agent stop 先关闭agent
service cloudera-scm-service stop 再关闭service
service mysql stop 再关闭mysql
开机顺序刚好相反，先启动mysql，再启动server，然后再启动agent。