https://archive.cloudera.com/cdh6/6.0.1/parcels/，需要下载以下文件：CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcelCDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha256manifest.json

RepositoriesType                 Location (baseurl)                                 Repo FileRHEL 7 Compatible    https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/   cloudera-manager.repo

[[email protected] ~]# uname -aLinux hadoop15 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux[[email protected] ~]# cat /etc/redhat-release CentOS Linux release 7.4.1708 (Core)

IP             内存 操作系统     角色192.168.106.155 12G Centos7.4 Server + agent192.168.106.156 9G Centos7.4 Agent192.168.106.157 8G Centos7.4 Agent 192.168.106.151 1.5G Centos7.4   mysql

yum -y install wget

yum -y install vim-enhanced

A:105机器[[email protected] ~]# hostnamectl set-hostname  cdh1B:106机器[[email protected] ~]# hostnamectl set-hostname  cdh2C:107机器[[email protected] ~]# hostnamectl set-hostname  cdh3

[[email protected] ~]# hostnamecdh1[[email protected] ~]# hostnamecdh2[[email protected] ~]# hostnamecdh2

[[email protected] etc]# cat /etc/hosts192.168.106.155   cdh1192.168.106.156   cdh2192.168.106.157   cdh3

[[email protected] ~]# vim /etc/sysconfig/network[[email protected] ~]# cat /etc/sysconfig/networkHOSTNAME=cdh1[[email protected] ~]# vim /etc/sysconfig/network[[email protected] ~]# cat /etc/sysconfig/networkHOSTNAME=cdh2[[email protected] ~]# vim /etc/sysconfig/network[[email protected] ~]# cat /etc/sysconfig/networkHOSTNAME=cdh3

停止： systemctl disable firewalld禁用： systemctl stop firewalld

vim /etc/sysconfig/selinuxSELINUX=enforcing 改为 SELINUX=disabled

[[email protected] etc]# getenforceDisabled[[email protected] etc]#

$ setenforce 0

setenforce 1

yum install ntp

vim /etc/ntp.conf

service ntpd startchkconfig ntpd on

yum install centos-release-scl

yum install scl-utils

yum install python27

[[email protected] ~]# source /opt/rh/python27/enable[[email protected] ~]# python --versionPython 2.7.13

ssh-****** -t rsa   （这个过程一直打enter）ssh-copy-id cdh1ssh-copy-id cdh2ssh-copy-id cdh3

ssh cdh1ssh cdh2ssh cdh3

yum install oracle-j2sdk1.8

rpm -ivh oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm

参考文章：https://www.cloudera.com/documentation/enterprise/6/6.0/topics/configure_cm_repo.htmlhttp://www.cnblogs.com/piperck/p/9944469.html首先Cloudera为用户准备好了专用的程序仓库，需要下载下来wget https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/# Import the repository signing GPG keyrpm --import https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPM-GPG-KEY-cloudera

OS 命令RHEL, CentOS, Oracle Linux  :    sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-serverSLES                       :    sudo zypper install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-serverUbuntu                         :   sudo apt-get install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpmcloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpmcloudera-manager-server-6.0.1-610811.el7.x86_64.rpmjdkoracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm

Master节点（cdh1）：yum -y install cloudera-manager-agent cloudera-manager-daemons cloudera-manager-serverNode节点(cdh2,cdh3)：yum -y install cloudera-manager-agent cloudera-manager-daemons本地安装：（网络好的话，可以忽略此步骤。）cloudera-manager-agent-6.0.1-610811.el7.x8664.rpmcloudera-manager-daemons-6.0.1-610811.el7.x8664.rpmcloudera-manager-server-6.0.1-610811.el7.x8664.rpm先安装 cloudera-manager-daemons-6.0.1-610811.el7.x8664.rpm 再安装其他的。

yum install bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi portmap httpd mod_ssl openssl-devel python-psycopg2 MySQL-python /lib/lsb/init-functions

rpm -ivh cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpmrpm -ivh cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpmrpm -ivh cloudera-manager-server-6.0.1-610811.el7.x86_64.rpmrpm -ivh cloudera-manager-server-db-2-6.0.1-610811.el7.x86_64.rpm

rpm -ivh cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpmrpm -ivh cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm

JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera /opt/cloudera/cm-agent/bin/certmanager --location /opt/cloudera/CMCA setup --configure-services

wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpmyum -y install mysql-community-release-el7-5.noarch.rpmyum -y updateyum -y install mysql-server*.Move old InnoDB log files /var/lib/mysql/iblogfile0 and /var/lib/mysql/iblogfile1 out of /var/lib/mysql/ to a backup location.rm -rf /var/lib/mysql/ib_logfile0 /var/lib/mysql/ib_logfile1启动mysqlsystemctl enable mysqldsystemctl start mysqld初始化Mysql/usr/bin/mysql_secure_installation[...]Enter current password for root (enter for none):OK, successfully used password, moving on...[...]Set root password? [Y/n] YNew password:rootRe-enter new password:rootRemove anonymous users? [Y/n] Y[...]Disallow root login remotely? [Y/n] N[...]Remove test database and access to it [Y/n] Y[...]Reload privilege tables now? [Y/n] YAll done!

wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gztar zxvf mysql-connector-java-5.1.46.tar.gz#要注意的是后面的名字必须是mysql-connector-java.jar（可以将下面的jar包放到3台机器上）cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

mysql -u root -p rootDROP DATABASE scm;DROP DATABASE amon;DROP DATABASE rman;DROP DATABASE hue;DROP DATABASE metastore;DROP DATABASE sentry;DROP DATABASE nav;DROP DATABASE oozie;DROP DATABASE activity_monitor;CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;GRANT ALL ON hive.* TO 'hive'@'%' IDENTIFIED BY 'root';create database scm default character set utf8 default collate utf8_general_ci;grant all on scm.* to 'scm'@'%' identified by 'root';create database amon default character set utf8 default collate utf8_general_ci;grant all on amon.* to 'amon'@'%' identified by 'root';create database rman default character set utf8 default collate utf8_general_ci;grant all on rman.* to 'rman'@'%' identified by 'root';create database hue default character set utf8 default collate utf8_general_ci;grant all on hue.* to 'hue'@'%' identified by 'root';create database metastore default character set utf8 default collate utf8_general_ci;grant all on metastore.* to 'hive'@'%' identified by 'root';create database sentry default character set utf8 default collate utf8_general_ci;grant all on sentry.* to 'sentry'@'%' identified by 'root';create database nav default character set utf8 default collate utf8_general_ci;grant all on nav.* to 'nav'@'%' identified by 'root';create database oozie default character set utf8 default collate utf8_general_ci;grant all on oozie.* to 'oozie'@'%' identified by 'root';CREATE DATABASE activity_monitor DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;GRANT ALL ON activity_monitor.* TO 'activity_monitor'@'%' IDENTIFIED BY 'root';flush privileges;

echo never > /sys/kernel/mm/transparent_hugepage/defragecho never > /sys/kernel/mm/transparent_hugepage/enabled并将上面的两条命令写入开机自启动vim /etc/rc.local

vim /etc/sysctl.confvm.swappiness = 10sysctl -p /etc/sysctl.conf

echo never>/sys/kernel/mm/transparent_hugepage/defragecho never>/sys/kernel/mm/transparent_hugepage/enabled

mysql在本地时执行：rm -rf /etc/cloudera-scm-server/db.mgmt.properties/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm如果不在一台机器上，执行类似如下命令（cdh2为mysql所在位置，cdh1为cloudera manager server所在位置）：[[email protected] cloudera-scm-server]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h 192.168.106.151 --scm-host cdh1 scm scm[[email protected] cloudera-scm-server]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h 192.168.106.151 --scm-host cdh1 scm scmEnter SCM password: JAVA_HOME=/usr/java/jdk1.8.0_141-clouderaVerifying that we can write to /etc/cloudera-scm-serverCreating SCM configuration file in /etc/cloudera-scm-serverExecuting:  /usr/java/jdk1.8.0_141-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.[                          main] DbCommandExecutor              INFO  Successfully connected to database.All done, your SCM database is configured correctly![[email protected] cloudera-scm-server]#

将CHD6相关的Parcel包放到主节点的/opt/cloudera/parcel-repo/目录中，如果没有此目录，可以自己创建。（如果是在线安装，那么此步骤可以忽略。）注意：最后将CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha256，重命名为CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha，这点必须注意，否则系统会重新下载CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel。存放了CDH和修改了HASH之后，设置Cloudera Manager数据库，启动Cloudera Manager Server，然后登录Cloudera Manager Admin Console：启动Cloudera Manager Server：systemctl start cloudera-scm-server可以查看启动日志tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log看到此条信息，说明启动完成INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

在Web浏览器中，转到 http://cdh1:7180/cmf/login，其中域名是运行Cloudera Manager Server的主机的FQDN或IP地址。如果有启用自动TLS，则会重定向到https: cdh1:7183，并显示安全警告。你可能需要表明你信任该证书，或单击以继续执行Cloudera Manager Server主机。登录Cloudera Manager Admin Console。默认凭据为：Username: adminPassword: admin

systemctl stop cloudera-scm-serversystemctl stop cloudera-scm-agentyum -y remove 'cloudera-manager-*'yum clean allumount cm_processesumount /var/run/cloudera-scm-agent/processrm -Rf /usr/share/cmf /var/lib/cloudera* /var/cache/yum/cloudera* /var/log/cloudera* /var/run/cloudera*rm -rf /tmp/.scmpreparenode.lockrm -Rf /var/lib/flume-ng /var/lib/hadoop* /var/lib/hue /var/lib/navigator /var/lib/oozie /var/lib/solr /var/lib/sqoop* /var/lib/zookeeperrm -Rf datadrivepath/dfs datadrivepath/mapred datadrivepath/yarnrm -rf /var/lib/hadoop-* /var/lib/impala /var/lib/solr /var/lib/zookeeper /var/lib/hue /var/lib/oozie  /var/lib/pgsql  /var/lib/sqoop2  /data/dfs/  /data/impala/ /data/yarn/  /dfs/ /impala/ /yarn/  /var/run/hadoop-*/ /var/run/hdfs-*/ /usr/bin/hadoop* /usr/bin/zookeeper* /usr/bin/hbase* /usr/bin/hive* /usr/bin/hdfs /usr/bin/mapred /usr/bin/yarn /usr/bin/sqoop* /usr/bin/oozie /etc/hadoop* /etc/zookeeper* /etc/hive* /etc/hue /etc/impala /etc/sqoop* /etc/oozie /etc/hbase* /etc/hcatalogsystemctl stop mysqldyum -y remove mysqlrm -rf /var/lib/mysqlrm -rf /var/log/mysqld.logrm -rf /var/lib/mysql/mysqlrm -rf /usr/lib64/mysqlrm -rf /usr/share/mysqlrm -rf /opt/clouderarpm -qa | grep -i mysql卸载MySQL相关的文件。

以下，第一行命令在cdh1上执行。第二行在所有的节点上执行systemctl stop cloudera-scm-serversystemctl stop cloudera-scm-agent(cdh1上执行)systemctl start cloudera-scm-server (cdh1,cdh2,cdh3上都执行)systemctl start cloudera-scm-agent  [[email protected] init.d]# ps -ef | grep cloudera-scm-server  （在CDH上执行）查看CM的日志：[[email protected] init.d]# tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

1 Cloudera 文档总地址

https://www.cloudera.com/documentation/enterprise/6/6.0.html

2 Cloudera Manager 和CDH下载地址

Cloudera Manager下载地址：
https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/

下载上线的rpm，然后把这些文件放到/root目录下

CDH下载地址：

https://archive.cloudera.com/cdh6/6.0.1/parcels/，需要下载以下文件：CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcelCDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha256manifest.json

• 1
• 2
• 3
• 4

以上三个文件放到/opt/cloudera/parcel-repo/目录下，如果没有此目录，可以自己手动创建。

此外在下面的官网介绍中：
https://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cm_6_version_download.html

里面有
Cloudera Manager Release Notes
Cloudera Installation Guide
Gloudera Enterprise Upgrade Guide

RepositoriesType                 Location (baseurl)                                 Repo FileRHEL 7 Compatible    https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/   cloudera-manager.repo

• 1
• 2
• 3

如果想下载其它的软件rpm包，下载地址是：
https://archive.cloudera.com/cdh6/6.1.0/redhat7/yum/RPMS/x86_64/

3 阅读Introduction中的CDH

阅读以下章节

4 安装CDH 6.0.1集群

以下安装过程分别参考：http://www.cnblogs.com/piperck/p/9944469.html 和 官网

4.1 软件环境

操作系统

[[email protected] ~]# uname -aLinux hadoop15 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux[[email protected] ~]# cat /etc/redhat-release CentOS Linux release 7.4.1708 (Core)

• 1
• 2
• 3
• 4
• 5

机器配置
要注意的是，不同角色的机器，内存可以配置不一样，一般做Master节点的机器所需要的内存要多一些

IP             内存 操作系统     角色192.168.106.155 12G Centos7.4 Server + agent192.168.106.156 9G Centos7.4 Agent192.168.106.157 8G Centos7.4 Agent 192.168.106.151 1.5G Centos7.4   mysql

• 1
• 2
• 3
• 4
• 5

4.2 安装CDH 的准备工作

4.2.1  安装wget

yum -y install wget

• 1

4.2.2  更新安装vim，安装vim增强包

yum -y install vim-enhanced

• 1

4.2.3  配置host和hostname

官网介绍：

配置域名相关，因为只有三台机器组集群，所以直接使用了hosts的方法，参考官网地址可以是：https://www.cloudera.com/documentation/enterprise/6/6.0/topics/configure_network_names.html
1.配置在集群中的每个节点的host

A:105机器[[email protected] ~]# hostnamectl set-hostname  cdh1B:106机器[[email protected] ~]# hostnamectl set-hostname  cdh2C:107机器[[email protected] ~]# hostnamectl set-hostname  cdh3

• 1
• 2
• 3
• 4
• 5
• 6

检查hostname

[[email protected] ~]# hostnamecdh1[[email protected] ~]# hostnamecdh2[[email protected] ~]# hostnamecdh2

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8

2、编辑/etc/hosts
参考网址：https://www.cloudera.com/documentation/enterprise/6/6.0/topics/configure_network_names.html，三台机器配置成下面相同的配置：

[[email protected] etc]# cat /etc/hosts192.168.106.155   cdh1192.168.106.156   cdh2192.168.106.157   cdh3

• 1
• 2
• 3
• 4

3、编辑/etc/sysconfig/network
参考地址：https://www.cloudera.com/documentation/enterprise/6/6.0/topics/configure_network_names.html

[[email protected] ~]# vim /etc/sysconfig/network[[email protected] ~]# cat /etc/sysconfig/networkHOSTNAME=cdh1[[email protected] ~]# vim /etc/sysconfig/network[[email protected] ~]# cat /etc/sysconfig/networkHOSTNAME=cdh2[[email protected] ~]# vim /etc/sysconfig/network[[email protected] ~]# cat /etc/sysconfig/networkHOSTNAME=cdh3

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11

验证配置
uname –a 需要和 hostname 得到一致的域名
修改前：

修改后：

4.2.4  设置防火墙

参考文章：https://www.cloudera.com/documentation/enterprise/6/6.0/topics/install_cdh_disable_iptables.html

1.保存存在的iptables规则设置(略)
2.禁用iptables (centos7发现没有),替代的是firewalled

停止： systemctl disable firewalld禁用： systemctl stop firewalld

• 1
• 2

4.2.5  设置SELinux

vim /etc/sysconfig/selinuxSELINUX=enforcing 改为 SELINUX=disabled

• 1
• 2

或使用如下配置：

1.检查SELinux的状态

[[email protected] etc]# getenforceDisabled[[email protected] etc]#

• 1
• 2
• 3

2.如果输出是Permissive或者Disabled,你可以跳过此任务，然后开始下面的操作
3.打开/etc/selinux/config文件，在一些系统中，是/etc/sysconfig/selinux 文件
4.修改文件中的SELINUX=enforcing为SELINUX=permissive
笔者配置如下：

5.保存和关闭这个文件
6.重启系统或者使用下面的命令立即停止SELinux

$ setenforce 0

• 1

当你已经安装部署了CDH之后，你可以在/etc/selinux/config（或/etc/sysconfig/selinux 文件）通过改变SELINUX=permissive为SELINUX=enforcing重新启用SELINUX，接着运行下面的命令快速切换到enforcing模式

setenforce 1

• 1

4.2.6 启用NTP

文档地址：https://www.cloudera.com/documentation/enterprise/6/6.0/topics/install_cdh_enable_ntp.html

CDH需要你在集群中的每个机器中配置Network Time Protocal（NTP）服务，在开机的时候自动重启NTP。在你的集群中执行下面的每个步骤。
1. 安装NTP
兼容RHEL的命令：

yum install ntp

• 1

2. 配置NTP
官网此处的配置，笔者没有配置

A:配置NTP

vim /etc/ntp.conf

• 1

master配置：(选用复旦大学ntp服务器)

slave配置：（同步master）

B:开启NTP服务

service ntpd startchkconfig ntpd on

• 1
• 2

C:查看同步效果
命令：ntpstat

4.2.7 安装Hue所需的Python2.7

1.安装Software Collections Library （3个节点都执行以下命令）

yum install centos-release-scl

• 1

2.安装Software Collections utilities （3个节点都执行以下命令）

yum install scl-utils

• 1

3.安装Python 2.7 （3个节点都执行以下命令）

yum install python27

• 1

4.验证python2.7已经被安装了

[[email protected] ~]# source /opt/rh/python27/enable[[email protected] ~]# python --versionPython 2.7.13

• 1
• 2
• 3

4.2.8  3台机器间设置SSH免密

在cdh1，cdh2，cdh3下执行

ssh-****** -t rsa   （这个过程一直打enter）ssh-copy-id cdh1ssh-copy-id cdh2ssh-copy-id cdh3

• 1
• 2
• 3
• 4

执行完成之后，分别执行

ssh cdh1ssh cdh2ssh cdh3

• 1
• 2
• 3

如果不需要再进行密码设置，说明免密设置已经完成了

4.3 安装Cloudera Manager和CDH

4.3.1 安装JDK

关于JDK要求：https://www.cloudera.com/documentation/enterprise/6/6.0/topics/cdh_ig_jdk_installation.html

这里使用oracle-j2sdk1.8，此步骤略,按照官网进行安装

Jdk要求：
A:JDK必须是64位的，不要使用32位的
B:每个集群节点都要安装
C:JDK必须安装在/usr/java/jdk-version.

yum install oracle-j2sdk1.8

• 1

如果下载了这个jdk，直接使用：

rpm -ivh oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm

• 1

4.3.2 为Clouder Manager配置一个Repository (如果已经下载了Cloudera Manager，此步骤可以忽略)

参考文章：https://www.cloudera.com/documentation/enterprise/6/6.0/topics/configure_cm_repo.htmlhttp://www.cnblogs.com/piperck/p/9944469.html首先Cloudera为用户准备好了专用的程序仓库，需要下载下来wget https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/# Import the repository signing GPG keyrpm --import https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPM-GPG-KEY-cloudera

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9

4.3.3 安装Cloudera Manager Package

参考地址：https://www.cloudera.com/documentation/enterprise/6/6.0/topics/install_cm_server.html

在下面这些步骤中，在Cloudera Manager机器（cdh1）上安装Cloudera Manager Packages，选择启用 自动-TLS

4.3.3.1  Install Cloudera Manager Packages

A:在Cloudera Manager服务器上，键入下面的命令安装Cloudera Manager

OS 命令RHEL, CentOS, Oracle Linux  :    sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-serverSLES                       :    sudo zypper install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-serverUbuntu                         :   sudo apt-get install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

• 1
• 2
• 3
• 4

B:如果你为Cloudera Manager使用的是Oracle database，在Cloudera Manager服务器上编辑/etc/default/cloudera-scm-server文件。定位到以export CMF_JAVA_OPTS的一行，改变-Xmx2G为-Xmx4G

如果网速不行，可以先下好Cloudera Manager和CDH。下载地址是：
Cloudera Manager: https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/
CDH下载地址：https://archive.cloudera.com/cdh6/6.0.1/parcels/
即：

https://archive.cloudera.com/cdh6/6.0.1/parcels/

需要下载的内容是：

cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpmcloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpmcloudera-manager-server-6.0.1-610811.el7.x86_64.rpmjdkoracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm

• 1
• 2
• 3
• 4
• 5
• 6

将以上安装文件可以放到/root目录下

关于安装：

Master节点（cdh1）：yum -y install cloudera-manager-agent cloudera-manager-daemons cloudera-manager-serverNode节点(cdh2,cdh3)：yum -y install cloudera-manager-agent cloudera-manager-daemons本地安装：（网络好的话，可以忽略此步骤。）cloudera-manager-agent-6.0.1-610811.el7.x8664.rpmcloudera-manager-daemons-6.0.1-610811.el7.x8664.rpmcloudera-manager-server-6.0.1-610811.el7.x8664.rpm先安装 cloudera-manager-daemons-6.0.1-610811.el7.x8664.rpm 再安装其他的。

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12

先安装下面的内容，因为cloudera-manager-agent依赖这些

yum install bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi portmap httpd mod_ssl openssl-devel python-psycopg2 MySQL-python /lib/lsb/init-functions

• 1

cdh1上执行以下命令：

rpm -ivh cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpmrpm -ivh cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpmrpm -ivh cloudera-manager-server-6.0.1-610811.el7.x86_64.rpmrpm -ivh cloudera-manager-server-db-2-6.0.1-610811.el7.x86_64.rpm

• 1
• 2
• 3
• 4

cdh2,cdh3上执行以下命令：

rpm -ivh cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpmrpm -ivh cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm

• 1
• 2

4.3.3.2 启用Auto-TLS以自动创建证书

JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera /opt/cloudera/cm-agent/bin/certmanager --location /opt/cloudera/CMCA setup --configure-services

• 1

4.3.3.3 安装mysql

参考：https://blog.****.net/Sky_WANGJ/article/details/84030458
参考：http://www.cnblogs.com/piperck/p/9944469.html

wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpmyum -y install mysql-community-release-el7-5.noarch.rpmyum -y updateyum -y install mysql-server*.Move old InnoDB log files /var/lib/mysql/iblogfile0 and /var/lib/mysql/iblogfile1 out of /var/lib/mysql/ to a backup location.rm -rf /var/lib/mysql/ib_logfile0 /var/lib/mysql/ib_logfile1启动mysqlsystemctl enable mysqldsystemctl start mysqld初始化Mysql/usr/bin/mysql_secure_installation[...]Enter current password for root (enter for none):OK, successfully used password, moving on...[...]Set root password? [Y/n] YNew password:rootRe-enter new password:rootRemove anonymous users? [Y/n] Y[...]Disallow root login remotely? [Y/n] N[...]Remove test database and access to it [Y/n] Y[...]Reload privilege tables now? [Y/n] YAll done!

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30

4.3.3.4 安装MySQL JDBC驱动程序

wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gztar zxvf mysql-connector-java-5.1.46.tar.gz#要注意的是后面的名字必须是mysql-connector-java.jar（可以将下面的jar包放到3台机器上）cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

• 1
• 2
• 3
• 4
• 5
• 6

4.3.3.5 必须配置数据库支持MySQL UTF8字符集编码

mysql -u root -p rootDROP DATABASE scm;DROP DATABASE amon;DROP DATABASE rman;DROP DATABASE hue;DROP DATABASE metastore;DROP DATABASE sentry;DROP DATABASE nav;DROP DATABASE oozie;DROP DATABASE activity_monitor;CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;GRANT ALL ON hive.* TO 'hive'@'%' IDENTIFIED BY 'root';create database scm default character set utf8 default collate utf8_general_ci;grant all on scm.* to 'scm'@'%' identified by 'root';create database amon default character set utf8 default collate utf8_general_ci;grant all on amon.* to 'amon'@'%' identified by 'root';create database rman default character set utf8 default collate utf8_general_ci;grant all on rman.* to 'rman'@'%' identified by 'root';create database hue default character set utf8 default collate utf8_general_ci;grant all on hue.* to 'hue'@'%' identified by 'root';create database metastore default character set utf8 default collate utf8_general_ci;grant all on metastore.* to 'hive'@'%' identified by 'root';create database sentry default character set utf8 default collate utf8_general_ci;grant all on sentry.* to 'sentry'@'%' identified by 'root';create database nav default character set utf8 default collate utf8_general_ci;grant all on nav.* to 'nav'@'%' identified by 'root';create database oozie default character set utf8 default collate utf8_general_ci;grant all on oozie.* to 'oozie'@'%' identified by 'root';CREATE DATABASE activity_monitor DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;GRANT ALL ON activity_monitor.* TO 'activity_monitor'@'%' IDENTIFIED BY 'root';flush privileges;

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30
• 31
• 32
• 33
• 34
• 35

笔者的如下：

创建完成之后查看数据库：

禁用透明大页面压缩(cdh1,cdh2,cdh3上都执行)

echo never > /sys/kernel/mm/transparent_hugepage/defragecho never > /sys/kernel/mm/transparent_hugepage/enabled并将上面的两条命令写入开机自启动vim /etc/rc.local

• 1
• 2
• 3
• 4
• 5

优化交换分区

vim /etc/sysctl.confvm.swappiness = 10sysctl -p /etc/sysctl.conf

• 1
• 2
• 3
• 4
• 5

其它

echo never>/sys/kernel/mm/transparent_hugepage/defragecho never>/sys/kernel/mm/transparent_hugepage/enabled

• 1
• 2

4.3.3.6 设置Cloudera Manager数据库

mysql在本地时执行：rm -rf /etc/cloudera-scm-server/db.mgmt.properties/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm如果不在一台机器上，执行类似如下命令（cdh2为mysql所在位置，cdh1为cloudera manager server所在位置）：[[email protected] cloudera-scm-server]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h 192.168.106.151 --scm-host cdh1 scm scm[[email protected] cloudera-scm-server]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h 192.168.106.151 --scm-host cdh1 scm scmEnter SCM password: JAVA_HOME=/usr/java/jdk1.8.0_141-clouderaVerifying that we can write to /etc/cloudera-scm-serverCreating SCM configuration file in /etc/cloudera-scm-serverExecuting:  /usr/java/jdk1.8.0_141-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.[                          main] DbCommandExecutor              INFO  Successfully connected to database.All done, your SCM database is configured correctly![[email protected] cloudera-scm-server]#

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17

上面过程中需要输入scm的密码，在上面的数据库创建语句中scm的密码被设置成了root，所以在此处也同样设置scm的密码为root

启动Cloudera Manager Server

将CHD6相关的Parcel包放到主节点的/opt/cloudera/parcel-repo/目录中，如果没有此目录，可以自己创建。（如果是在线安装，那么此步骤可以忽略。）注意：最后将CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha256，重命名为CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha，这点必须注意，否则系统会重新下载CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel。存放了CDH和修改了HASH之后，设置Cloudera Manager数据库，启动Cloudera Manager Server，然后登录Cloudera Manager Admin Console：启动Cloudera Manager Server：systemctl start cloudera-scm-server可以查看启动日志tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log看到此条信息，说明启动完成INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16

要注意的是，在上传完上面的CDH之后，要在manifest.json 中找到CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel ，然后找到上面的hash值，即：

将2e650f1f1ea020a3efc98a231b85c2df1a50b030 替换掉CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha 中的内容

登录

在Web浏览器中，转到 http://cdh1:7180/cmf/login，其中域名是运行Cloudera Manager Server的主机的FQDN或IP地址。如果有启用自动TLS，则会重定向到https: cdh1:7183，并显示安全警告。你可能需要表明你信任该证书，或单击以继续执行Cloudera Manager Server主机。登录Cloudera Manager Admin Console。默认凭据为：Username: adminPassword: admin

• 1
• 2
• 3
• 4
• 5
• 6
• 7

进入之后

点击第一个免费的即可

如果出现下面的错误：


网络上说的解决办法，参考：https://yq.aliyun.com/articles/578871。然后笔者使用了之后并不成功。最后定位到是上面各章节的配置没有配置好导致的，需要验证：网络是否通，各机器之间是否能够ssh免密，hostname,hosts等是否设置的正确，网络防火墙是否按照博文叙述进行配置。（为此，笔者重新虚拟过3遍3个集群，熬了3个到凌晨2，3点的夜晚）。所以在配置过程中一定要细心又细心。

最后：

5 卸载Cloudera Manager

systemctl stop cloudera-scm-serversystemctl stop cloudera-scm-agentyum -y remove 'cloudera-manager-*'yum clean allumount cm_processesumount /var/run/cloudera-scm-agent/processrm -Rf /usr/share/cmf /var/lib/cloudera* /var/cache/yum/cloudera* /var/log/cloudera* /var/run/cloudera*rm -rf /tmp/.scmpreparenode.lockrm -Rf /var/lib/flume-ng /var/lib/hadoop* /var/lib/hue /var/lib/navigator /var/lib/oozie /var/lib/solr /var/lib/sqoop* /var/lib/zookeeperrm -Rf datadrivepath/dfs datadrivepath/mapred datadrivepath/yarnrm -rf /var/lib/hadoop-* /var/lib/impala /var/lib/solr /var/lib/zookeeper /var/lib/hue /var/lib/oozie  /var/lib/pgsql  /var/lib/sqoop2  /data/dfs/  /data/impala/ /data/yarn/  /dfs/ /impala/ /yarn/  /var/run/hadoop-*/ /var/run/hdfs-*/ /usr/bin/hadoop* /usr/bin/zookeeper* /usr/bin/hbase* /usr/bin/hive* /usr/bin/hdfs /usr/bin/mapred /usr/bin/yarn /usr/bin/sqoop* /usr/bin/oozie /etc/hadoop* /etc/zookeeper* /etc/hive* /etc/hue /etc/impala /etc/sqoop* /etc/oozie /etc/hbase* /etc/hcatalogsystemctl stop mysqldyum -y remove mysqlrm -rf /var/lib/mysqlrm -rf /var/log/mysqld.logrm -rf /var/lib/mysql/mysqlrm -rf /usr/lib64/mysqlrm -rf /usr/share/mysqlrm -rf /opt/clouderarpm -qa | grep -i mysql卸载MySQL相关的文件。

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30
• 31
• 32
• 33

6 Cloudera Manager重启，停止等涉及到的相关问题

发现如果是通过rpm -ivh *.rpm方式安装的软件，在机器重启之后，CM和CDH自动重启了。

如果发现没有自动重启或为了保险起见，启动Cloudera Manager和cdh的方式如下：

以下，第一行命令在cdh1上执行。第二行在所有的节点上执行systemctl stop cloudera-scm-serversystemctl stop cloudera-scm-agent(cdh1上执行)systemctl start cloudera-scm-server (cdh1,cdh2,cdh3上都执行)systemctl start cloudera-scm-agent  [[email protected] init.d]# ps -ef | grep cloudera-scm-server  （在CDH上执行）查看CM的日志：[[email protected] init.d]# tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13

查看日志，然后一直等待到其启动完毕，大约要等待3~5分钟左右，然后在浏览器中输入：http://cdh1:7180/cmf/login