小Q大数据-hadoop环境搭建
系统环境Centos 7
一、服务器环境配置
1、服务器列表
名称 | 地址 | 操作系统 | root密码 |
master | x.x.x.1 | Centos 7 | root |
slave1 | x.x.x.2 | Centos 7 | root |
slave2 | x.x.x.3 | Centos 7 | root |
2、配置ssh信任
1）在主节点生成公私秘钥对：
进入/.ssh/目录下，
ssh-****** -t rsa -P ''
cat .ssh/*.pub > .ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
scp -r .ssh [email protected]:/root/
scp -r [email protected]:/root/
3、更改hosts
在master 机器上执行如下命令:
vi /etc/hosts
x.x.x.1 master.bigdata.com master
x.x.x.2 slave1.bigdata.com slave1
x.x.x.3 slave2.bigdata.com slave2
执行以下命令：
scp /etc/hosts [email protected]:/etc/hosts
scp /etc/[email protected]:/etc/hosts
4、配置防火墙
iptables –F
systemctldisable firewalld
servicefirewalld stop
setenforce 0
5、配置阿里yum源（可选）
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak
wget -O /etc/yum.repos.d/CentOS-Base.repohttp://mirrors.aliyun.com/repo/Centos-7.repo
yum makecache
6、安装mysql数据库
在mysql服务安装在了master节点，也可选择其他节点或机器。具体步骤如下：
|
wget http://repo.mysql.com/mysql57-community-release-el7-8.noarch.rpm rpm -ivh mysql57-community-release-el7-8.noarch.rpm yum install mysql-server [[email protected] ~]# service mysqld start [[email protected] ~]# grep "password" /var/log/mysqld.log [[email protected] ~]# mysql -u root -p Enter password: mysql> set password = password("[email protected]#"); mysql> grant all privileges on *.* to [email protected]"%" identified by "[email protected]#" ; mysql> flush privileges; |
7、时钟同步
|
yum install -y ntp systemctl enable ntpd |
8、关闭SELinux、配置umask值
|
setenforce 0 echo umask 0022 >> /etc/profile |
9、安装mysql-java驱动
yum install mysql-connector-java.noarch
二、配置本地yum库
1、服务安装配置
|
yum install yum-utils createrepo yum install httpd service httpd start |
2、下载镜像包，解压到指定目录
操作系统 |
HDP/Ambari 版本 | 库名称 | 下载链接 |
CentOS 7 | 2.6.4.0 | HDP | http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0/HDP-2.6.4.0-centos7-rpm.tar.gz |
HDP-UTILS | http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz | ||
HDP-GPL | http://public-repo-1.hortonworks.com/HDP-GPL/centos7/2.x/updates/2.6.4.0/HDP-GPL-2.6.4.0-centos7-rpm.tar.gz | ||
CentOS 7 | 2.6.1.5 | Ambari | http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.5/ambari-2.6.1.5-centos7.tar.gz |
下载后把镜像包解压缩，并且移动到/var/www/html/hdp 目录下；
tar -zxvf ambari-2.6.1.5-centos7.tar.gz |
完成后目录结构如下：
三、安装配置ambari Server
1、配置ambari 的yum源（在master节点）
|
wget -nv http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.6.1.5/ ambari.repo -O /etc/yum.repos.d/ambari.repo |
2、修改ambari.repo配置信息
3、执行ambari server安装
yum install ambari-server |
4、配置ambari server
|
[[email protected] yum.repos.d]# ambari-server setup Checking SELinux... SELinux status is 'enabled' SELinux mode is 'permissive' WARNING: SELinux is set to 'permissive' mode and temporarily disabled. OK to continue [y/n] (y)? y Customize user account for ambari-server daemon [y/n] (n)? Adjusting ambari-server permissions and ownership... Checking firewall status... Checking JDK... [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7 [3] Custom JDK ============================================================================== Enter choice (1): 1 ………… Successfully downloaded JDK distribution to /var/lib/ambari-server/resources/jdk-8u112-linux-x64.tar.gz Installing JDK to /usr/jdk64/ Successfully installed JDK to /usr/jdk64/ Successfully downloaded JCE Policy archive to /var/lib/ambari-server/resources/jce_policy-8.zip Installing JCE policy... Checking GPL software agreement... GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? Completing setup... Configuring database... Enter advanced database configuration [y/n] (n)? y Configuring database... ============================================================================== Choose one of the following options: [1] - PostgreSQL (Embedded) [2] - Oracle [3] - MySQL / MariaDB [4] - PostgreSQL [5] - Microsoft SQL Server (Tech Preview) [6] - SQL Anywhere [7] - BDB ============================================================================== Enter choice (1): 3 Hostname (localhost): master.bigdata.com Port (3306): Database name (ambari): Username (ambari): Enter Database Password (bigdata): BigData123_ Configuring ambari database... Configuring remote database connection properties... WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql (此处转下表的数据库操作) Proceed with configuring remote database connection properties [y/n] (y)? y Extracting system views... ambari-admin-2.6.1.5.3.jar Ambari Server 'setup' completed successfully. |
5、数据库操作
|
[[email protected] ~]# mysql -u root -p Enter password: mysql> create database ambari; mysql> grant all privileges on ambari.* to [email protected]"%" identified by "BigData123_"; mysql> use ambari; Database changed mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql Query OK, 0 rows affected (0.00 sec) |
6、服务启停
启动：ambari-server start
停止：ambari-server stop
启动成功后可以通过浏览器访问管理界面。
URL: x.x.x.1:8080
用户名：admin
密码：admin
四、HDP部署
1、登录ambari server管理界面
在浏览器打开http://x.x.x.1:8080
输入用户名密码 默认admin/admin
2、进入集群安装向导
3、输入集群名称
4、自定义hdp版本（可选）
在HDP/centos7/2.6.4.0-91/HDP-2.6.4.0-91.xml文件基础上，修改baseurl。
选择增加版本、上传修改后的版本文件。
增加版本后，直接下一步
5、选择版本
6、确认主机列表（并上传id_rsa文件）
7、主机注册、主机环境检查
8、选择安装部署组件
9、分配服务器角色
10、设置slave节点和客户端
11、配置hive组件
选择已有的mysql数据库，填入相关的信息（用户名、密码、数据库地址）
数据库操作
|
[[email protected] ~]# mysql -u root -p Enter password: mysql> create database hive; mysql> grant all privileges on hive.* to [email protected]"%" identified by "BigData123_"; mysql> flush privileges; |
Ambari-server设置
|
[[email protected] bin]# rpm -qa |grep mysql-connector mysql-connector-java-5.1.25-3.el7.noarch [[email protected] bin]# rpm -ql mysql-connector-java-5.1.25-3.el7.noarch …… /usr/share/java/mysql-connector-java.jar …… [[email protected] java]# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar Using python /usr/bin/python Setup ambari-server Copying /usr/share/java/mysql-connector-java.jar to /var/lib/ambari-server/resources If you are updating existing jdbc driver jar for mysql with mysql-connector-java.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts. JDBC driver was successfully initialized. Ambari Server 'setup' completed successfully. |
测试数据库连接
12、设置Ambari Metrics的admin密码
13、设置smartsence的admin密码
14、确认安装信息，点击下一步
15、执行安装
16、安装成功，点击完成进入图形界面