您的位置: 首页  >  文章  >  CDH6.3.2+Phoenix5

CDH6.3.2+Phoenix5

分类: 文章 • 2022-10-04 11:01:24

官方网站

https://docs.cloudera.com/documentation/enterprise/6/6.3.html

https://archive.cloudera.com/phoenix/

https://archive.cloudera.com/cdh6/

https://archive.cloudera.com/cm6/

需下载文件：
CDH6.3.2+Phoenix5

提示：该方案是在离线的情况下安装

目录

前言

提示：CDH收购HDP后进行了相关的优化，现在CDH变得更加主流

一、Hadoop的3种发行版：

Apache Hadoop
Cloudera’s Distribution Including Apache Hadoop（CDH）
Hortonworks Data Platform (HDP)

二、操作步骤

1.修改ip

ip addr 查看第二个 （ifcfg-eth0）

vi /etc/sysconfig/network-scripts/ifcfg-eth0(不确定)

BOOTPROTO=static 静态ip
ONBOOT=yes
IPADDR=172.20.182.242 ip地址
NETMASK=255.255.255.0 子网掩码
GATEWAY=172.20.182.1 网关

重启服务 service network restart

2.修改hosts文件

vi /ect/hosts

10.80.4.88 cdh-tpf001
10.80.4.89 cdh-tpf002
10.80.4.90 cdh-tpf003
10.80.4.91 cdh-tpf004
10.80.4.92 cdh-tpf005
10.80.4.93 cdh-tpf006

3.关闭防火墙（所有机器）

systemctl stop firewalld
systemctl disable firewalld

4.关闭selinux（所有机器）

vi /etc/selinux/config
将SELINUX=enforcing改为SELINUX=disabled

设置后需要重启才能生效

4.安装相关服务

yum install -y httpd //安装cdh需要使用 （主服务器）
yum install -y perl //安装cdh需要使用（所有服务器）

如果安装失败就挂载本地yum源

mkdir /home/iso

mount -t iso9660 -o loop /mysoft/CentOS-7-x86_64-DVD-1804.iso /home/iso

cd /etc/yum.repos.d/

注意要把没有用到的.repo中每个文件的备份一下

mv CentOS-Base.repo CentOS-Base.repo.bak

cp CentOS-CR.repo CentOS-Local.repo

vi CentOS-Local.repo（编辑文件）

设置enabled=0
baseurl=file:///home/iso
#启动yum源： 1-启用 0-不启用
enabled=1
#安全检测： 1-开启 0-不开启
gpgcheck=1
gpgkey=file:///home/iso/RPM-GPG-KEY-CentOS-7

更新缓存

yum clean all
yum update
yum makecache

5.安装jdk

mkdir /usr/java

tar -zxvf jdk-8u45-linux-x64.gz -C /usr/java/

#切记必须修正所属用户及用户组
chown -R root:root /usr/java/jdk1.8.0_45

vi /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_45
export PATH=${JAVA_HOME}/bin:${PATH}
source /etc/profile

6.安装mysql

1.解压
tar -zxvf mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz -C /usr/local
cd /usr/local/
mv mysql-5.7.11-linux-glibc2.5-x86_64 mysql
mkdir mysql/arch mysql/data mysql/tmp
mv /etc/my.cnf /etc/my.cnf.bak

2.创建my.cnf

vi /etc/my.cnf

[client]
port = 3306
socket = /usr/local/mysql/data/mysql.sock
default-character-set=utf8mb4

[mysqld]
port = 3306
socket = /usr/local/mysql/data/mysql.sock

skip-slave-start

skip-external-locking
key_buffer_size = 256M
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 4M
query_cache_size= 32M
max_allowed_packet = 16M
myisam_sort_buffer_size=128M
tmp_table_size=32M

table_open_cache = 512
thread_cache_size = 8
wait_timeout = 86400
interactive_timeout = 86400
max_connections = 600
#Try number of CPU’s*2 for thread_concurrency
#thread_concurrency = 32

#isolation level and default engine
default-storage-engine = INNODB
transaction-isolation = READ-COMMITTED

server-id = 1739
basedir = /usr/local/mysql
datadir = /usr/local/mysql/data
pid-file = /usr/local/mysql/data/hostname.pid

#open performance schema
log-warnings
sysdate-is-now

binlog_format = ROW
log_bin_trust_function_creators=1
log-error = /usr/local/mysql/data/hostname.err
log-bin = /usr/local/mysql/arch/mysql-bin
expire_logs_days = 7

innodb_write_io_threads=16

relay-log = /usr/local/mysql/relay_log/relay-log
relay-log-index = /usr/local/mysql/relay_log/relay-log.index
relay_log_info_file= /usr/local/mysql/relay_log/relay-log.info

log_slave_updates=1
gtid_mode=OFF
enforce_gtid_consistency=OFF
#slave
slave-parallel-type=LOGICAL_CLOCK
slave-parallel-workers=4
master_info_repository=TABLE
relay_log_info_repository=TABLE
relay_log_recovery=ON

#other logs
#general_log =1
#general_log_file = /usr/local/mysql/data/general_log.err
#slow_query_log=1
#slow_query_log_file=/usr/local/mysql/data/slow_log.err

#for replication slave
sync_binlog = 500

#for innodb options
innodb_data_home_dir = /usr/local/mysql/data/
innodb_data_file_path = ibdata1:1G;ibdata2:1G:autoextend

innodb_log_group_home_dir = /usr/local/mysql/arch
innodb_log_files_in_group = 4
innodb_log_file_size = 1G
innodb_log_buffer_size = 200M

#根据生产需要，调整pool size
innodb_buffer_pool_size = 2G
#innodb_additional_mem_pool_size = 50M #deprecated in 5.6
tmpdir = /usr/local/mysql/tmp

innodb_lock_wait_timeout = 1000
#innodb_thread_concurrency = 0
innodb_flush_log_at_trx_commit = 2

innodb_locks_unsafe_for_binlog=1

#innodb io features: add for mysql5.5.8
performance_schema
innodb_read_io_threads=4
innodb-write-io-threads=4
innodb-io-capacity=200
#purge threads change default(0) to 1 for purge
innodb_purge_threads=1
innodb_use_native_aio=on

#case-sensitive file names and separate tablespace
innodb_file_per_table = 1
lower_case_table_names=1

[mysqldump]
quick
max_allowed_packet = 128M

[mysql]
no-auto-rehash
default-character-set=utf8mb4

[mysqlhotcopy]
interactive-timeout

[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

3：创建用户
groupadd -g 101 dba
useradd -u 514 -g dba -G root -d /usr/local/mysql mysql

4.copy 环境变量配置文件至mysql用户的home目录中,为了以下步骤配置个人环境变量
cp /etc/skel/.* /usr/local/mysql ###important (切换用户su - mysql) 出现的-bash-$问题的解决

5.配置环境变量
vi /usr/local/mysql/.bashrc
配置：
export MYSQL_BASE=/usr/local/mysql
export PATH= M Y S Q L B A S E / b i n : {MYSQL_BASE}/bin: MYSQLB​ASE/bin:PATH

6.赋权限和用户组，安装
[[email protected] local]# chown mysql:dba /etc/my.cnf //设置所有者
[[email protected] local]# chmod 640 /etc/my.cnf //设置文件的权限（6：所有者拥有读写，同组拥有读，其他没有权限）
[[email protected] local]# chown -R mysql:dba /usr/local/mysql
[[email protected] local]# chmod -R 775 /usr/local/mysql

7.配置服务及开机自启动
[[email protected] local]# cd /usr/local/mysql
#将服务文件拷贝到init.d下，并重命名为mysql
[[email protected] mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysql
#赋予可执行权限
[[email protected] mysql]# chmod +x /etc/rc.d/init.d/mysql
#添加服务
[[email protected] mysql]# chkconfig --add mysql

8.初始化
bin/mysqld
–defaults-file=/etc/my.cnf
–user=mysql
–basedir=/usr/local/mysql/
–datadir=/usr/local/mysql/data/
–initialize

9.查看临时密码（在data目录下面）
cd /usr/local/mysql/data
cat hostname.err |grep password
出现：
[Note] A temporary password is generated for [email protected]: ebBL3#OJu-C

10.启动：
su - mysql//切换用户
service mysql start //启动

11.登录及修改用户密码
/usr/local/mysql/data:>mysql -uroot -p
输入上面查看的密码
登陆出现：ERROR 1045 (28000): Access denied for user ‘root’@‘localhost’ (using password: YES)
在my.cnf中的[mysqld]后面任意一行添加“skip-grant-tables”用来跳过密码验证的过程
service mysqld restart

然后直接输入mysql就能进入 执行 flush privileges;
修改密码
mysql> alter user [email protected] identified by ‘haiyisoft123$’;
quit退出，然后修改my.cnf文件，去掉skip-grant-tables

mysql> alter user [email protected] identified by ‘haiyisoft123$’;

mysql> GRANT ALL PRIVILEGES ON . TO ‘root’@’%’ IDENTIFIED BY ‘haiyisoft123$’ ;

//cm启动需要存储的元数据
create database cmf DEFAULT CHARACTER SET utf8;
//cdh启动需要存储的元数据
create database amon DEFAULT CHARACTER SET utf8;
grant all on cmf.* TO ‘cmf’@’%’ IDENTIFIED BY ‘cmf’;
grant all on amon.* TO ‘amon’@’%’ IDENTIFIED BY ‘amon’;

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
mysql> quit;

12:部署mysql驱动(/etc/cloudera-scm-agent/config.ini)
mkdir -p /usr/share/java/
cp /mysoft/mysql-connector-java-5.1.47.jar /usr/share/java/
cd /usr/share/java/
mv mysql-connector-java-5.1.47.jar mysql-connector-java.jar

7.rpm包部署CM

主节点：
cd /mysoft
tar -zxvf cm6.3.1-redhat7.tar.gz
cd /mysoft/cm6.3.1/RPMS/x86_64

rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force

rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm --nodeps --force

rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force

cd /etc/cloudera-scm-server/

vi db.properties
修改内容：
com.cloudera.cmf.db.host=hadoop0001:3306
com.cloudera.cmf.db.name=cmf
com.cloudera.cmf.db.user=cmf
com.cloudera.cmf.db.password=cmf
com.cloudera.cmf.db.setupType=EXTERNAL

启动cm

service cloudera-scm-server start

/var/log/cloudera-scm-server/cloudera-scm-server.log

service cloudera-scm-server status

cat /var/log/cloudera-scm-server/cloudera-scm-server.log

tail -F /var/log/cloudera-scm-server/cloudera-scm-server.log 查看启动过程

启动完成后：
http://172.20.182.242:7180/cmf/login (此时不用操作任何东西进行如下操作可以免去免秘登陆)

从节点
cd /mysoft/cm6.3.1/RPMS/x86_64

rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force

rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force

vi /etc/cloudera-scm-agent/config.ini

service cloudera-scm-agent start(需要安装组件的地方都有执行)

8.配置cdh pracel

mkdir /var/www/html/cdh6_parcel

cp CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel /var/www/html/cdh6_parcel

cp CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 /var/www/html/cdh6_parcel

cp manifest.json /var/www/html/cdh6_parcel/

cd /var/www/html/cdh6_parcel/

mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha

systemctl start httpd.service #启动

http://10.80.4.88/cdh6_parcel/ 查看一下是否httpd正常

8.配置phoenix pracel

mkdir /var/www/html/cdh6_phoenix

cp PHOENIX-5.0.0-cdh6.2.0.p0.1308267-el7.parcel /var/www/html/cdh6_phoenix/

cp PHOENIX-5.0.0-cdh6.2.0.p0.1308267-el7.parcel.sha /var/www/html/cdh6_phoenix/

cp manifest-phoenix.json /var/www/html/cdh6_phoenix/

cp PHOENIX-1.0.jar /opt/cloudera/csd

cd /var/www/html/cdh6_phoenix

mv manifest-phoenix.json manifest.json

9.安装CDH

总结

提示：这里对文章进行总结：
例如：以上就是今天要讲的内容，本文仅仅简单介绍了pandas的使用，而pandas提供了大量能使我们快速便捷地处理数据的函数和方法。

相关推荐