Hive安装部署
1.布局
| hadoop-fwl02 | hadoop-fwl04 | |
|---|---|---|
| mysql | 1 | |
| hive | 1 |
2 前提
2.1 运行hadoop集群
启动hadoop集群，需要在启动hadoop集群的情况下才能安装hive。
2.2 安装mySql
安装Hive之前，需要安装一个关系型数据库，用来存储表的信息。
因为无法连接互联网，所以采用RPM包安装。
2.2.1 下载mySql
官网地址：https://dev.mysql.com/downloads/mysql/5.7.html#downloads
在官网上，根据linux系统版本，选择下图中画红线的部分，最后进行下载。得到 mysql-5.7.25-1.el6.x86_64.rpm-bundle.tar ，解压后得到
最后将下图的rpm包上传到linux服务器
### 2.2.2 mysql具体安装步骤
选择合适的服务器安装mysql，本次布局中，mysql安装在hadoop-fwl04上。
2.2.2.1 查看系统已经安装的mysql
输入以下代码查看linux是否已经预装了mysql
[[email protected] lib]$ rpm -qa | grep mysql
mysql-libs-5.1.66-2.el6_3.x86_64
[[email protected] lib]$
如果如上面的效果，显示了 mysql-libs-5.1.66-2.el6_3.x86_64 ，则代表需要卸载mysql-libs-5.1.66-2.el6_3.x86_6，重新安装。
2.2.2.2 yum删除mysql
输入以下代码，使用功能yum方式删除mysql
[[email protected] lib]$ yum -y remove mysql-libs-5.1.66-2.el6.x86_64
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Remove Process
Resolving Dependencies
--> Running transaction check
---> Package mysql-libs.x86_64 0:5.1.71-1.el6 will be erased
--> Processing Dependency: libmysqlclient.so.16()(64bit) for package: 2:postfix-2.6.6-2.2.el6_1.x86_64
--> Processing Dependency: libmysqlclient.so.16(libmysqlclient_16)(64bit) for package: 2:postfix-2.6.6-2.2.el6_1.x86_64
--> Processing Dependency: mysql-libs for package: 2:postfix-2.6.6-2.2.el6_1.x86_64
--> Running transaction check
---> Package postfix.x86_64 2:2.6.6-2.2.el6_1 will be erased
--> Processing Dependency: /usr/sbin/sendmail for package: redhat-lsb-core-4.0-7.el6.centos.x86_64
--> Processing Dependency: /usr/sbin/sendmail for package: cronie-1.4.4-12.el6.x86_64
--> Running transaction check
---> Package cronie.x86_64 0:1.4.4-12.el6 will be erased
--> Processing Dependency: cronie = 1.4.4-12.el6 for package: cronie-anacron-1.4.4-12.el6.x86_64
---> Package redhat-lsb-core.x86_64 0:4.0-7.el6.centos will be erased
--> Processing Dependency: redhat-lsb-core(x86-64) = 4.0 for package: redhat-lsb-printing-4.0-7.el6.centos.x86_64
--> Processing Dependency: redhat-lsb-core(x86-64) = 4.0-7.el6.centos for package: redhat-lsb-4.0-7.el6.centos.x86_64
--> Processing Dependency: redhat-lsb-core(x86-64) = 4.0 for package: redhat-lsb-graphics-4.0-7.el6.centos.x86_64
--> Running transaction check
---> Package cronie-anacron.x86_64 0:1.4.4-12.el6 will be erased
---> Package redhat-lsb.x86_64 0:4.0-7.el6.centos will be erased
--> Processing Dependency: redhat-lsb(x86-64) = 4.0-7.el6.centos for package: redhat-lsb-compat-4.0-7.el6.centos.x86_64
---> Package redhat-lsb-graphics.x86_64 0:4.0-7.el6.centos will be erased
---> Package redhat-lsb-printing.x86_64 0:4.0-7.el6.centos will be erased
--> Running transaction check
---> Package redhat-lsb-compat.x86_64 0:4.0-7.el6.centos will be erased
--> Processing Dependency: /etc/cron.d for package: crontabs-1.10-33.el6.noarch
--> Processing Dependency: /etc/cron.d for package: sysstat-9.0.4-22.el6.x86_64
--> Restarting Dependency Resolution with new changes.
--> Running transaction check
---> Package crontabs.noarch 0:1.10-33.el6 will be erased
---> Package sysstat.x86_64 0:9.0.4-22.el6 will be erased
--> Finished Dependency Resolution
base | 3.7 kB 00:00
extras | 3.4 kB 00:00
updates | 3.4 kB 00:00
Dependencies Resolved
======================================================================================================================================
Package Arch Version Repository Size
======================================================================================================================================
Removing:
mysql-libs x86_64 5.1.71-1.el6 @anaconda-CentOS-201311272149.x86_64/6.5 4.0 M
Removing for dependencies:
cronie x86_64 1.4.4-12.el6 @anaconda-CentOS-201311272149.x86_64/6.5 174 k
cronie-anacron x86_64 1.4.4-12.el6 @anaconda-CentOS-201311272149.x86_64/6.5 43 k
crontabs noarch 1.10-33.el6 @anaconda-CentOS-201311272149.x86_64/6.5 2.4 k
postfix x86_64 2:2.6.6-2.2.el6_1 @anaconda-CentOS-201311272149.x86_64/6.5 9.7 M
redhat-lsb x86_64 4.0-7.el6.centos @anaconda-CentOS-201311272149.x86_64/6.5 0.0
redhat-lsb-compat x86_64 4.0-7.el6.centos @anaconda-CentOS-201311272149.x86_64/6.5 0.0
redhat-lsb-core x86_64 4.0-7.el6.centos @anaconda-CentOS-201311272149.x86_64/6.5 22 k
redhat-lsb-graphics x86_64 4.0-7.el6.centos @anaconda-CentOS-201311272149.x86_64/6.5 0.0
redhat-lsb-printing x86_64 4.0-7.el6.centos @anaconda-CentOS-201311272149.x86_64/6.5 0.0
sysstat x86_64 9.0.4-22.el6 @anaconda-CentOS-201311272149.x86_64/6.5 824 k
Transaction Summary
======================================================================================================================================
Remove 11 Package(s)
Installed size: 15 M
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
Erasing : redhat-lsb-compat-4.0-7.el6.centos.x86_64 1/11
Erasing : redhat-lsb-4.0-7.el6.centos.x86_64 2/11
Erasing : redhat-lsb-graphics-4.0-7.el6.centos.x86_64 3/11
Erasing : redhat-lsb-printing-4.0-7.el6.centos.x86_64 4/11
Erasing : redhat-lsb-core-4.0-7.el6.centos.x86_64 5/11
Erasing : sysstat-9.0.4-22.el6.x86_64 6/11
/var/tmp/rpm-tmp.3nhGx8: line 1: lsb_release: command not found
Erasing : crontabs-1.10-33.el6.noarch 7/11
Erasing : cronie-anacron-1.4.4-12.el6.x86_64 8/11
Erasing : cronie-1.4.4-12.el6.x86_64 9/11
Erasing : 2:postfix-2.6.6-2.2.el6_1.x86_64 10/11
Erasing : mysql-libs-5.1.71-1.el6.x86_64 11/11
Verifying : redhat-lsb-printing-4.0-7.el6.centos.x86_64 1/11
Verifying : cronie-anacron-1.4.4-12.el6.x86_64 2/11
Verifying : 2:postfix-2.6.6-2.2.el6_1.x86_64 3/11
Verifying : redhat-lsb-4.0-7.el6.centos.x86_64 4/11
Verifying : redhat-lsb-core-4.0-7.el6.centos.x86_64 5/11
Verifying : mysql-libs-5.1.71-1.el6.x86_64 6/11
Verifying : crontabs-1.10-33.el6.noarch 7/11
Verifying : redhat-lsb-compat-4.0-7.el6.centos.x86_64 8/11
Verifying : sysstat-9.0.4-22.el6.x86_64 9/11
Verifying : cronie-1.4.4-12.el6.x86_64 10/11
Verifying : redhat-lsb-graphics-4.0-7.el6.centos.x86_64 11/11
Removed:
mysql-libs.x86_64 0:5.1.71-1.el6
Dependency Removed:
cronie.x86_64 0:1.4.4-12.el6 cronie-anacron.x86_64 0:1.4.4-12.el6
crontabs.noarch 0:1.10-33.el6 postfix.x86_64 2:2.6.6-2.2.el6_1
redhat-lsb.x86_64 0:4.0-7.el6.centos redhat-lsb-compat.x86_64 0:4.0-7.el6.centos
redhat-lsb-core.x86_64 0:4.0-7.el6.centos redhat-lsb-graphics.x86_64 0:4.0-7.el6.centos
redhat-lsb-printing.x86_64 0:4.0-7.el6.centos sysstat.x86_64 0:9.0.4-22.el6
Complete!
[[email protected] lib]$
注意：如果使用 rpm -ef mysql-libs-5.1.66-2.el6.x86_6，会出现以下情况，无法删除mysql-libs-5.1.66-2.el6.x86_6；
[[email protected] lib]$ rpm -ef mysql-libs-5.1.66-2.el6.x86_64
error: Failed dependencies:
libmysqlclient.so.16()(64bit) is needed by (installed) postfix-2:2.6.6-2.2.el6_1.x86_64
libmysqlclient.so.16(libmysqlclient_16)(64bit) is needed by (installed) postfix-2:2.6.6-2.2.el6_1.x86_64
mysql-libs is needed by (installed) postfix-2:2.6.6-2.2.el6_1.x86_64
[[email protected] lib]$
2.2.2.3 安装mysql组件
因为存在依赖关系，所以需要按顺序安装以下四个.rpm。
mysql-community-common-5.7.25-1.el6.x86_64.rpm
mysql-community-libs-5.7.25-1.el6.x86_64.rpm
mysql-community-client-5.7.25-1.el6.x86_64.rpm
mysql-community-server-5.7.25-1.el6.x86_64.rpm
依次执行以下四个安装命令：
[[email protected] lib]$ rpm -ivh mysql-community-common-5.7.25-1.el6.x86_64.rpm
[[email protected] lib]$ rpm -ivh mysql-community-libs-5.7.25-1.el6.x86_64.rpm
[[email protected] lib]$ rpm -ivh mysql-community-client-5.7.25-1.el6.x86_64.rpm
[[email protected] lib]$ rpm -ivh mysql-community-server-5.7.25-1.el6.x86_64.rpm
2.2.2.4 启动mysql
# 查看mysql是否启动
[[email protected] lib]$ service mysqld status
# 启动mysql
[[email protected] lib]$ service mysqld start
# 停止mysql
[[email protected] lib]$ service mysqld stop
# 重启mysql
[[email protected] lib]$ service mysqld restart
#设置mysql开机启动
[[email protected] lib]$ chkconfig mysqld on
2.2.2.5 登录mysql
mysql安装完成之后我们是没有设置密码的，但是mysql为我们设置了一个临时的密码，需要查看mysql的日志知道这个临时密码，我们才可以登录mysql。
#查看临时密码
[[email protected] lib]$ grep password /var/log/mysqld.log
2017-09-01T16:43:10.889769Z 1 [Note] A temporary password is generated for [email protected]: hcLMTxbOh2?w
这样我们得知临时密码是：hcLMTxbOh2?w。
#登录mysql，之后需要输入密码
[[email protected] lib]$ mysql -p
# 设置新密码
mysql> set password = password("123");
# 退出当前登录
mysql> quit;
#用新密码再次登录mysql,mysql -u用户名 -p密码
[[email protected] lib]$ mysql -uroot -p123
2.2.2.6 设置mysql用户
需要对用户的host进行设置，以便集群中的其它节点可以访问mysql
#显示databases
mysql> show databases;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id: 129
Current database: *** NONE ***
+--------------------+
| Database |
+--------------------+
| information_schema |
| hive_remote |
| mysql |
| performance_schema |
| sys |
+--------------------+
5 rows in set (0.90 sec)
#使用上面的mysql数据库
mysql> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql>
#显示mysql中的table
mysql> show tables;
+---------------------------+
| Tables_in_mysql |
+---------------------------+
| columns_priv |
| db |
| engine_cost |
| event |
| func |
| general_log |
| gtid_executed |
| help_category |
| help_keyword |
| help_relation |
| help_topic |
| innodb_index_stats |
| innodb_table_stats |
| ndb_binlog_index |
| plugin |
| proc |
| procs_priv |
| proxies_priv |
| server_cost |
| servers |
| slave_master_info |
| slave_relay_log_info |
| slave_worker_info |
| slow_log |
| tables_priv |
| time_zone |
| time_zone_leap_second |
| time_zone_name |
| time_zone_transition |
| time_zone_transition_type |
| user |
+---------------------------+
31 rows in set (0.07 sec)
mysql>
#显示user表
mysql> desc user;
+------------------------+-----------------------------------+------+-----+-----------------------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------------------+-----------------------------------+------+-----+-----------------------+-------+
| Host | char(60) | NO | PRI | | |
| User | char(32) | NO | PRI | | |
| Select_priv | enum('N','Y') | NO | | N | |
| Insert_priv | enum('N','Y') | NO | | N | |
| Update_priv | enum('N','Y') | NO | | N | |
| Delete_priv | enum('N','Y') | NO | | N | |
| Create_priv | enum('N','Y') | NO | | N | |
| Drop_priv | enum('N','Y') | NO | | N | |
| Reload_priv | enum('N','Y') | NO | | N | |
| Shutdown_priv | enum('N','Y') | NO | | N | |
| Process_priv | enum('N','Y') | NO | | N | |
| File_priv | enum('N','Y') | NO | | N | |
| Grant_priv | enum('N','Y') | NO | | N | |
| References_priv | enum('N','Y') | NO | | N | |
| Index_priv | enum('N','Y') | NO | | N | |
| Alter_priv | enum('N','Y') | NO | | N | |
| Show_db_priv | enum('N','Y') | NO | | N | |
| Super_priv | enum('N','Y') | NO | | N | |
| Create_tmp_table_priv | enum('N','Y') | NO | | N | |
| Lock_tables_priv | enum('N','Y') | NO | | N | |
| Execute_priv | enum('N','Y') | NO | | N | |
| Repl_slave_priv | enum('N','Y') | NO | | N | |
| Repl_client_priv | enum('N','Y') | NO | | N | |
| Create_view_priv | enum('N','Y') | NO | | N | |
| Show_view_priv | enum('N','Y') | NO | | N | |
| Create_routine_priv | enum('N','Y') | NO | | N | |
| Alter_routine_priv | enum('N','Y') | NO | | N | |
| Create_user_priv | enum('N','Y') | NO | | N | |
| Event_priv | enum('N','Y') | NO | | N | |
| Trigger_priv | enum('N','Y') | NO | | N | |
| Create_tablespace_priv | enum('N','Y') | NO | | N | |
| ssl_type | enum('','ANY','X509','SPECIFIED') | NO | | | |
| ssl_cipher | blob | NO | | NULL | |
| x509_issuer | blob | NO | | NULL | |
| x509_subject | blob | NO | | NULL | |
| max_questions | int(11) unsigned | NO | | 0 | |
| max_updates | int(11) unsigned | NO | | 0 | |
| max_connections | int(11) unsigned | NO | | 0 | |
| max_user_connections | int(11) unsigned | NO | | 0 | |
| plugin | char(64) | NO | | mysql_native_password | |
| authentication_string | text | YES | | NULL | |
| password_expired | enum('N','Y') | NO | | N | |
| password_last_changed | timestamp | YES | | NULL | |
| password_lifetime | smallint(5) unsigned | YES | | NULL | |
| account_locked | enum('N','Y') | NO | | N | |
+------------------------+-----------------------------------+------+-----+-----------------------+-------+
45 rows in set (0.13 sec)
mysql>
#显示user表中的User，和Host字段
mysql> select user,host from user;
+------+------+
| user | host |
+------+------+
| root |localhost |
+------+------+
1 row in set (0.12 sec)
mysql>
#赋予权限
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123' WITH GRANT OPTION;
+------+------+
| user | host |
+------+------+
| root |localhost |
| root | % |
+------+------+
1 row in set (0.12 sec)
mysql>
#防止权限冲突，删除其余的字段
mysql> delete from user where host !='%';
mysql>
#刷新
mysql> flush privileges;
#退出mysql，并重新登录
mysql> quit;
[[email protected] lib]$ mysql -uroot -p123
mysql>
3 Hive的安装
官网地址：http://hive.apache.org/downloads.html
3.1 Hive的下载
直接进入下载地址：http://mirror.bit.edu.cn/apache/hive/
或者
1.进入官网http://hive.apache.org/downloads.html下载
3.2 installing Hive
3.2.1 浏览官方文档
3.2.2 installing Hive的具体步骤
3.2.2.1 解压 apache-hive-1.2.1-bin.tar.gz
#将解压到apache-hive-1.2.1-bin.tar.gz /usr/local下
[[email protected] tempdown]$ tar -zxvf apache-hive-1.2.1-bin.tar.gz -C /usr/local/
#更改文件夹的组、用户我的用户名和群主都为 hadoop
[[email protected] local]$ sudo chown -R hadoop:hadoop apache-hive-1.2.1-bin
#开放全部权限
[[email protected] local]$ sudo chmod 777 apache-hive-1.2.1-bin
#重命名
[[email protected] local]$ mv apache-hive-1.2.1-bin apache-hive-1.2.1
3.2.2.2 配置环境变量
[[email protected] local]$ sudo vim /etc/profile
export HIVE_HOME=/usr/local/apache-hive-1.2.1
export PATH=$PATH:$HIVE_HOME/bin
[[email protected] local]$ source /etc.profile
3.2.2.3 配置hive-site.xml
[[email protected] local]$ cd apache-hive-1.2.1/conf
#创建hive-site.xml
[hado[email protected] conf]$ cp hive-default.xml.template hive-site.xml
插入hive-site.xml的具体内容（参考网址：https://cwiki.apache.org/confluence/display/Hive/AdminManual+Metastore+Administration）
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
#hive在hdfs的存储位置
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive_remote/warehouse</value>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
</property>
#mysql所在节点
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://主机名/hive_remote?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
#数据库用户名
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>用户名</value>
</property>
#数据库密码
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>密码</value>
</property>
</configuration>
最后结果如下图所示：
[[email protected] conf]$ vim hive-site.xml
3.2.2.4 复制mysql的驱动程序到hive/lib下面
[[email protected] conf]$ cp /usr/local/tempdown/mysql-connector-java-5.1.32-bin.jar /usr/local/apache-hive-1.2.1/lib
3.2.3 测试Hive
3.2.3.1 Hive建立table
#建表
hive> CREATE TABLE person(
> id INT,
> name STRING,
> age INT,
> likes ARRAY<STRING>,
> address MAP<STRING,STRING>
> )
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ','
> COLLECTION ITEMS TERMINATED BY '-'
> MAP KEYS TERMINATED BY ':'
> LINES TERMINATED BY '\n';
OK
Time taken: 13.821 seconds
hive>
3.2.3.2 查看HDFS
查看HDFS的情况：person表创建成功。
3.2.3.3 查看mysql
通过Navicat 8查看mysql的情况，进入hive_remote下TBLS查看结果：person表创建成功。
遇到的问题
针对hive启动时，出现[ERROR] Terminal initialization failed; falling back to unsupported
参考：https://blog.****.net/zhc794429974/article/details/8082416
1、是因为hive/lib下的 jline.jar 和 Hadoop 目录下 /usr/local/src/hadoop-2.6.1/share/hadoop/yarn/lib 版本冲突。
将 /usr/local/src/hadoop-2.6.1/share/hadoop/yarn/lib/jline-0.9.94.jar删除。