Sentry手动安装、使用手册
1 Sentry简介
Apache Sentry 是Cloudera公司发布的一个Hadoop开源组件,截止目前还是Apache的孵化项目,它提供了细粒度级、基于角色的授权以及多租户的管理模式。Sentry当前可以和Hive/Hcatalog、Apache Solr 和Cloudera Impala集成,未来会扩展到其他的Hadoop组件,例如HDFS和HBase.
2 准备
2.1 环境说明
1 Sentry安装采用rpm包的方式.
2 Hadoop版本为hadoop-2.5.0-cdh5.3.3,Hive版本为hive-0.13.1-cdh5.3.3,Sentry版本为sentry-1.4.0-cdh5.3.3
3 Sentry下载地址:
http://archive-primary.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.3.3/RPMS/noarch/
2.1.1虚拟机里Linux系统版本
[[email protected] ranger-0.5.0-usersync]# cat /etc/issue | grep Linux
Red Hat Enterprise Linux Server release 6.5 (Santiago)
[[email protected] native]# java -version
java version "1.7.0_67"
Java(TM) SE RuntimeEnvironment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-BitServer VM (build 24.65-b04, mixed mode)
2.1.3mysql版本
[[email protected] native]# mysql -uroot -proot-e"select version()";
Warning: Using a password onthe command line interface can be insecure.
+-----------+
| version() |
+-----------+
| 5.6.14 |
+-----------+
注:
1 Mysql 驱动为mysql-connector-java-5.1.31-bin.jar
2 改jar被重命名后放置在/usr/share/java/内被其它Ranger插件共享
3 安装
3.1安装Mysql
1)安装mysql相关服务
rpm -ivh MySQL-shared-5.6.14-1.el6.x86_64.rpm
rpm -ivh MySQL-server-5.6.14-1.el6.x86_64.rpm时报如下错误:
file/usr/share/mysql/charsets/macroman.xml from install of MySQL-server-5.6.14-1.el6.x86_64conflicts with file from package mysql-libs-5.1.71-1.el6.x86_64 file/usr/share/mysql/charsets/swe7.xml from install ofMySQL-server-5.6.14-1.el6.x86_64 conflicts with file from packagemysql-libs-5.1.71-1.el6.x86_64
rpm -q mysql-libs-5.1.71-1.el6.x86_64
rpm -e --nodeps mysql-libs-5.1.71-1.el6.x86_64
rpm -ivh MySQL-server-5.6.14-1.el6.x86_64.rpm
ARANDOM PASSWORD HAS BEEN SET FOR THE MySQL root USER !
Youwill find that password in '/root/.mysql_secret'.(生成mysql root密码)
Youmust change that password on your first connect,
noother statement but 'SET PASSWORD' will be accepted.
rpm -ivh MySQL-client-5.6.14-1.el6.x86_64.rpm
service mysql start
#这里密码 来自/root/.mysql_secret
mysql -uroot -p9RNrbk9O
#首次执行会提示修改mysql root密码
SET PASSWORD=PASSWORD('root');
#创建Hive数据库为Hive的元数据库
create database hive;
GRANT all ON hive.* TO [email protected]'%' IDENTIFIED BY 'root';
#创建sentry数据库为sentry元数据库
create database sentry;
CREATE USER sentry IDENTIFIED BY 'sentry';
GRANT all ON sentry.* TO [email protected]'%' IDENTIFIED BY'sentry';
flush privileges;
3.2 安装Hive
1)解压Hive压缩包,并配置环境变量
cd /root
tar –zxvf hive-0.13.1-cdh5.3.3.tar.gz
vi ~/.bash_profile
exportHIVE_HOME=/root/hive-0.13.1-cdh5.3.3
#追加HIVE执行文件路径到PATH内
exportPATH=$PATH:$HIVE_HOME/bin
2) 拷贝Mysql驱动到Hive的lib目录下:
cp /root/mysql-connector-java-5.1.31-bin.jar/root/hive-0.13.1-cdh5.3.3/lib/mysql-connector-java-5.1.31-bin.jar
3)配置Hive的conf.详细的hive-site.xml配置内容见下:
3.3 安装Sentry
1)安装Sentry相关服务
rpm -ivh --nodepssentry-1.4.0+cdh5.3.3+137-1.cdh5.3.3.p0.8.el6.noarch.rpm
rpm -ivh --nodeps sentry-hdfs-plugin-1.4.0+cdh5.3.3+137-1.cdh5.3.3.p0.8.el6.noarch.rpm
rpm -ivh --nodepssentry-store-1.4.0+cdh5.3.3+137-1.cdh5.3.3.p0.8.el6.noarch.rpm
2)替换Sentry内的Hadoop、Hive、Impala、Hbase、Zookeeper、Parquet、Avro等jar包
rm -rf /usr/lib/sentry/lib/hive*.jar
rm -rf /usr/lib/sentry/lib/hadoop*.jar
rm -rf /usr/lib/sentry/lib/zookeeper*.jar
rm -rf /usr/lib/sentry/lib/avro*.jar
rm -rf /usr/lib/sentry/lib/server/hive-beeline.jar
cp ~/SentryLibs/* /usr/lib/sentry/lib/
注:这里的Sentrylibs里的jar包是通过/usr/lib/sentry/lib里的文件名分别在Hadoop、Hive、Hbase等里的安装目录里一一找到的.
mv /root/SentryLibs/hive-beeline.jar/usr/lib/sentry/lib/server/
#拷贝mysql驱动到sentry的lib目录内
cp/root/hive-0.13.1-cdh5.3.3/lib/mysql-connector-java-5.1.31-bin.jar/usr/lib/sentry/lib/
#如果装了Impala,此步可忽略
rpm -ivh bigtop-utils-0.7.0+cdh5.3.3+0-1.cdh5.3.3.p0.8.el6.noarch.rpm
3) 配置Sentry,详细配置见sentry-site.xml内容如下:
4)初始化Sentry元数据
sentry --command schema-tool --conffile/etc/sentry/conf/sentry-site.xml --dbType mysql --initSchema
..........................
No rows affected (0.094 seconds)
No rows affected (0.015 seconds)
No rows affected (0.075 seconds)
1 row affected (0.007 seconds)
Closing: 0: jdbc:mysql://localhost:3306/sentry
Initialization script completed
Sentry schemaTool completed
3.4 安装问题
暂无.
4 配置
Ranger在solr里存储日志,RangerAdmin UI依赖solr组件完成审计日志的查询,所以需要先安装和配置好Solr
注:目前(HDFS-Plugin)的测试日志审计时没选择Solr方式,但还是先配置好Standalone模式的solr.
4.1 Hive集成Sentry配置
1) 拷贝Sentry相关jar包到Hive的lib目录内
cp /usr/lib/sentry/lib/sentry*.jar$HIVE_HOME/lib/
cp /usr/lib/sentry/lib/shiro-core-*.jar$HIVE_HOME/lib/
2) Hive conf内新增Sentry-site.xml,配置内容见下:
3) 修改Hive conf内的hive-site.xml,修改成如下:
4) 验证Sentry的权限控制,详见 5 使用:
4.2 Impala集成Sentry配置
1) Impala的conf里新增sentry配置,/etc/impala/conf/sentry-site.xml内容如下:
2) 修改impala的参数
vi/etc/default/impala
IMPALA_CATALOG_ARGS 参数里追加 -sentry_config=/etc/impala/conf/sentry-site.xml
IMPALA_SERVER_ARGS 参数里追加 -server_name=server1\
-sentry_config=/etc/impala/conf/sentry-site.xml
5 使用
5.1 Hive集成Sentry权限验证
1) 启动sentry服务,hive的metastore和hiveserver2服务
sentry --command service--conffile /etc/sentry/conf/sentry-site.xml
nohup hive --service metastore-hiveconf hive.root.logger=INFO,console > myout1.file 2>&1 &
nohup hiveserver2 -hiveconfhive.root.logger=INFO,console > myout2.file 2>&1 &
#hiveserver2里新增hive和test用户,通过beeline方式访问.
groupadd hive; useradd hive -ghive;passwd hive (type hive123)
groupadd test; useradd test -gtest;passwd test (type test123)
beeline -u "jdbc:hive2://data1:10000"-n hive -p test
2)验证整体思路及场景概述
思路:
1 HIVE用户属于管理员组,服务整个server的权限
2 Test用户创建两个数据库test_only(all权限),test_select_only(仅有select权限)
具体验证:
1 通过show databases和use database验证访问权限
2 通过test用户的use test_only能drop table,create table验证all权限
3 通过test用户的use test_select_only不能drop table验证仅有select权限
#场景一 root用户连入,尝试执行DDL操作.(无权操作)
[[email protected] conf]#beeline-u "jdbc:hive2://data1:10000" -n root
scan complete in 17ms
Connecting to jdbc:hive2://data1:10000
Connected to: Apache Hive (version 0.13.1-cdh5.3.3)
Driver: Hive JDBC (version 0.13.1-cdh5.3.3)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 0.13.1-cdh5.3.3 by Apache Hive
0:jdbc:hive2://data1:10000>create database sensitive;
Error: Error while compiling statement: FAILED:SemanticException No valid privileges
Required privileges for this query: Server=server1->action=*;(state=42000,code=40000)
0: jdbc:hive2://data1:10000> create role admin_role;
Error: Error while processing statement: FAILED: Execution Error, return code 1from org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask.SentryAccessDeniedException: Access denied to root (state=08S01,code=1)
0:jdbc:hive2://data1:10000> !q
Closing: 0: jdbc:hive2://data1:10000
#场景二 hive用户连入,对用户hive、test进行授权.
[[email protected] conf]#beeline-u "jdbc:hive2://data1:10000" -n hive
scan complete in 5ms
Connecting to jdbc:hive2://data1:10000
Connected to: Apache Hive (version 0.13.1-cdh5.3.3)
Driver: Hive JDBC (version 0.13.1-cdh5.3.3)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 0.13.1-cdh5.3.3 by Apache Hive
0:jdbc:hive2://data1:10000>create role admin_role;
No rows affected (1.383 seconds)
GRANT ALL ONSERVER server1 TO ROLE admin_role;
GRANT ROLEadmin_role TO GROUP hive;
create roletest_role;
GRANT ALL ONDATABASE test_only TO ROLE test_role;
GRANT ROLEtest_role TO GROUP test;
GRANT SELECT ONDATABASE test_select_only TO ROLE test_role;
#场景三 test用户连入,对数据库admin_only、test_select_only进行权限验证.
beeline -u"jdbc:hive2://data1:10000" -n test
0:jdbc:hive2://data1:10000> showdatabases;
+----------------+--+
| database_name |
+----------------+--+
| admin_only |
| default |
| test_only |
+----------------+--+
3 rows selected (0.721 seconds)
0:jdbc:hive2://data1:10000>use admin_only;
Error: Error while compiling statement: FAILED:SemanticException No valid privileges
Required privileges for this query:Server=server1->Db=admin_only->Table=*->action=insert;Server=server1->Db=admin_only->Table=*->action=select;(state=42000,code=40000)
0:jdbc:hive2://data1:10000>use test_select_only;
No rows affected (0.313 seconds)
0:jdbc:hive2://data1:10000>show tables;
+--------------+--+
| tab_name |
+--------------+--+
| select_only |
+--------------+--+
1 row selected (0.337 seconds)
0:jdbc:hive2://data1:10000>drop table select_only;
Error: Error while processing statement: FAILED: Execution Error, return code 1from org.apache.hadoop.hive.ql.exec.DDLTask.MetaException(message:hive.metastore.execute.setugi can't be false in nonsecure mode) (state=08S01,code=1)
#这里需要在HIVE服务端添加如下参数
<property>
<name>hive.metastore.execute.setugi</name>
<value>true</value>
</property>
4 rows selected (2.9 seconds)
0:jdbc:hive2://data1:10000>use test_select_only;
Norows affected (0.46 seconds)
0:jdbc:hive2://data1:10000>show tables;
+--------------+--+
| tab_name |
+--------------+--+
| select_only |
+--------------+--+
1 row selected (0.507 seconds)
0:jdbc:hive2://data1:10000>drop table select_only;
Error: Error while compilingstatement: FAILED: SemanticException No valid privileges
Required privileges for this query:Server=server1->Db=test_select_only->Table=select_only->action=*;(state=42000,code=40000)
#场景四 test用户连入,对数据库test_only进行验证,拥有所有权限:可以查看、删除表
0: jdbc:hive2://data1:10000>usetest_only;
No rows affected (0.819 seconds)
0:jdbc:hive2://data1:10000> showtables;
+--------------+--+
| tab_name |
+--------------+--+
| test_itself |
+--------------+--+
1 row selected (0.426 seconds)
0:jdbc:hive2://data1:10000>drop table test_itself;
No rows affected (6.336 seconds)
0: jdbc:hive2://data1:10000> create table test_newone ( ont string);
Norows affected (1.377 seconds)
0:jdbc:hive2://data1:10000>show tables;
+--------------+--+
| tab_name |
+--------------+--+
| test_newone |
+--------------+--+
5.2 Impala集成Sentry权限验证
1) 进入impala-shell,执行权限验证
create role supervisor;
grant all on server to rolesupervisor;
grant role supervisor to groupimpala;
GRANT ALL ON SERVER server1 TOROLE any_operation;
GRANT ROLE any_operation TOGROUP hive;
grant ALL ON database EDA TOtest_role;
grant select ON databasepriselect TO test_role;
2) 切换至test用户测试impala权限:
6 总结
仔细跟踪log,具体问题具体分析