StreamSet之数据流从JDBC到Hbase
环境准备:
5.5.56-MariaDB
HDP-2.6.3.0-Hbase-1.1.2
zookeeper-3.4.10
- 点击创建管道流
- 随便写一个有意义的title和描述
- 3.击创建好的管道流的title进入管理页面,然后选择All Stages,在下面输入jdbc
- 4.首先配置JDBC
注意:sql一定要加上as,as “hbase列族:hbase列名” ,${OFFSET}和Initial Offset的值一样,
下边用户认证和增量模式打上勾,sql quer 要遵循以下格式:
SELECT * FROM table WHERE id > ${OFFSET} ORDER BY id,要不然会不成功。
数据库用户名和密码
- 配置Hbase
下面3个都勾选
Ignore Missing Field:忽略空字段
Implicit field mapping:自动匹配字段映射,勾上这个我们就不用配置字段映射了(fields)
Ignore Invalid Column:忽略空行
- 启动
注意:jdbc语句要和Hbase字段对应:
SELECT id as ID,
id as "f1:id",
name as "f1:name",
idcard as "f1:idcard",
sex as "f1:sex",
created_date as "f1:created_date",
last_modified_date as "f1:last_modified_date",
password as "f1:password",
username as "f1:username",
created_by_id as "f1:created_by_id",
last_modified_by_id as "f1:last_modified_by_id"
FROM user where id > ${OFFSET} ORDER BY id;