Spark2与Oozie整合
背景:
纯粹个人瞎摸索,相关资料太少了,估计大多数公司还没使用spark2吧?
1.在oozie共享目录创建spark2目录
可直接在页面上创建
2.将集群中spark2的jar复制到spark2目录下
hadoop fs -put / /opt/cloudera/parcels/SPARK2-2.2.0.cloudera2-1.cdh5.12.0.p0.232957/lib/spark2/jars /user/oozie/share/lib/lib_20180523180130/spark2
hadoop fs -cp /user/oozie/share/lib/lib_20180523180130/spark/oozie-sharelib-spark-4.1.0-cdh5.14.0.jar /user/oozie/share/lib/lib_20180523180130/spark2
hadoop fs -cp /user/oozie/share/lib/lib_20180523180130/spark/oozie-sharelib-spark.jar /user/oozie/share/lib/lib_20180523180130/spark2
其目录结构如下:
4.创建oozie的spark工作流
进入oozie计划工作页面
上图中的Jar/py名称框里 只需选中spark-examples_2.11-2.2.0.cloudera2.jar 即可
选中/user/hue/oozie/workspaces/hue-oozie-1527557010.76/lib/spark-examples_2.11-2.2.0.cloudera2.jar 点击选择此文件夹
选完之后自己检查一下看看是不是全选了
图中的例子是官方的例子
hadoop fs -put /opt/cloudera/parcels/SPARK2/lib/spark2/examples/jars/spark-examples_2.11-2.2.0.cloudera2.jar /user/hue/oozie/workspaces/hue-oozie-1527557010.76/lib
Main class :org.apache.spark.examples.SparkPi
点击上图的齿轮设置属性
重点来了:
点击上图中的 齿轮
设置spark2 : name=oozie.action.sharelib.for.spark
value=spark2
填写完之后保存提交即可
注意:应该注意一些权限问题
这些可能会用上
sudo -u hdfs hadoop fs -chown -R oozie:oozie /user/oozie/share/lib/lib_20180528183927/spark2
sudo -u hdfs hadoop fs -chown -R hue:hue /user/hue/oozie/workspaces
sudo -u hdfs hadoop fs -chmod 777 /user/hue/oozie/workspaces
参考:https://blog.****.net/qq_24908345/article/details/80017660
https://cloud.tencent.com/developer/article/1078092