Pyspark,编程初始化spark:IllegalArgumentException:缺少应用程序资源
问题描述:
在Python中创建Spark上下文时,出现以下错误。Pyspark,编程初始化spark:IllegalArgumentException:缺少应用程序资源
app_name="my_app"
master="local[*]"
sc = SparkContext(appName=app_name, master=master)
Exception in thread "main" java.lang.IllegalArgumentException: Missing application resource.
at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitArgs(SparkSubmitCommandBuilder.java:160)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:276)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:151)
at org.apache.spark.launcher.Main.main(Main.java:86)
....
pyspark.zip/pyspark/java_gateway.py", line 94, in launch_gateway
raise Exception("Java gateway process exited before sending the driver its port number")
Exception: Java gateway process exited before sending the driver its port number
火花发射器似乎失败了。
答
这是由于先前存在的env变量发生冲突。我在Python程序中删除了它们,现在它可以顺利运行。
例如:
import os
#check if pyspark env vars are set and then reset to required or delete.
del os.environ['PYSPARK_SUBMIT_ARGS']
哇,这工作!你是怎么弄出来的?有更多信息的参考/网址? – javadba
谢谢。没有找到任何参考,需要一些时间才能发现在〜/ .bash_profile中导出的env变量与python脚本冲突。 –