PyCharm中通过pyspark调用spark报错的解决办法
-
问题:PyCharm中通过pyspark无法调起spark
2019-10-16 20:39:09,343 | Dummy-1:22492 | django.db.backends:90 | utils:execute | DEBUG | (0.000) SELECT @@SQL_AUTO_IS_NULL; args=None
2019-10-16 20:39:09,344 | Dummy-1:22492 | django.db.backends:90 | utils:execute | DEBUG | (0.000) SELECT VERSION(); args=None
System check identified no issues (0 silenced).
2019-10-16 20:39:09,923 | Dummy-1:22492 | django.db.backends:90 | utils:execute | DEBUG | (0.003) SHOW FULL TABLES; args=None
2019-10-16 20:39:09,927 | Dummy-1:22492 | django.db.backends:90 | utils:execute | DEBUG | (0.002) SELECTdjango_migrations
.app
,django_migrations
.name
FROMdjango_migrations
; args=()
October 16, 2019 - 20:39:09
Django version 1.10.1, using settings ‘mysite.settings’
Starting development server at http://0.0.0.0:8001/
Quit the server with CTRL-BREAK.
2019-10-16 20:39:22,058 | Thread-4:15288 | django.db.backends:90 | utils:execute | DEBUG | (0.000) SELECT @@SQL_AUTO_IS_NULL; args=None
2019-10-16 20:39:22,058 | Thread-4:15288 | django.db.backends:90 | utils:execute | DEBUG | (0.000) SELECT VERSION(); args=None
2019-10-16 20:39:22,070 | Thread-4:15288 | django.db.backends:90 | utils:execute | DEBUG | (0.000) SELECTdjango_session
.session_key
,django_session
.session_data
,django_session
.expire_date
FROMdjango_session
WHERE (django_session
.session_key
= ‘k4micifggqpl2xjc4wapt12wwq0j12b3’ ANDdjango_session
.expire_date
> ‘2019-10-16 20:39:22’); args=(‘k4micifggqpl2xjc4wapt12wwq0j12b3’, ‘2019-10-16 20:39:22’)
2019-10-16 20:39:22,073 | Thread-4:15288 | django.db.backends:90 | utils:execute | DEBUG | (0.001) SELECTauth_user
.id
,auth_user
.password
,auth_user
.last_login
,auth_user
.is_superuser
,auth_user
.username
,auth_user
.first_name
,auth_user
.last_name
,auth_user
.email
,auth_user
.is_staff
,auth_user
.is_active
,auth_user
.date_joined
FROMauth_user
WHEREauth_user
.id
= 1; args=(1,)
Using Spark’s default log4j profile: org/apache/spark/log4j-defaults.properties
19/10/16 20:39:22 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[main,5,main]
java.util.NoSuchElementException: key not found: _PYSPARK_DRIVER_CALLBACK_HOST
at scala.collection.MapLikeclass.apply(MapLike.scala:141)
at scala.collection.AbstractMap.apply(Map.scala:58)
at org.apache.spark.api.python.PythonGatewayServerKaTeX parse error: Can't use function '$' in math mode at position 8: anonfun$̲main$1.apply$mc…runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
19/10/16 20:39:22 INFO ShutdownHookManager: Shutdown hook called
2019-10-16 20:39:23,355 | Thread-4:15288 | django.server:131 | basehttp:log_message | ERROR | “GET /cif/cifmas_echart_0002/ HTTP/1.1” 500 102233 -
解决方法:在pycharm中增加变量,指向 spark目录即可。