错误使用SPARK时:java.lang.NoClassDefFoundError:调度/ HTTP
问题描述:
使用时总是火花提交此错误发生:错误使用SPARK时:java.lang.NoClassDefFoundError:调度/ HTTP
ERROR Executor: Exception in task 0.0 in stage 137.0 (TID 35)
java.lang.NoClassDefFoundError: dispatch/Http$
虽然我有进口的lib(我使用的Scala):
import dispatch._, Defaults._
的代码部分:
def publishToNsq(data: String) = {
val topic = "post_similar_items_to_elasticsearch"
val host = "192.168.100.160:4151"
def myRequest = url(s"$host/pub?topic=$topic")
def myRequestAsJson = myRequest.setContentType("application/json", "UTF-8")
def myPostWithBody = myRequestAsJson << data
val response = Http(myPostWithBody OK as.String)
println(response)
}
这是我build.sbt的一部分:
scalaVersion := "2.10.5"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1"
libraryDependencies += "org.apache.spark" %% "spark-mllib" % "1.6.1"
libraryDependencies += "org.json4s" %% "json4s-native" % "3.4.0"
libraryDependencies += "org.json4s" %% "json4s-jackson" % "3.4.0"
libraryDependencies += "org.scalaj" %% "scalaj-http" % "1.1.4"
libraryDependencies += "net.databinder.dispatch" %% "dispatch-core" % "0.11.2"
resolvers += Resolver.mavenLocal
任何解决方案?
答
的依赖是不是在罐子里,它会被解决,如果你配置为在java.lang.NoClassDefFoundError: org/apache/spark/streaming/twitter/TwitterUtils
+0
谢谢,实际上它在使用sbt程序集时有效 –
那是因为你需要组装您的应用程序创建一个超级罐子。 – eliasah