阿帕奇星火 - java.lang.NoSuchMethodError:breeze.linalg.Vector $ .scalarOf()Lbreeze/linalg /支持/ ScalarOf
问题描述:
以下是错误:
Exception in thread "main" java.lang.NoSuchMethodError: breeze.linalg.Vector$.scalarOf()Lbreeze/linalg/support/ScalarOf;
at org.apache.spark.ml.knn.Leaf$$anonfun$4.apply(MetricTree.scala:95)
at org.apache.spark.ml.knn.Leaf$$anonfun$4.apply(MetricTree.scala:93)
at scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:57)
at scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:66)
at scala.collection.mutable.ArrayBuffer.foldLeft(ArrayBuffer.scala:48)
at org.apache.spark.ml.knn.Leaf$.apply(MetricTree.scala:93)
at org.apache.spark.ml.knn.MetricTree$.build(MetricTree.scala:169)
at org.apache.spark.ml.knn.KNN.fit(KNN.scala:388)
at org.apache.spark.ml.classification.KNNClassifier.train(KNNClassifier.scala:109)
at org.apache.spark.ml.classification.KNNClassifier.fit(KNNClassifier.scala:117)
at SparkKNN$.main(SparkKNN.scala:23)
at SparkKNN.main(SparkKNN.scala)
这里是触发错误的程序:
object SparkKNN {
def main(args: Array[String]) {
val spark = SparkSession.builder().master("local").config("spark.sql.warehouse.dir", "file:///c:/tmp/spark-warehouse").getOrCreate()
val sc = spark.sparkContext
import spark.implicits._
//read in raw label and features
val training = spark.read.format("com.databricks.spark.csv").option("header", true).load("E:/Machine Learning/knn_input.csv")
var df = training.selectExpr("cast(label as double) label", "cast(feature1 as int) feature1","cast(feature2 as int) feature2","cast(feature3 as int) feature3")
val assembler = new VectorAssembler().setInputCols(Array("feature1","feature2","feature3")).setOutputCol("features")
df = assembler.transform(df)
//MLUtils.loadLibSVMFile(sc, "C:/Program Files (x86)/spark-2.0.0-bin-hadoop2.7/data/mllib/sample_libsvm_data.txt").toDF()
val knn = new KNNClassifier()
.setTopTreeSize(df.count().toInt/2)
.setK(10)
val splits = df.randomSplit(Array(0.7, 0.3))
val (trainingData, testData) = (splits(0), splits(1))
val knnModel = knn.fit(trainingData)
val predicted = knnModel.transform(testData)
predicted.show()
}
}
我使用Apache Spark 2.0和Scala版本2.11.8。它看起来像一个版本差异问题。有任何想法吗?
答
星火MLLib 2.0带来了这个版本的微风:
"org.scalanlp" % "breeze_2.11" % "0.11.2"
您必须在您的类路径也有微风,但不同版本的依赖另一个库,这是一个被加载。因此,MLLib在运行时使用不同版本的Breeze,而不是在编译时。
您有多个选项。你可以在Breeze上发现不希望的传递依赖并排除它。您可以添加一个直接依赖于具有与MLLib相同的Breeze依赖项的库的版本。或者您可以添加对Breeze MLLib需求的直接依赖。
RE:火花2.1 MLIB取决于微风0.12 https://github.com/apache/spark/blob/v2.1.0/pom.xml#L655-L671 –
的OP是使用2.0。你是否说升级Spark会解决问题? – Vidya