斯卡拉
我试图用星火GraphX,并且遇到什么,我认为“不是一个类型参数的成员”是我如何使用Scala的一个问题。我是Scala和Spark的新手。斯卡拉
创建通过调用我自己的函数图:
val initialGraph: Graph[VertexAttributes, Int] = sim.createGraph
VertexAttributes是我定义的类:
class VertexAttributes(var pages: List[Page], var ads: List[Ad], var step: Long, val inDegree: Int, val outDegree: Int)
extends java.io.Serializable
{
// Define alternative methods to be used as the score
def averageScore() =
{
this.ads.map(_.score).sum/this.ads.length
}
def maxScore() =
{
if(this.ads.length == 0) None else Some(this.ads.map(_.score).max)
}
// Select averageScore as the function to be used
val score = averageScore _
}
一些计算后,我用的是GraphX顶点()函数来得到每个顶点的分数:
val nodeRdd = g.vertices.map(v => if(v._2.score() == 0)(v._1 + ",'0,0,255'") else (v._1 + ",'255,0,0'"))
但是这样不会编译,这个sbt消息是:
value score is not a member of type parameter VertexAttributes
我搜索了这个错误信息,但坦率地说不能跟随对话。任何人都可以请解释错误的原因以及我如何解决它?
谢谢。
P.S.下面是我对createGraph方法的代码:
// Define a class to run the simulation
class Butterflies() extends java.io.Serializable
{
// A boolean flag to enable debug statements
var debug = true
// A boolean flag to read an edgelist file rather than compute the edges
val readEdgelistFile = true;
// Create a graph from a page file and an ad file
def createGraph(): Graph[VertexAttributes, Int] =
{
// Just needed for textFile() method to load an RDD from a textfile
// Cannot use the global Spark context because SparkContext cannot be serialized from master to worker
val sc = new SparkContext
// Parse a text file with the vertex information
val pages = sc.textFile("hdfs://ip-172-31-4-59:9000/user/butterflies/data/1K_nodes.txt")
.map { l =>
val tokens = l.split("\\s+") // split("\\s") will split on whitespace
val id = tokens(0).trim.toLong
val tokenList = tokens.last.split('|').toList
(id, tokenList)
}
println("********** NUMBER OF PAGES: " + pages.count + " **********")
// Parse a text file with the ad information
val ads = sc.textFile("hdfs://ip-172-31-4-59:9000/user/butterflies/data/1K_ads.txt")
.map { l =>
val tokens = l.split("\\s+") // split("\\s") will split on whitespace
val id = tokens(0).trim.toLong
val tokenList = tokens.last.split('|').toList
val next: VertexId = 0
val score = 0
//val vertexId: VertexId = id % 1000
val vertexId: VertexId = id
(vertexId, Ad(id, tokenList, next, score))
}
println("********** NUMBER OF ADS: " + ads.count + " **********")
// Check if we should simply read an edgelist file, or compute the edges from scratch
val edgeGraph =
if (readEdgelistFile)
{
// Create a graph from an edgelist file
GraphLoader.edgeListFile(sc, "hdfs://ip-172-31-4-59:9000/user/butterflies/data/1K_edges.txt")
}
else
{
// Create the edges between similar pages
// Create of list of all possible pairs of pages
// Check if any pair shares at least one token
// We only need the pair id's for the edgelist
val allPairs = pages.cartesian(pages).filter{ case (a, b) => a._1 < b._1 }
val similarPairs = allPairs.filter{ case (page1, page2) => page1._2.intersect(page2._2).length >= 1 }
val idOnly = similarPairs.map{ case (page1, page2) => Edge(page1._1, page2._1, 1)}
println("********** NUMBER OF EDGES: " + idOnly.count + " **********")
// Save the list of edges as a file, to be used instead of recomputing the edges every time
//idOnly.saveAsTextFile("hdfs://ip-172-31-4-59:9000/user/butterflies/data/saved_edges")
// Create a graph from an edge list RDD
Graph.fromEdges[Int, Int](idOnly, 1);
}
// Copy into a graph with nodes that have vertexAttributes
//val attributeGraph: Graph[VertexAttributes, Int] =
val attributeGraph =
edgeGraph.mapVertices{ (id, v) => new VertexAttributes(Nil, Nil, 0, 0, 0) }
// Add the node information into the graph
val nodeGraph = attributeGraph.outerJoinVertices(pages) {
(vertexId, attr, pageTokenList) =>
new VertexAttributes(List(Page(vertexId, pageTokenList.getOrElse(List.empty), 0)),
attr.ads, attr.step, attr.inDegree, attr.outDegree)
}
// Add the node degree information into the graph
val degreeGraph = nodeGraph
.outerJoinVertices(nodeGraph.inDegrees)
{
case (id, attr, inDegree) => new VertexAttributes(attr.pages, attr.ads, attr.step, inDegree.getOrElse(0), attr.outDegree)
}
.outerJoinVertices(nodeGraph.outDegrees)
{
case (id, attr, outDegree) =>
new VertexAttributes(attr.pages, attr.ads, attr.step, attr.inDegree, outDegree.getOrElse(0))
}
// Add the ads to the nodes
val adGraph = degreeGraph.outerJoinVertices(ads)
{
(vertexId, attr, ad) =>
{
if (ad.isEmpty)
{
new VertexAttributes(attr.pages, List.empty, attr.step, attr.inDegree, attr.outDegree)
}
else
{
new VertexAttributes(attr.pages, List(Ad(ad.get.id, ad.get.tokens, ad.get.next, ad.get.score)),
attr.step, attr.inDegree, attr.outDegree)
}
}
}
// Display the graph for debug only
if (debug)
{
println("********** GRAPH **********")
//printVertices(adGraph)
}
// return the generated graph
return adGraph
}
}
VertexAttributes
在你的代码是指一个类型参数,不给VertexAttributes
类。该错误可能在您的createGraph
函数中。例如,它可能是这样的:
class Sim {
def createGraph[VertexAttributes]: Graph[VertexAttributes, Int]
}
或:
class Sim[VertexAttributes] {
def createGraph: Graph[VertexAttributes, Int]
}
在有一个名为VertexAttributes
类型参数两种情况。这是一样的,如果你写:
class Sim[T] {
def createGraph: Graph[T, Int]
}
编译器不知道T
有score
方法(因为它没有)。你不需要那个类型参数。只要写:
class Sim {
def createGraph: Graph[VertexAttributes, Int]
}
现在VertexAttributes
将引用类,而不是当地的类型参数。
感谢您解释类型参数如何无意中蠕动。但是,我检查了我的代码,认为我找到了一个类似于您描述的构造,但仍然收到相同的编译器错误。也许我以不太明显的形式制作了类型参数错误?我已将我的createGraph代码添加到原始问题中。如果你想看看它,我非常感谢你的帮助。谢谢。 –
我没有看到任何代码错误。我会建议删除一堆东西,直到你有几行重现问题。那时你可能会看到自己的问题是什么!至少,如果我有时间的话,我会采取这种做法。祝你好运! –
再次感谢您的帮助。至少现在我有一些想法寻找什么。 –
欢迎来到SO!你能否按照[sscce.org](http://sscce.org/)提供一个简短的,自包含的程序?就目前而言,这个错误非常明显--g.vertices返回一个元组列表(我在这里猜测) - 元组中的第一项是VertexAttribute,第二项是index位置。你可能想做:'v._1.score'而不是'v._2.score'。 –
感谢您的回复。 顶点的签名:val顶点:顶点RDD [VD] 图的签名:类图[VD,ED] 因此,顶点返回VertexAttributes的一个RDD –
这大致适用于我。你能发布一个更完整的例子吗?看起来你正在设置一个类型VertexAttribute,在某个影射你的类的地方。 –