Scala-Spark：将 Dataframe 转换为 RDD [Edge]

Question

我有一个表示图形边缘的数据框；这是架构：

root |-- src: string (nullable = true) 
     |-- dst: string (nullable = true) 
     |-- relationship: struct (nullable = false) 
     | |-- business_id: string (nullable = true) 
     | |-- normalized_influence: double (nullable = true)

我想将其转换为 RDD[Edge] 以使用 Pregel API，我的困难在于属性“关系”。怎么转换呢？

Answer 1

Edge 是一个参数化类。这意味着除了源 ID 和目标 ID 之外，您可以在每个边缘存储您喜欢的任何内容。在您的情况下，它可能是Edge[Relationship]。您可以使用案例类来映射数据框和RDD[Edge[Relationship]]：

import scala.util.hashing.MurmurHash3
case class Relationship(business_id: String, normalized_influence: Double)
case class MyEdge(src: String, dst: String, relationship: Relationship)

val edges: RDD[Edge[Relationship]] = df.as[MyEdge].rdd.map { edge =>
    Edge(
        MurmurHash3.stringHash(edge.src).toLong, // VertexId type is a Long, so we need to hash your string
        MurmurHash3.stringHash(edge.dst).toLong,
        edge.relationship
    )
}