无法获取火花案例类的输出

Question

我正在尝试实现

使用 spark 2.4.8 和使用 intellij 的 sbt 版本 1.4.3

代码：

val sqlContext = new org.apache.spark.sql.SQLContext(sc) 导入 sqlContext.implicits._

case class Person(id:Int,Name:String,cityId:Long)
case class City(id:Long,Name:String)

val family=Seq(Person(1,"john",11),(2,"MAR",12),(3,"Iweta",10)).toDF
val cities=Seq(City(11,"boston"),(12,"dallas")).toDF


error:
Exception in thread "main" java.lang.NoClassDefFoundError: no Java class corresponding to Product with Serializable found
    at scala.reflect.runtime.JavaMirrors$JavaMirror.typeToJavaClass(JavaMirrors.scala:1300)
    at scala.reflect.runtime.JavaMirrors$JavaMirror.runtimeClass(JavaMirrors.scala:192)
    at scala.reflect.runtime.JavaMirrors$JavaMirror.runtimeClass(JavaMirrors.scala:54)
    at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:60)
    at org.apache.spark.sql.Encoders$.product(Encoders.scala:275)
    at org.apache.spark.sql.LowPrioritySQLImplicits$class.newProductEncoder(SQLImplicits.scala:248)
    at org.apache.spark.sql.SQLImplicits.newProductEncoder(SQLImplicits.scala:34)
    at usingcaseclass$.main(usingcaseclass.scala:26)
    at usingcaseclass.main(usingcaseclass.scala)

  case class Salary(depName: String, empNo: Long, salary: Long)
val empsalary = Seq(Salary("sales", 1, 5000), Salary("personnel", 2, 3900)).toDS
empsalary.show(false)

value toDS is not a member of Seq[Salary]
val empsalary = Seq(Salary("sales", 1, 5000), Salary("personnel", 2, 3900)).toDS

知道如何防止此错误

Answer 1

您以错误的方式定义了 Seq，这将导致 Seq[Product with Serializable] 而不是 Seq[T] 工作的 Seq[T]。

以下修改后的行应该适合你。

val family=Seq(Person(1,"john",11),Person(2,"MAR",12),Person(3,"Iweta",10))

family.toDF().show()

+---+-----+------+
| id| Name|cityId|
+---+-----+------+
|  1| john|    11|
|  2|  MAR|    12|
|  3|Iweta|    10|
+---+-----+------+