Scala Spark：在使用重载时尝试避免类型擦除[重复]

Question

我对 Scala/Spark 比较陌生

我正在尝试根据类类型将一个函数重载到 DStream 中

def persist(service1DStream: DStream[Service1]): Unit = {...}
def persist(service2DStream: DStream[Service2]): Unit = {...}

我收到编译错误：

persist(_root_.org.apache.spark.streaming.dstream.DStream) is already defined in the scope

似乎是由于类型擦除。 如何让编译器识别DStream[Service1]与DStream[Service2]不同

谢谢

Answer 1

def persist(serviceDStream: DStream[Any]): Unit = serviceDStream match {
case _: DStream[Service1] => println("it is a Service1")
case _: DStream[Service2] => println("it is a Service2")    
case _ => println("who knows")     
}

改进的解决方案 re runtime type erasure 使用 shapeless 更多信息 shapeless-guide:

import org.apache.spark.sql.SparkSession
import org.apache.spark.streaming.dstream.DStream
import shapeless.TypeCase

object Test {

 def main(args: Array[String]): Unit = {

val spark = SparkSession
  .builder
  .getOrCreate()

case class Service1 (a: String)
case class Service2 (a: Int)

val Service1Typed = TypeCase[DStream[Service1]]
val Service2Typed    = TypeCase[DStream[Service2]]

def persist(serviceDStream: DStream[Any]): Unit = serviceDStream match {

  case Service1Typed => println("it is a Service1")
  case Service2Typed => println("it is a Service2")
  case _ => println("who knows")
}

}

}

您也可以使用import scala.reflect.ClassTag 了解更多信息：ClassTag example