如何使这些函数定义 D.R.Y.？答案

【问题标题】：How do I make these function definitions D.R.Y.?如何使这些函数定义 D.R.Y.？
【发布时间】：2020-10-14 02:43:08
【问题描述】：

我有多个函数定义来编写 parquet 文件，但我宁愿有一个通用函数。如何使一项通用功能起作用？问题是一个通用函数返回错误
could not find implicit value for parameter writerFactory: com.github.mjakubowski84.parquet4s.ParquetWriter.ParquetWriterFactory[Product]

多种功能：

import com.github.mjakubowski84.parquet4s.ParquetWriter
import org.apache.parquet.hadoop.metadata.CompressionCodecName
import org.apache.parquet.hadoop.ParquetFileWriter
import java.time.format.DateTimeFormatter
import java.time.LocalDate

def writeParquetUnsegmented(rows: List[ACaseClass], date: String) = {
    val outPath = s"s3a://redacted/path-$date.parquet"
    ParquetWriter.writeAndClose(outPath,
                                rows,
                                    ParquetWriter.Options(
                                                        writeMode = ParquetFileWriter.Mode.OVERWRITE,
                                                        compressionCodecName = CompressionCodecName.SNAPPY
                                                        ))
}

def writeParquetSegmented(rows: List[BCaseClass], date: String) = {
    val outPath = s"s3a://redacted/path-$date.parquet"
    ParquetWriter.writeAndClose(outPath,
                                rows,
                                    ParquetWriter.Options(
                                                        writeMode = ParquetFileWriter.Mode.OVERWRITE,
                                                        compressionCodecName = CompressionCodecName.SNAPPY
                                                        ))
}

一个功能：

def writeParquetSegmented(rows: List[Product], date: String) = {
    val outPath = s"s3a://redacted/path-$date.parquet"
    ParquetWriter.writeAndClose(outPath,
                                rows,
                                    ParquetWriter.Options(
                                                        writeMode = ParquetFileWriter.Mode.OVERWRITE,
                                                        compressionCodecName = CompressionCodecName.SNAPPY
                                                        ))
}

我也尝试了不同的函数签名，得到了同样的错误。

def writeParquetSegmented[A](rows: List[A], date: String)

def writeParquetSegmented[A :< RowParent](rows: List[A], date: String)

【问题讨论】：

标签： scala amazon-web-services hadoop parquet

【解决方案1】：

我会试试这个签名：

def writeParquetSegmented[A : ParquetWriterFactory](rows: List[A], date: String)

编译器会将context bound [A : ParquetWriterFactory] 转换为错误消息所抱怨的隐式参数。

def writeParquetSegmented[A](rows: List[A], date: String)(implicit pwf: ParquetWriterFactory)

【讨论】：

谢谢。没想到加(implicit pwf: ParquetWriterFactory[A])

【解决方案2】：

只需将函数设为通用：

def writeParquetUnsegmented[A](
  rows: List[A],
  date: String)(
  implicit writerFactory: ParquetWriterFactory[A]) = {
    val outPath = s"s3a://redacted/path-$date.parquet"
    ParquetWriter.writeAndClose(
      outPath,
      rows,
      ParquetWriter.Options(
        writeMode = ParquetFileWriter.Mode.OVERWRITE,
        compressionCodecName = CompressionCodecName.SNAPPY))
}

【讨论】：