R - 在函数中一次插入所有列类型答案

【问题标题】：R - insert all column types at once in a functionR - 在函数中一次插入所有列类型
【发布时间】：2018-01-18 18:02:53
【问题描述】：

如果我想使用任何函数一次将character 类型归因于我的所有列，例如sparklyr 中的spark_read_csv，我会这样做

flights <- spark_read_csv(sc, "flights_spark", 
                          path =  "/path/flights.csv", 
                          memory = TRUE, 
                          columns = list(
                            Year = "character",
                            Month = "character",
                            DayofMonth = "character",
                            DayOfWeek = "character",
                            DepTime = "character",
                            CRSDepTime = "character",
                            ArrTime = "character",
                            CRSArrTime = "character",
                            UniqueCarrier = "character",
                            FlightNum = "character",
                            TailNum = "character",
                            ActualElapsedTime = "character",
                            CRSElapsedTime = "character",
                            AirTime = "character",
                            ArrDelay = "character",
                            DepDelay = "character",
                            Origin = "character",
                            Dest = "character",
                            Distance = "character",
                            TaxiIn = "character",
                            TaxiOut = "character",
                            Cancelled = "character",
                            CancellationCode = "character",
                            Diverted = "character",
                            CarrierDelay = "character",
                            WeatherDelay = "character",
                            NASDelay = "character",
                            SecurityDelay = "character",
                            LateAircraftDelay = "character"), 
                          infer_schema = FALSE)

有没有办法减轻疼痛？

使用来自data.table 的fread 的示例：

iris <- data.table::fread("path/iris", colClasses = c(`Sepal.Length` = "character",
                                          `Sepal.Width` = "character",
                                          `Petal.Length` = "character",
                                          `Petal.Width` = "character",
                                          `Species` = "character",)

【问题讨论】：

标签： r apache-spark sparklyr

【解决方案1】：

由于所有字段都是 character 并且您禁用了架构推断，因此简单的名称列表就足够了：

spark_read_csv(sc,
  "flights_spark", 
  path =  "/path/flights.csv", 
  columns = list("Year", "Month", ..., "LateAircraftDelay")
  infer_schema = FALSE)

虽然没有架构推断，您应该能够完全跳过它，而不会显着降低性能。

spark_read_csv(sc,
  "flights_spark", 
  path =  "/path/flights.csv", 
  infer_schema = FALSE)

在一般情况下（不同类型），命名列表可以解决问题：

names_ <- c("Year", "Month", ..., "LateAircraftDelay")
dtypes <- list("integer", "integer", ..., "string")

spark_read_csv(sc,
  "flights_spark", 
  path =  "/path/flights.csv", 
  columns = setNames(dtypes, names_),
  infer_schema = FALSE)

【讨论】：