在 R 中对大型数据框进行重塑、熔化和铸造

Question

我有一堆数据框，我想对使用包 tidyr、reshape/reshape2 进行一些更改。

Y      C        S      A    B_B_m  B_B_p  C_m  C_p  D_m  D_p 
2000 "AUSTRIA" "total" "no"  33      44   55   66   77   99
2001 "AUSTRIA" "total" "no"  22      11   0    23   24   25
2002 "AUSTRIA" "total" "no"  88      45   56   47   38   39
2003 "AUSTRIA" "total" "no"  90      48   67   67   69   74

应该来

       "C"    "Y"    "S"    "A"      "moment" "B_B" "C"  "D"
    "AUSTRIA" 2000 "total" "no"        "m"     33    55  77
    "AUSTRIA" 2000 "total" "no"        "p"     44    66  99
    "AUSTRIA" 2001 "total" "no"        "m"     22    0   24
    "AUSTRIA" 2001 "total" "no"        "p"     11    23  25
    "AUSTRIA" 2002 "total" "no"        "m"     88    56  38
    "AUSTRIA" 2002 "total" "no"        "p"     45    47  39
    "AUSTRIA" 2003 "total" "no"        "m"     90    67  69
    "AUSTRIA" 2003 "total" "no"        "p"     48    67  74

我使用以下代码来完成此操作：

setwd("C:\\...)
files = list.files(pattern="*.dta") #making a list for the files.
dflist <- list()
    for (i in 1:length(files)){                                  
      dflist[[i]] <- read.dta13(files[i], nonint.factors = TRUE)  
      dflist[[i]] <- melt(dflist[[i]], id=c("C","Y","S","A"))
      dflist[[i]] <- extract(dflist[[i]], variable, c('type', 'moment'), '^(.+)_([^_]+)$')
      dflist[[i]] <- cast(dflist[[i]],...~type)
    }

现在，此代码有效，但不适用于大型数据帧。我的一些数据帧有数百个变量，如果不是数千个变量，并且使用此代码，我会不断耗尽内存或 R 只是崩溃。有什么想法吗？

编辑：

有人对 ff 包发表了评论，但删除了他们的评论。无论如何，我已经对这个包进行了一些研究，但我什至似乎无法将数据帧读入 R...

我试过了：ffdfbig <- read.csv.ffdf(file="dfbig.csv") 但这给了我错误：

`Error in scan(file = file, what = what, sep = sep, quote = quote, dec = dec,  : 
  scan() expected 'an integer', got '"1001"'`

我也尝试过使用colClasses 参数：

sampleData <- read.csv("dfbig.csv", header = TRUE, nrows = 5)
    > classes <- sapply(sampleData, class)
    > ffdfbig <- read.csv.ffdf(file="dfbig.csv",header = TRUE, colClasses=classes)

得到了同样的错误：

Error in scan(file = file, what = what, sep = sep, quote = quote, dec = dec,  : 
  scan() expected 'an integer', got '"1"'

:(

Answer 1

如果您的数据集更大，您可以尝试使用 ff 包。 Here你可以找到一些如何使用它的例子。

另一种选择是使用data.table 包，here 可以找到基本教程。

已编辑

好的，这就是我目前所拥有的。假设您有一个包含您提供的示例数据的 .csv 文件：

Y,C,S,A,B_B_m,B_B_p,C_m,C_p,D_m,D_p
2000,"AUSTRIA","total","no",33,44,55,66,77,99
2001,"AUSTRIA","total","no",22,11,0,23,24,25
2002,"AUSTRIA","total","no",88,45,56,47,38,39
2003,"AUSTRIA","total","no",90,48,67,67,69,74

您可以使用 ff 包读取文件：

library(ff)
library(ffbase)
library(reshape2)

ffdfbig <- read.csv.ffdf(file="/path/to/your/file/dataFile.csv", 
                         colClasses=c("numeric", rep("factor", 3), rep("numeric", 6)), 
                         header = T)

您说您在读取整数时遇到了问题（当它们是从 .csv 中推断出来的），我能够通过显式传递列类来读取文件并生成 ffdf 对象。拥有ffdf 对象后，您可以使用以下方法生成重塑过程的第一部分：

res <- ffdfdply(x=ffdfbig, split=ffdfbig$Y, FUN=function(x){
  df <- reshape(x, 
                v.names = "value", 
                varying = c("B_B_m", "B_B_p", "C_m", "C_p", "D_m", "D_p"),
                timevar = "variable",
                times = c("B_B_m", "B_B_p", "C_m", "C_p", "D_m", "D_p"),
                direction = "long")
  as.data.frame(df)
})

我不知道如何将函数应用到 ffdf 包，但 this 答案给了我钥匙。

上面代码的结果是这样的：

"Y","C","S","A","variable","value","id"
2000,"AUSTRIA","total","no","B_B_m",33,1
2001,"AUSTRIA","total","no","B_B_m",22,2
2002,"AUSTRIA","total","no","B_B_m",88,3
2003,"AUSTRIA","total","no","B_B_m",90,4
2000,"AUSTRIA","total","no","B_B_p",44,1
2001,"AUSTRIA","total","no","B_B_p",11,2
2002,"AUSTRIA","total","no","B_B_p",45,3
2003,"AUSTRIA","total","no","B_B_p",48,4

最后是“m”s和“p”s的“拆分”过程以及向宽格式的转换：

res <- ffdfdply(x=res, split = res$Y, FUN = function(y){
  vars <- c("prefix", "moment")
  df <- extract(y, variable, c('type', 'moment'), '^(.+)_([^_]+)$')
})

res <- ffdfdply(x = res, split = res$Y, FUN = function(x){
  df <- dcast(x, ...~type)
})

res$id <- NULL

如果您想再次将其写入 .csv，您可以使用此函数：

write.csv.ffdf(res, "final.csv")

这将产生以下csv

"","Y","C","S","A","moment","B_B","C.1","D"
"1",2000,"AUSTRIA","total","no","m",33,55,77
"2",2000,"AUSTRIA","total","no","p",44,66,99
"3",2001,"AUSTRIA","total","no","m",22,0,24
"4",2001,"AUSTRIA","total","no","p",11,23,25
"5",2002,"AUSTRIA","total","no","m",88,56,38
"6",2002,"AUSTRIA","total","no","p",45,47,39
"7",2003,"AUSTRIA","total","no","m",90,67,69
"8",2003,"AUSTRIA","total","no","p",48,67,74

你可以用你所有的 csv 来试试这些函数，看看它是否会导致内存异常。我希望这会有所帮助。