如何按多列拆分/聚合大型数据框（ffdf）？

Question

ffbase 提供函数ffdfdply 来拆分和聚合数据行。这个答案（https://stackoverflow.com/a/20954315/336311）解释了它是如何工作的。我仍然无法弄清楚如何按多列进行拆分。

我的挑战是需要拆分变量。对于两个变量的每个组合，这必须是唯一的，我想分开。尽管如此，在我的 4 列数据帧（大约 50M 行）中，如果通过 paste() 创建字符向量，则需要大量内存。

这就是我卡住的地方......

require("ff")
require("ffbase")
load.ffdf(dir="ffdf.shares.02")

# Aggregation by articleID/measure
levels(ffshares$measure) #  "comments", "likes", "shares", "totals", "tw"
splitBy = paste(as.character(ffshares$articleID), ffshares$measure, sep="")

tmp = ffdfdply(fftest, split=splitBy, FUN=function(x) {
  return(list(
    "articleID" = x[1,"articleID"],
    "measure" = x[1,"measure"],
    # I need vectors for each entry
    "sx" = unlist(x$value), 
    "st" = unlist(x$time)
  ))
}
)

当然，我可以为ffshares$measure 使用较短的级别，或者只使用数字代码，但这仍然不能解决splitBy 变得非常大的根本问题。

样本数据

    articleID  measure                time value
100        41   shares 2015-01-03 23:20:34     4
101        41       tw 2015-01-03 23:30:30    24
102        41   totals 2015-01-03 23:30:38     6
103        41    likes 2015-01-03 23:30:38     2
104        41 comments 2015-01-03 23:30:38     0
105        41   shares 2015-01-03 23:30:38     4
106        41       tw 2015-01-03 23:40:24    24
107        41   totals 2015-01-03 23:40:35     6
108        41    likes 2015-01-03 23:40:35     2
...
1000       42   shares 2015-01-04 20:10:50     0
1001       42       tw 2015-01-04 21:10:45    24
1002       42   totals 2015-01-04 21:10:35     0
1003       42    likes 2015-01-04 21:10:35     0
1004       42 comments 2015-01-04 21:10:35     0
1005       42   shares 2015-01-04 21:10:35     0
1006       42       tw 2015-01-04 22:10:45    24
1007       42   totals 2015-01-04 22:10:43     0
1008       42    likes 2015-01-04 22:10:43     0
...

Answer 1

# Use this, this makes sure your data does not get into RAM completely but only in chunks of 100000 records
ffshares$splitBy <- with(ffshares[c("articleID", "measure")], paste(articleID, measure, sep=""), 
                         by = 100000)
length(levels(ffshares$splitBy)) ## how many levels are in there - don't know from your question

tmp <- ffdfdply(ffshares, split=ffshares$splitBy, FUN=function(x) {
  ## In x you are getting a data.frame in RAM with all records of possibly several articleID/measure combinations
  ## You should write a function which returns a data.frame. E.g. the following returns the mean value by articleID/measure and the first and last timepoint
  x <- data.table::setDT(x)
  xagg <- x[, list(value = mean(value), 
                   first.timepoint = min(time),
                   last.timepoint = max(time)), by = list(articleID, measure)]
  ## the function should return a data frame as indicated in the help of ffdfdply, not a list
  setDF(xagg)
})
## tmp is an ffdf