使用 data.table 在日期范围内聚合/合并

Question

假设我有两个 data.tables：

summary <- data.table(period = c("A","B","C","D"),
                 from_date = ymd(c("2017-01-01", "2017-01-03", "2017-02-08", "2017-03-07")),
                 to_date = ymd(c("2017-01-31", "2017-04-01", "2017-03-08", "2017-05-01"))
)

log <- data.table(date = ymd(c("2017-01-03","2017-01-20","2017-02-01","2017-03-03",
                               "2017-03-15","2017-03-28","2017-04-03","2017-04-23")),
                  event1 = c(4,8,8,4,3,4,7,3), event2 = c(1,8,7,3,8,4,6,3))

看起来像这样：

> summary
   period  from_date    to_date
1:      A 2017-01-01 2017-01-31
2:      B 2017-01-03 2017-04-01
3:      C 2017-02-08 2017-03-08
4:      D 2017-03-07 2017-05-01
> log
         date event1 event2
1: 2017-01-03      4      1
2: 2017-01-20      8      8
3: 2017-02-01      8      7
4: 2017-03-03      4      3
5: 2017-03-15      3      8
6: 2017-03-28      4      4
7: 2017-04-03      7      6
8: 2017-04-23      3      3

我想得到 summary 表中每个时间段的 event1 和 event2 的总和。

我知道我可以做到：

summary[, c("event1","event2") := .(sum(log[date>=from_date & date<=to_date, event1]),
                               sum(log[date>=from_date & date<=to_date, event2]))
   , by=period][]

得到想要的结果：

   period  from_date    to_date event1 event2
1:      A 2017-01-01 2017-01-31     12      9
2:      B 2017-01-03 2017-04-01     31     31
3:      C 2017-02-08 2017-03-08      4      3
4:      D 2017-03-07 2017-05-01     17     21

现在，在我的实际问题中，我有大约 30 列要求和，我可能想稍后更改，summary 有大约 35,000 行，log 有约 40,000,000 行。有没有一种有效的方法来实现这一点？

注意：这是我在这里的第一篇文章。我希望我的问题足够清楚和具体，如果我应该做些什么来改进这个问题，请提出建议。谢谢！

Answer 1

是的，您可以执行 非 equi 连接。

（请注意，我已将 log 和 summary 更改为 Log 和 Summary，因为原件已经是 R 中的函数。）

Log[Summary,
   on = c("date>=from_date", "date<=to_date"),
   nomatch=0L, 
   allow.cartesian = TRUE][, .(from_date = date[1],
                               to_date = date.1[1],
                               event1 = sum(event1),
                               event2 = sum(event2)),
                           keyby = "period"]

要对列模式求和，请使用 lapply 和 .SD：

joined_result <- 
  Log[Summary,
      on = c("date>=from_date", "date<=to_date"),
      nomatch = 0L, 
      allow.cartesian = TRUE]

cols <- grep("event[a-z]?[0-9]", names(joined_result), value = TRUE)

joined_result[, lapply(.SD, sum),
              .SDcols = cols,
              keyby = .(period,
                        from_date = date,
                        to_date = date.1)]

Answer 2

使用data.table，可以在非等值连接期间使用by = .EACHI 进行聚合。

log[summary, on = .(date >= from_date, date <= to_date), nomatch=0L, 
    lapply(.SD, sum), by = .EACHI]

         date       date event1 event2
1: 2017-01-01 2017-01-31     12      9
2: 2017-01-03 2017-04-01     31     31
3: 2017-02-08 2017-03-08      4      3
4: 2017-03-07 2017-05-01     17     21

加上一些额外的清理：

log[summary, on = .(date >= from_date, date <= to_date), nomatch=0L, 
    c(period = period, lapply(.SD, sum)), by = .EACHI][
      , setnames(.SD, 1:2, c("from_date", "to_date"))]

    from_date    to_date period event1 event2
1: 2017-01-01 2017-01-31      A     12      9
2: 2017-01-03 2017-04-01      B     31     31
3: 2017-02-08 2017-03-08      C      4      3
4: 2017-03-07 2017-05-01      D     17     21