data.table：二分查找 VS 矢量扫描的性能答案

【问题标题】：data.table: performance of binary search VS vector scandata.table：二分查找 VS 矢量扫描的性能
【发布时间】：2020-08-17 23:18:22
【问题描述】：

我正在寻找对 data.table 进行子集化的最佳方法，定义如下：

library(data.table)
library(microbenchmark)

set.seed(2L)
N = 1e7L
DT = data.table(x = sample(letters, N, TRUE),
                y = sample(1000L, N, TRUE),
                val = runif(N))
setkey(DT, x, y)

有二分查找（SUBSET1）和“矢量扫描方式”（SUBSET2）。

SUBSET1 <- function(){
  a <- DT[.(c("a"), c(5L)), .N, nomatch = NULL]
}
SUBSET2 <- function(){
  a <- DT[ x == "a" & y == 5L, .N, nomatch = NULL]
}

我非常喜欢“矢量扫描方式”的一点是，它确实不言自明且可读性强。不过，与原生二分搜索方式相比，它似乎要慢 2 倍。

microbenchmark(SUBSET1(), 
               SUBSET2(), 
               times = 500 )
  Unit: milliseconds
        expr    min      lq     mean  median     uq      max neval
   SUBSET1() 1.0328 1.27790 1.878415 1.53370 1.8924  20.5789   500
   SUBSET2() 2.4896 3.06665 4.476864 3.52685 4.3682 179.1607   500

我的问题
我不明白为什么 SUBSET2 更慢。是因为有一种从“矢量扫描方式”到二进制搜索的内部转换，还是因为“矢量扫描方式”是这样执行的（因此比二进制搜索慢）？

【问题讨论】：

使用verbose=TRUE，您的矢量扫描可能已经在后台优化为二进制搜索。以毫秒为单位的时间不会那么相关，因为查询的所有编排都需要一些时间，所以单位越小，关于二分搜索与向量扫描的结论就越不有用。对工作流程的“大局”进行基准测试更为有用。我的回答是，由于编排成本和“毫秒”，它会变慢。

标签： r data.table subset

【解决方案1】：

正如@jangorecki 所指出的，这两个查询都已经在使用该密钥——后者只需要少量额外的时间来将“矢量扫描”表单映射到二进制搜索表单。你可以通过verbose=TRUE看到这个：

DT[ x == "a" & y == 5L, .N, nomatch = NULL, verbose = TRUE]

显示输出：

Optimized subsetting with key 'x, y'
forder.c received 1 rows and 2 columns
forder took 0.001 sec
x is already ordered by these columns, no need to call reorder
i.x has same type (character) as x.x. No coercion needed.
i.y has same type (integer) as x.y. No coercion needed.
on= matches existing key, using key
Starting bmerge ...
bmerge done in 0.000s elapsed (0.000s cpu) 
Constructing irows for '!byjoin || nqbyjoin' ... 0.000s elapsed (0.000s cpu) 
Detected that j uses these columns: <none>

与直接二分查找版本比较：

DT[.("a", 5L), .N, nomatch = NULL, verbose = TRUE]

i.V1 has same type (character) as x.x. No coercion needed.
i.V2 has same type (integer) as x.y. No coercion needed.
on= matches existing key, using key
Starting bmerge ...
forder.c received 1 rows and 2 columns
bmerge done in 0.001s elapsed (0.000s cpu) 
Constructing irows for '!byjoin || nqbyjoin' ... 0.000s elapsed (0.000s cpu) 
Detected that j uses these columns: <none>

但速度只有一半，对吧？同样如所指出的，时间尺度非常小。一个更有用的比较是与根本不使用密钥的情况。让我们为您的数据制作一个未排序的副本：

DTrand = DT[sample(.N)]

另一个快速的问题——我们必须小心基准测试，因为data.table 还在进行一些自动优化，以帮助对您的数据进行排序，即使在这种未排序的情况下：

DTrand[ x == "a" & y == 5L, .N, nomatch = NULL, verbose = TRUE]

仔细阅读输出：

Creating new index 'y__x'
Creating index y__x done in ... forder.c received 10000000 rows and 3 columns
forder took 0.424 sec
0.286s elapsed (1.117s cpu) 
Optimized subsetting with index 'y__x'
forder.c received 1 rows and 2 columns
forder took 0.002 sec
x is already ordered by these columns, no need to call reorder
i.y has same type (integer) as x.y. No coercion needed.
i.x has same type (character) as x.x. No coercion needed.
on= matches existing index, using index
Starting bmerge ...
bmerge done in 0.000s elapsed (0.000s cpu) 
Constructing irows for '!byjoin || nqbyjoin' ... 0.000s elapsed (0.001s cpu) 
Reorder irows for 'mult=="all" && !allGrp1' ... forder.c received 360 rows and 2 columns
0.000s elapsed (0.002s cpu) 
Detected that j uses these columns: <none> 
[1] 360

data.table 已自动将setindex 应用到您的表中（虽然不如setkey 那样的物理排序快），但仍会加速任何未来的子集；简单地重复（就像基准测试一样）：

DTrand[ x == "a" & y == 5L, .N, nomatch = NULL, verbose = TRUE]

注意与键控大小写的相似性（将key 替换为index）：

Optimized subsetting with index 'y__x'
forder.c received 1 rows and 2 columns
forder took 0 sec
x is already ordered by these columns, no need to call reorder
i.y has same type (integer) as x.y. No coercion needed.
i.x has same type (character) as x.x. No coercion needed.
on= matches existing index, using index
Starting bmerge ...
bmerge done in 0.000s elapsed (0.000s cpu) 
Constructing irows for '!byjoin || nqbyjoin' ... 0.000s elapsed (0.000s cpu) 
Reorder irows for 'mult=="all" && !allGrp1' ... forder.c received 360 rows and 2 columns
0.001s elapsed (0.001s cpu) 
Detected that j uses these columns: <none> 
[1] 360

因此，即使在DTrand 上进行简单的基准测试也不会是真正的比较——在第一次基准测试运行后，表将被索引，后续子集将使用此和二分搜索。详情请见the vignette on secondary indices。

我们可以通过将选项 datatable.auto.index 设置为 FALSE 并重置现有索引来回避这一点并获得适当的基准：

options(datatable.auto.index = FALSE)
setindex(DTrand, NULL)

现在data.table 忘记了如何通过x 和y 对DTrand 进行排序，我们可以比较二分查找方法和真向量子集：

microbenchmark::microbenchmark(
  times = 50L,
  vector = DTrand[ x == "a" & y == 5L, .N, nomatch = NULL],
  binary = DT[     x == "a" & y == 5L, .N, nomatch = NULL]
)
# Unit: milliseconds
#    expr       min         lq       mean     median        uq        max neval
#  vector 101.43306 114.325340 134.154362 119.367909 128.05273 345.721296    50
#  binary   1.06033   1.160188   1.631119   1.367017   1.57334   5.508802    50

因此，虽然使用 .() 的直接方法的速度是使用 == 的优化方法的两倍，但 == 仍然比 true 向量子集快 100 倍。

您还可以从data.table benchmarking vignette 中受益

【讨论】：