使用对预先存在的向量的引用创建 data.table

Question

我想从大型预先存在的向量创建一个 data.table 而不复制这些向量。也就是说，我想创建一个 data.table ，它只对底层向量的指针进行浅拷贝，而不是向量中数据的完整拷贝。

我认为这是一个普遍的愿望，但我还没有找到任何方法来做到这一点。一旦向量是另一个 data.table 中的列，我可以廉价地制作更多副本，但我还没有看到有关如何通过引用创建该初始表的说明。

这可能吗？这是我尝试单个向量的方式，尽管我的实际目标是使用多个大向量创建一个 data.table：

nate@ubuntu:~/R/byreference$ cat dt.R

library(data.table)

# Some large vector that needs to be created anyway
largeVector = rnorm(1000*1000)

# I'd like to see no large memory allocations
Rprofmem("allocations.txt")

# I want to create dt without copying largeVector
dt = as.data.table(list(x = largeVector))

# This variation doesn't work either:
# dt = data.table(list(x = largeVector))

# This one comes closest to working, but acts as copy-on-write
# dt = setDT(list(x = largeVector))

# Currently, I see lots and lots of memory allocations
# (some may be https://github.com/Rdatatable/data.table/issues/1062)
Rprofmem(NULL)

# The addresses of the vectors should be identical if no copy occurred
identical(address(largeVector), address(dt$x))  # FIXME: should be TRUE

# For comparison, the addresses are identical if I copy 'dt'
dtCopy = dt
identical(address(dtCopy$x), address(dt$x))

# I'm not looking for copy-on-write semantics.  I'd like a simple
# reference, the same as would occur with a shallow copy of a data.table
dt[, x := 2.0*x]

# But this works! (see second edit at bottom)
# dt[1:.N, x := 2.0*x]    

# All of these should be true (currently only the last two are)
identical(dt$x, largeVector)                   # FIXME: should be TRUE
identical(address(largeVector), address(dt$x)) # FIXME: should be TRUE
identical(dt$x, dtCopy$x)
identical(address(dtCopy$x), address(dt$x))

这就是我在 R 3.1.2 和 data.table 1.9.4 中看到的：

nate@ubuntu:~/R/byreference$ Rscript dt.R
[1] FALSE
[1] TRUE
[1] FALSE
[1] FALSE
[1] TRUE
[1] TRUE

nate@ubuntu:~/R/byreference$ cat allocations.txt
1480 :"as.data.table"
6320 :"as.data.table"
6320 :"as.data.table"
1064 :"as.data.table"
344 :"as.data.table"
928 :"as.data.table"
1808 :"as.data.table"
600 :"as.data.table"
192 :"as.data.table"
408 :"as.data.table.list" "as.data.table"
1256 :"as.data.table.list" "as.data.table"
1248 :"as.data.table.list" "as.data.table"
1064 :"as.data.table.list" "as.data.table"
240 :"as.data.table.list" "as.data.table"
432 :"as.data.table.list" "as.data.table"
184 :"as.data.table.list" "as.data.table"
8000040 :"copy" "as.data.table.list" "as.data.table"
216 :"copy" "as.data.table.list" "as.data.table"
440 :"copy" "as.data.table.list" "as.data.table"
440 :"copy" "as.data.table.list" "as.data.table"
1064 :"copy" "as.data.table.list" "as.data.table"
536 :"as.data.table.list" "as.data.table"
1816 :"as.data.table.list" "as.data.table"
1808 :"as.data.table.list" "as.data.table"
1064 :"as.data.table.list" "as.data.table"
384 :"as.data.table.list" "as.data.table"
720 :"as.data.table.list" "as.data.table"
256 :"as.data.table.list" "as.data.table"
1024 :"as.data.table.list" "as.data.table"
4016 :"as.data.table.list" "as.data.table"
4016 :"as.data.table.list" "as.data.table"
1064 :"as.data.table.list" "as.data.table"
208 :"as.data.table.list" "as.data.table"
656 :"as.data.table.list" "as.data.table"
1264 :"as.data.table.list" "as.data.table"
416 :"as.data.table.list" "as.data.table"
184 :"eval" "eval" "alloc.col" "as.data.table.list" "as.data.table"
336 :"eval" "eval" "alloc.col" "as.data.table.list" "as.data.table"
336 :"eval" "eval" "alloc.col" "as.data.table.list" "as.data.table"
1064 :"eval" "eval" "alloc.col" "as.data.table.list" "as.data.table"
304 :"get" "dim" "ncol" "eval" "eval" "alloc.col" "as.data.table.list" "as.data.table"
872 :"get" "dim" "ncol" "eval" "eval" "alloc.col" "as.data.table.list" "as.data.table"
872 :"get" "dim" "ncol" "eval" "eval" "alloc.col" "as.data.table.list" "as.data.table"
1064 :"get" "dim" "ncol" "eval" "eval" "alloc.col" "as.data.table.list" "as.data.table"
208 :"get" "dim" "ncol" "eval" "eval" "alloc.col" "as.data.table.list" "as.data.table"
368 :"get" "dim" "ncol" "eval" "eval" "alloc.col" "as.data.table.list" "as.data.table"
840 :"alloc.col" "as.data.table.list" "as.data.table"
840 :"alloc.col" "as.data.table.list" "as.data.table"

哇，构建“dt”的分配比我预期的要多得多！虽然大多数都很小，但我真的希望能够避免使用大的，因为我的向量可能每个都有几个 GB。

编辑：Eddi 最初将此标记为 Sub-assign by reference on vector in R 的副本。它不是。我的目标不是就地修改向量；我的目标是从向量创建一个 data.table 而不复制该向量。我只使用了修改，因为大多数读者不会以允许 Rprofmem 用户的方式编译 R，并且检查副作用是保证没有复制发生。我已经更改了示例以尝试使这一点更清楚。

编辑：也就是说，我认为 Eddi 是对的，我的问题实际上是由于他刚刚提交的错误（更新：现已修复）：https://github.com/Rdatatable/data.table/issues/1248。 “dt = setDT(list(x = largeVector))”和“dt[1:.N, x := 2.0*x]”的组合正如我所料：修改到位，没有大的分配。所以虽然我不认为这实际上是重复的，但让这个问题消失可能很好。

Answer 1

尽管打开了data.tableissue #1248 [更新：现已解决]，但在不复制数据的情况下将一组向量转换为data.table 的方法是：

a = 1:5
b = 5:1
address(a)
#[1] "000000000FFE6AE0"
address(b)
#[1] "000000000FFE6A50"

dt = setDT(list(a, b))
sapply(dt, address)
#                V1                 V2 
#"000000000FFE6AE0" "000000000FFE6A50"