R+data.table 子集数据集基于 3 列的组合并仅保留第一行答案

【问题标题】：R+data.table subsetting dataset based on the combination of 3 columns and keeping only first rowsR+data.table 子集数据集基于 3 列的组合并仅保留第一行
【发布时间】：2015-12-05 07:05:20
【问题描述】：

我对该线程有一个简短的后续问题： R subsetting dataframe based on the combination of 3 columns and excluding duplicate combinations

我有一个这样的数据集：

Experiment  Sequence    Parameter   Time
Exp1        JJJJ        2           10     *
Exp1        JJJJ        2           11     *
Exp1        JJJJ        2           12     *
Exp2        JJJJ        2           13
Exp3        JJJJ        2           15
Exp1        ZZZZ        3           12    
Exp2        ZZZZ        3           23     *
Exp2        ZZZZ        3           23.5   *
Exp2        ZZZZ        3           24     *
Exp3        ZZZZ        3           15
.....

现在，对于每个序列，我只想保留唯一的 Experiment+Sequence+Parameter 组合中的第一个。实际上，减少标有 * 的行，最终得到如下数据集：

Experiment  Sequence    Parameter   Time
Exp1        JJJJ        2           10     *
Exp2        JJJJ        2           13    
Exp3        JJJJ        2           15
Exp1        ZZZZ        3           12
Exp2        ZZZZ        3           23     *
Exp3        ZZZZ        3           15
.....

我想使用 data.table 包，这真的很棒，我想出了这个解决方案。这需要相当长的时间，所以我想知道是否有更好/最快的方式/语法。

keycols = c("Sequence","Parameter","Experiment")
setkeyv(DT,keycols) 
DT <- DT[,head(.SD,1), by = key(DT)]

【问题讨论】：

虽然这样做也可以，但是没有特别需要事先设置keycols。只要DT[, head(.SD,1), by = .(Sequence, Parameter, Experiment)] 就可以了。
我会使用.SD[1,] 而不是head，但您的方法似乎是正确的。
@Beginner，您应该将数据包含在您的问题中，而不仅仅是打印它的一部分。会话信息也很有用。
你好jangorecki，你是对的对不起，下次我会更详细的。

标签： r data.table

【解决方案1】：

下面将上面两个cmet中建议的两种方法的速度与下面的方法进行比较。

setkey(DT,Sequence,Parameter,Experiment)
DT[unique(DT[,.(Experiment,Sequence,Parameter)]),mult = "first"]

首先我生成一个包含 5000 万行的 data.table。

Sequ  <-   c("AAAA","BBBB","CCCC","DDDD","EEEE","FFFF","GGGG","HHHH","IIII","JJJJ")
DT <- as.data.table(cbind(Experiment = sample(1:2000,50000000,replace = TRUE),Parameter = sample(1:9,50000000,replace = TRUE),
 Sequence = sample(Sequ,10,replace = TRUE) ))

DT[,Time := sample(1:60,.N,replace = TRUE)]
setkey(DT,NULL)

start1 <- Sys.time()
DT[, head(.SD,1), by = .(Sequence, Parameter, Experiment)]
end1 <- Sys.time()

start2 <- Sys.time()
DT[, .SD[1], by = .(Sequence, Parameter, Experiment)]
end2 <- Sys.time()

start3 <- Sys.time()
setkey(DT,Sequence,Parameter,Experiment)
DT[unique(DT[,.(Experiment,Sequence,Parameter)]),mult = "first"]
end3 <- Sys.time()
paste("Time 1 = ",t1 <- end1 - start1,sep ="")

“时间 1 = 5.45559597015381”

paste("Time 2 = ",t2 <- end2 - start2,sep ="")

“时间 2 = 3.54731583595276”

paste("Time 3 = ",t3 <- end3 - start3,sep ="")

“时间 3 = 10.0164358615875”

【讨论】：

评论中的解决方案应该比你的更快
感谢您发布此问题的答案！在 Stack Overflow 上不鼓励仅使用代码的答案，因为原始发布者（或未来的读者）可能难以理解它们背后的逻辑。请编辑您的问题并包含对您的代码的解释，以便其他人可以从您的回答中受益。谢谢！（此注释是从另一个用户那里复制的...）
@jangorecki 我不这么认为。我已经编辑了我的解决方案来演示。
@HywelMJ 如果你想比较时间，你应该包括setkey
@jangorecki OK：只为我的解决方案包括 setkey 意味着时间增加了 5%、15% 和 50% - 但这仍然意味着我的解决方案只需要 0.374566078186035，即仍然是第二个解决方案的两倍.

【解决方案2】：

TL;DR

如果你有 data.table 1.9.6 使用

DT[, .SD[1L], by = .(Sequence, Parameter, Experiment)]

如果您使用的是 1.9.7+，您也可以使用

DT[, head(.SD,1L), by = .(Sequence, Parameter, Experiment)]

一些基准测试

低于 1e6 和 1e7 的基准。由于最近引入了head(.SD, 1) 的优化，还测试了data.table 的dev 版本。
我使用了@HywelMJ 生成的数据集，但它似乎没有按照Time 列的顺序反映OP 数据集，因此数据是无序的。一旦 OP 将提供可重现的示例，我可能会更新时间。
由于不同的排序和 HywelMJ 中使用的mult="first" 答案，结果不同。我假设（查看 OP 数据的打印）Jaap 和 nicola 的答案是正确的。

# 1e6 - data.table 1.9.6 ----

# install.packages("data.table")
packageVersion("data.table")
#[1] ‘1.9.6’
library(data.table)
DT <- as.data.table(cbind(Experiment = round(runif(1000000,min = 1, max = 2000)),Parameter = round(runif(1000000,min = 1,max = 9))))
DT[,Sequence:= seq_len(.N),keyby = Experiment]
DT[,Time := sample(1:60,.N,replace = TRUE)]
DT.backup = DT[sample(nrow(DT))] # ensure random order

DT = copy(DT.backup)
system.time(
    r.head <- DT[, head(.SD,1L), by = .(Sequence, Parameter, Experiment)]
)
#   user  system elapsed 
#  8.420   0.000   8.408 

DT = copy(DT.backup)
system.time(
    r.sd1 <- DT[, .SD[1L], by = .(Sequence, Parameter, Experiment)]
)
#   user  system elapsed 
#  0.664   0.000   0.664 

DT = copy(DT.backup)
system.time({
    setkey(DT,Sequence,Parameter,Experiment)
    r.join <- DT[unique(DT[,.(Experiment,Sequence,Parameter)]),mult = "first"]
})
#    user  system elapsed 
#   0.332   0.000   0.331

all.equal(r.head, r.sd1)
#[1] TRUE
all.equal(r.head[order(Sequence,Parameter,Experiment), .(Sequence,Parameter,Experiment,Time)],
          r.join[order(Sequence,Parameter,Experiment), .(Sequence,Parameter,Experiment,Time)])
#[1] "Attributes: < Length mismatch: comparison on first 1 components >"

# 1e7 - data.table 1.9.6 ----

DT <- as.data.table(cbind(Experiment = round(runif(1e7,min = 1, max = 2000)),Parameter = round(runif(1e7,min = 1,max = 9))))
DT[,Sequence:= seq_len(.N),keyby = Experiment]
DT[,Time := sample(1:60,.N,replace = TRUE)]
DT.backup = DT[sample(nrow(DT))] # ensure random order

DT = copy(DT.backup)
system.time(
    r.head <- DT[, head(.SD,1L), by = .(Sequence, Parameter, Experiment)]
)
#   user  system elapsed 
#  85.848   0.064  85.829

DT = copy(DT.backup)
system.time(
    r.sd1 <- DT[, .SD[1L], by = .(Sequence, Parameter, Experiment)]
)
#   user  system elapsed 
#  7.164   0.044   7.201

DT = copy(DT.backup)
system.time({
    setkey(DT,Sequence,Parameter,Experiment)
    r.join <- DT[unique(DT[,.(Experiment,Sequence,Parameter)]),mult = "first"]
})
#    user  system elapsed 
#   3.440   0.080   3.516

all.equal(r.head, r.sd1)
#[1] TRUE
all.equal(r.head[order(Sequence,Parameter,Experiment,Time), .(Sequence,Parameter,Experiment,Time)],
          r.join[order(Sequence,Parameter,Experiment,Time), .(Sequence,Parameter,Experiment,Time)])
#[1] "Attributes: < Length mismatch: comparison on first 1 components >"

# 1e6 - data.table 1.9.7 ----

# devtools::install_github("Rdatatable/data.table")
packageVersion("data.table")
#[1] ‘1.9.7’
library(data.table)

DT <- as.data.table(cbind(Experiment = round(runif(1000000,min = 1, max = 2000)),Parameter = round(runif(1000000,min = 1,max = 9))))
DT[,Sequence:= seq_len(.N),keyby = Experiment]
DT[,Time := sample(1:60,.N,replace = TRUE)]
DT.backup = DT[sample(nrow(DT))] # ensure random order

DT = copy(DT.backup)
system.time(
    r.head <- DT[, head(.SD,1L), by = .(Sequence, Parameter, Experiment)]
)
#   user  system elapsed 
#  0.236   0.008   0.242

DT = copy(DT.backup)
system.time(
    r.sd1 <- DT[, .SD[1L], by = .(Sequence, Parameter, Experiment)]
)
#   user  system elapsed 
#  0.216   0.004   0.220 

DT = copy(DT.backup)
system.time({
    setkey(DT,Sequence,Parameter,Experiment)
    r.join <- DT[unique(DT[,.(Experiment,Sequence,Parameter)]),mult = "first"]
})
#    user  system elapsed 
#   0.324   0.000   0.324 

all.equal(r.head, r.sd1)
#[1] TRUE
all.equal(r.head[order(Sequence,Parameter,Experiment,Time), .(Sequence,Parameter,Experiment,Time)],
          r.join[order(Sequence,Parameter,Experiment,Time), .(Sequence,Parameter,Experiment,Time)])
#[1] "Attributes: < Length mismatch: comparison on first 1 components >"

# 1e7 - data.table 1.9.7 ----

DT <- as.data.table(cbind(Experiment = round(runif(1e7,min = 1, max = 2000)),Parameter = round(runif(1e7,min = 1,max = 9))))
DT[,Sequence:= seq_len(.N),keyby = Experiment]
DT[,Time := sample(1:60,.N,replace = TRUE)]
DT.backup = DT[sample(nrow(DT))] # ensure random order

DT = copy(DT.backup)
system.time(
    r.head <- DT[, head(.SD,1L), by = .(Sequence, Parameter, Experiment)]
)
#   user  system elapsed 
#  2.676   0.056   2.732

DT = copy(DT.backup)
system.time(
    r.sd1 <- DT[, .SD[1L], by = .(Sequence, Parameter, Experiment)]
)
#   user  system elapsed 
#  2.620   0.112   2.728

DT = copy(DT.backup)
system.time({
    setkey(DT,Sequence,Parameter,Experiment)
    r.join <- DT[unique(DT[,.(Experiment,Sequence,Parameter)]),mult = "first"]
})
#    user  system elapsed 
#   3.636   0.084   3.714

all.equal(r.head, r.sd1)
#[1] TRUE
all.equal(r.head[order(Sequence,Parameter,Experiment,Time), .(Sequence,Parameter,Experiment,Time)],
          r.join[order(Sequence,Parameter,Experiment,Time), .(Sequence,Parameter,Experiment,Time)])
#[1] "Attributes: < Length mismatch: comparison on first 1 components >"

【讨论】：