如何按条件和按组过滤我的 data.table？

Question

问题

我在 data.table 上工作，其中每一行都是医学观察。问题是我的数据中有一些错误，我需要在进行分析之前更正它们。例如，一名男性患者可以观察到他被编码为女性的情况。

解决方案

我的解决方案是由患者选择变量的模式（最常见的值）。如果患者有 10 次观察为男性，1 次观察为女性，则可以安全地假设他是男性。

我发现了使用 data.table 的巧妙方法。

DATA[j  = .N, 
     by = .(ID, SEX)][i = base::order(-N), 
     j = .(SEX = SEX[1L]), 
     keyby = ID]

问题是当一个病人作为多个模式时，它只保留一个。所以一个50%男性和50%女性的患者会被算作男性，这最终会导致偏差。我想将它们编码为 NA。

我创建的唯一纠正这个问题的方法是使用 dplyr

DATA[j  = .N, 
     by = .(ID, SEX)] %>% 
     group_by(ID) %>% 
     filter(N == max(N))

如果重复，则将 SEX 值替换为 NA。但它比 data.table 需要更长的时间，它不是很优化，而且我有一个大数据集，其中有很多变量也需要更正。

简历

如果不是唯一的，我如何获取患者变量的模式并将其替换为 NA？

示例

ID <- c(rep(x = "1", 6), rep(x = "2", 6))
SEX <- c("M","M","M","M","F","M","M","F","M","F","F","M")

require(data.table)
DATA <- data.table(ID, SEX)

# First method (doesn't work)
DATA[j  = .N, 
     by = .(ID, SEX)][i = base::order(-N), 
     j = .(SEX = SEX[1L]), 
     keyby = ID]

# Second method (work with dplyr)
require(dplyr)
DATA[j  = .N, 
     by = .(ID, SEX)] %>% 
     group_by(ID) %>% 
     filter(N == max(N)) %>%
     mutate(SEX = if_else(condition = duplicated(ID) == TRUE,
                          true = "NA",
                          false = SEX)) %>%
     filter(row_number() == n())

# Applied to my data it took 84.288 seconds

更新

@Cole 基于@Sindri_baldur 的想法提出的解决方案：

DATA <- data.table(
 ID = c(rep(x = "1", 6), rep(x = "2", 6)),
 SEX = c("M","M","M","M","F","M","M","F","M","F","F",NA),
 V1 = c("a", NA, "a", "a", "b", "a", "b", "b", "b", "c", "b", "c")
)

our_mode_fac <- function(x) {
  freq <- tabulate(x)
       if (length(freq) == 0 || sum(freq == max(freq)) > 1 ) {NA}
       else {levels(x)[which.max(freq)]}
  }

vars <- c("SEX", "V1")

DATA[j = paste0(vars) := lapply(.SD, as.factor), 
     .SDcols = vars][j = vars := lapply(.SD, our_mode_fac),
                     .SDcols = vars, 
                     by = ID]

它工作得很好。即使 NA 多于因子，它也采用模式，当模式超过 1 时，用 NA 替换值。

现在它也非常快：3M+ 观察和 1M+ 患者需要 11 秒（@Sindri_baldur 回答需要 117 秒）。非常感谢你们两位，我非常感谢！

Answer 1

our_mode <- function(x) {
  freq <- table(x)
  if (length(freq) == 0 || sum(freq == max(freq)) > 1 ) {
    NA
  } else {
    names(freq)[which.max(freq)]
  }
}

vars <- c("SEX", "V1")
DATA[, paste0(vars, "_corrected") := lapply(.SD, our_mode), .SDcols = vars, by = ID]

    ID  SEX   V1 SEX_corrected V1_corrected
 1:  1    M    a             M            a
 2:  1    M <NA>             M            a
 3:  1    M    a             M            a
 4:  1    M    a             M            a
 5:  1    F    b             M            a
 6:  1    M    a             M            a
 7:  2    M    b             F            b
 8:  2    F    b             F            b
 9:  2    M    b             F            b
10:  2    F    c             F            b
11:  2    F    b             F            b
12:  2 <NA>    c             F            b

可重复的数据

DATA <- data.table(
 ID = c(rep(x = "1", 6), rep(x = "2", 6)),
 SEX = c("M","M","M","M","F","M","M","F","M","F","F",NA),
 V1 = c("a", NA, "a", "a", "b", "a", "b", "b", "b", "c", "b", "c")
)

请注意，our_mode() 并未针对速度进行优化。请参阅 Cole 提出的关于提高 cmets 速度的建议。