过滤多个 R data.table 列以消除异常值

Question

对于许多具有相似名称的变量（太多而无法在代码中单独指定），我想消除高于或低于 2 个标准差的异常值。

library(data.table)

irisdt <- data.table(iris)
myCols <- grep("Sepal", colnames(irisdt), value=TRUE) 

# This works if I specify one column, 
# but I have too many columns to specify, so need to use grep approach.
irisdt[, Sepal.Length.Outlier := (scale(Sepal.Length) < -2 | scale(Sepal.Length) > 2)]

# This does not work
irisdt[, (myCols) := lapply(myCols, function(x) {(scale(x) < -2 | scale(x) > 2)} )] 

# This partially works, but changes in place
irisdt[, (myCols) := lapply(myCols, function(x) {(scale(irisdt[[x]]) < -2 | scale(irisdt[[x]]) > 2)} )] 
# How do I make new variables, for example "Sepal.Length.Outlier"?

myOutlierCols <- grep(".Outlier", colnames(irisdt), value=TRUE) 

# How do I select rows matching multiple columns (&)? 
irisdt[myOutlierCols=="FALSE"] # does not work
irisdt[, hasOutlier := lapply(myCols, myCols==TRUE)] # does not work
irisdt[hasOutlier=="FALSE"] # relies on line above, which doesn't work

也许一个函数可以获取一个 data.table 列并将其去除高于或低于 z 分数截止值的值。这可以与 lapply 一起使用。

# This does not work
removeOutliers <- function(myColumn, cutoff = 3) {
  lapply(myColumn, function (x) {
    if (scale(myColumn[[x]]) < -cutoff | scale(myColumn[[x]]) > cutoff) {
      x <- NA #specify individual value instead of column?
    } 
  })
}
removeOutliers(irisdt[,Sepal.Length]) # for testing
trimmedIrisdt <- irisdt[,lapply(.SD, removeOutliers(.SD)), .SDcols = myCols] # could do by = grouping variable

# Once outliers are made NA, this would work:
trimmedIrisdt <- complete.cases(trimmedIrisdt)

Answer 1

我猜这达到了目的：

irisdt[, keep := 
  as.logical(do.call(pmin, lapply(.SD, function(x) abs(scale(x)) <= 2)))
, .SDcols = myCols]

res = irisdt[(keep), !"keep"]

     Sepal.Length Sepal.Width Petal.Length Petal.Width   Species
  1:          5.1         3.5          1.4         0.2    setosa
  2:          4.9         3.0          1.4         0.2    setosa
  3:          4.7         3.2          1.3         0.2    setosa
  4:          4.6         3.1          1.5         0.2    setosa
  5:          5.0         3.6          1.4         0.2    setosa
 ---                                                            
135:          6.7         3.0          5.2         2.3 virginica
136:          6.3         2.5          5.0         1.9 virginica
137:          6.5         3.0          5.2         2.0 virginica
138:          6.2         3.4          5.4         2.3 virginica
139:          5.9         3.0          5.1         1.8 virginica

如果有分组变量，这也应该可以正常工作。我不知道它的统计可靠性。

---

工作原理：

1. 测试每个单元格的abs(scale(x)) <= 2。
2. 如果跨列的最小结果为 TRUE，则保留该行。

逐个单元地查看它是如何工作的......

library(data.table)

mynewCols = paste0(myCols,"_outly")
irisdt[, (mynewCols) := 
  lapply(.SD, function(x) replace(x, abs(scale(x)) <= 2, NA))
, .SDcols = myCols]

然后像View(irisdt[rowSums(!is.na(irisdt[, ..mynewCols])) > 0])一样浏览。