加速函数

Question

我想计算大型面板数据集的一阶差异。然而，目前这需要一个多小时。我真的很想知道是否还有任何选择可以加快这一进程。以数据库为例：

set.seed(1)
DF <- data.table(panelID = sample(50,50),                                                    # Creates a panel ID
                      Country = c(rep("A",30),rep("B",50), rep("C",20)),                      
                      Group = c(rep(1,20),rep(2,20),rep(3,20),rep(4,20),rep(5,20)),
                      Time = rep(seq(as.Date("2010-01-03"), length=20, by="1 month") - 1,5),
                      norm = round(runif(100)/10,2),
                      Income = sample(100,100),
                      Happiness = sample(10,10),
                      Sex = round(rnorm(10,0.75,0.3),2),
                      Age = round(rnorm(10,0.75,0.3),2),
                      Educ = round(rnorm(10,0.75,0.3),2))           
DF [, uniqueID := .I]  

所以我尝试过的是：

DFx <- DF
start_time <- Sys.time()
    DF <- DF[, lapply(.SD, function(x) x - shift(x)), by = panelID, .SDcols = (sapply(DF, is.numeric))]
end_time <- Sys.time()
DF <- DFx
start_time2 <- Sys.time()
    cols = sapply(DF, is.numeric)
    DF <- DF[, lapply(.SD, function(x) x - shift(x)), by = panelID, .SDcols = cols]
end_time2 <- Sys.time()
DF <- DFx
start_time3 <- Sys.time()
DF <- DF[order(panelID)] # Sort on year
nm1 <- sapply(DF, is.numeric) # Get the numerical columns  
nm1 = names(nm1) 
nm2 <- paste("delta", nm1, sep="_")[-6] # Paste
DF <- DF[,(nm2) := .SD - shift(.SD), by=panelID] # Creates 
end_time3 <- Sys.time()
end_time3 - start_time3
end_time2 - start_time2
end_time - start_time

由于某种原因，第三个选项适用于我的实际数据库，但不适用于本示例。它给出了错误：Error in FUN(left, right) : non-numeric argument to binary operator。对于我的实际数据库，这种计算方式也相当慢（然后我仍然需要子集）。

有什么方法可以加快速度吗？

Answer 1

data.table 针对多行而不是多列进行了优化。由于您有很多列，您可以尝试融合 data.table：

DFm <- melt(DF[, cols, with = FALSE][, !"uniqueID"], id = "panelID") 
#coerces all numers to double (common type), 
#you could separate the data.table by integer/double to avoid this

DFm[, value := c(NA, diff(value)), by = .(panelID, variable)]


dcast(DFm, panelID + rowidv(DFm, cols = c("panelID", "variable")) ~ variable, value.var = "value")