合并 R 中的大数据集并标记不匹配答案

【问题标题】：merging big datasets in R and flagging unmatched合并 R 中的大数据集并标记不匹配
【发布时间】：2013-01-08 12:43:25
【问题描述】：

我正在尝试连续加入多个数据集，并标记第一个数据集中在后续数据集中找不到匹配项的观察结果。下面是一个例子，我模拟原始数据集加上三个附加加入。当前的代码可以满足我的要求，但效率非常低。对于大型数据集，可能需要几天时间。是否可以使用 apply 或其他功能来完成此任务？

#Toy datasets: x, y, z and w

#dataset X
id <- c(1:10, 1:100)
X1 <- rnorm(110, mean = 0, sd = 1)
year <- c("2004","2005","2006","2001","2002") 
year <- rep(year, 22)

month = c("Jul","Aug","Sep","Oct","Nov","Dec","Jan","Feb","Mar","Apr")
month <- rep(month, 11)

x <- data.frame(id, X1, month, year)

#dataset Y
id2 <- c(1:10, 41:110)
Y1 <- rnorm(80, mean = 0 , sd = 1)
year <- c("2004","2005","2006","2001") 
year <- rep(year, 20)

month = c("Jul","Aug","Sep","Oct","Nov","Dec","Jan","Feb","Mar","Apr")
month <- rep(month, 8)

y <- data.frame(id2,Y1, year,month)


#dataset z 
id3 = c(1:60, 401:10000)
Z1 = rpois(9660, 10) 
year = c('2004','2005','2006','2002')
year = rep(year, 2415)

month = c("Jul","Aug","Sep","Oct","Nov","Dec","Jan","Feb","Mar","Apr")
month <- rep(month, 966)

z = data.frame(id3,Z1,year,month)

#dataset w
id4 = c(1:300, 20:29)
W1 = rnorm(310, 20, 36)
year = c('2004','2005','2006','2000','2002')
year = rep(year, 62)

month = c("Jul","Aug","Sep","Oct","Nov","Dec","Jan","Feb","Mar","Apr")
month <- rep(month, 31)

w = data.frame(id4, W1, year, month)


x$id2 = x$yflag = x$zflag = x$wflag = rep(NA, nrow(x))


y.index = rep(NA, nrow(x))
z.index = rep(NA, nrow(x))
w.index = rep(NA, nrow(x))

for(i in 1:nrow(x)) {

  #compare to dataset y, insert yflag == 1 if the same ID, month, year is in x, otherwise 0 
  y.index = which(as.character(y$id2) == as.character(x$id[i]) 
                     & as.character(y$year) == as.character(x$year[i])
                     & as.character(y$month) == as.character(x$month[i])) 
  x$yflag[i] = ifelse(length(y.index==1), 1, 0)
  x$id2[i] = ifelse(length(y.index) == 1, y$id2[y.index], x$id[i])

  ## compare to dataset z, insert zflag == 1 if the same ID, month, year is in x, otherwise 0
  z.index <- which(as.character(z$id3) == as.character(x$id[i])
                   & as.character(z$month) == as.character(x$month[i])
                   & as.character(z$year) == as.character(x$year[i]))
  x$zflag[i] <- ifelse(length(z.index == 1), 1, 0)


  ## compare to dataset w, insert wflag == 1 if the same ID, month, year is in x, otherwise 0
  w.index <- which(as.character(w$id4) == as.character(x$id[i]) 
                   & as.character(w$month) == as.character(x$month[i])
                   & as.character(w$year) == as.character(x$year[i]))
  x$wflag[i] <- ifelse(length(w.index == 1), 1, 0)  
}

print(x)

【问题讨论】：

你试过merge()吗？
merge 没有正确标记观察结果，它会抛出信息，因为我看不到标记的等效项。例如test.merge = merge(x,y, by.x = 'id', by.y='id2') 并不能解决问题。当然，我可能没有正确实施它
你试过match()函数了吗？
match 返回 x 和 y 之间匹配的位置。例如test = match(x$id,y$id2)，但这并没有以更好的方式标记观察结果。同样匹配不允许多个“ID”，因此您不能使用月份和年份的信息。
您可以做的一件事是在每个 data.frame (x,y,z & w) 中添加 stringsAsFactors=FALSE。在你的功能之后，你可以调用例如which(y$id2 == x$id[i] & y$year == x$year[i] & y$month == x$month[i])而不是which(as.character(y$id2) == as.character(x$id[i]) & as.character(y$year) == as.character(x$year[i]) & as.character(y$month) == as.character(x$month[i]))

标签： r loops join merge match

【解决方案1】：

众多解决方案之一：
创建所有四个data.frames 后，

x$match.idx <- do.call(paste, c(x[,c("id", "month", "year")], sep=":"))
y$match.idx <- do.call(paste, c(y[,c("id2", "month", "year")], sep=":"))
z$match.idx <- do.call(paste, c(z[,c("id3", "month", "year")], sep=":"))
w$match.idx <- do.call(paste, c(w[,c("id4", "month", "year")], sep=":"))

xy.m <- match(x$match.idx, y$match.idx)
xz.m <- match(x$match.idx, z$match.idx)
xw.m <- match(x$match.idx, w$match.idx)
x$yflag <- x$zflag <- x$wflag <- 0
x$yflag[which(!is.na(xy.m))] <- 1
x$zflag[which(!is.na(xz.m))] <- 1
x$wflag[which(!is.na(xw.m))] <- 1

x <- subset(x, select=-c(match.idx))
> head(x)

  id         X1 month year wflag zflag yflag
1  1 -0.2470932   Jul 2004     1     1     1
2  2  0.2262816   Aug 2005     1     1     1
3  3  0.8473442   Sep 2006     1     1     1
4  4  0.9338628   Oct 2001     0     0     1
5  5 -0.1385540   Nov 2002     1     0     0
6  6  0.7825385   Dec 2004     1     0     0

【讨论】：

将索引列与paste() 结合起来的好主意！ +1
不错。 paste() 实际上比我使用的 interaction() 快得多。这是一个data.table 方法，假设您已经创建了所有data.frames 中的data.tables（例如DTx、DTy...）。我发现语法 much 更简洁：temp <- DTx[, paste(id, year, month)]; DTx[, ':='(wflag = as.numeric(temp %in% DTw[, paste(id4, year, month)]), zflag = as.numeric(temp %in% DTz[, paste(id3, year, month)]), yflag = as.numeric(temp %in% DTy[, paste(id2, year, month)]))]

【解决方案2】：

我建议将within() 和interaction() 组合如下：

output <- within(x, {
    temp <- interaction(id, month, year) # Something to match to
    # The actual matching takes place here
    # The `+0` at the end is a lazy way to convert
    #   TRUE and FALSE logical values to numeric 1 and 0
    wflag <- temp %in% with(w, interaction(id4, month, year)) + 0
    zflag <- temp %in% with(z, interaction(id3, month, year)) + 0
    yflag <- temp %in% with(y, interaction(id2, month, year)) + 0
    # Remove the temp variable that we created 
    #   since it's no longer required.
    rm(temp)
})

head(output)
#   id          X1 month year yflag zflag wflag
# 1  1 -0.03595218   Jul 2004     1     1     1
# 2  2  0.56329165   Aug 2005     1     1     1
# 3  3  0.74372988   Sep 2006     1     1     1
# 4  4  1.49634088   Oct 2001     1     0     0
# 5  5  0.23107131   Nov 2002     0     0     1
# 6  6  0.15121196   Dec 2004     0     0     1
tail(output)
#      id         X1 month year yflag zflag wflag
# 105  95 -0.0911546   Nov 2002     0     0     1
# 106  96 -0.4140724   Dec 2004     0     0     1
# 107  97 -0.1477702   Jan 2005     0     0     1
# 108  98 -0.3164388   Feb 2006     0     0     1
# 109  99 -0.5082118   Mar 2001     0     0     0
# 110 100 -0.6072856   Apr 2002     0     0     1

【讨论】：