使用 R 更正/整理数据答案

【问题标题】：Correcting / Tidying Data using R使用 R 更正/整理数据
【发布时间】：2017-07-13 13:02:30
【问题描述】：

我正在处理的交易数据来自两个来源，各有利弊。第一个 only 具有准确的 $ 和售出单位 (DF_Lookup)，第二个具有正确的人口统计 (DFI) 但某些 $ 和售出的单位不正确。因此，我编写了以下代码来处理这个问题。

这是我的数据：

DFI

dput(DFI)
structure(list(PO_ID = c("P1234", "P1234", "P1234", "P1234", 
"P1234", "P1234", "P2345", "P2345", "P3456", "P4567"), SO_ID = c("S1", 
"S1", "S1", "S2", "S2", "S2", "S3", "S4", "S7", "S10"), F_Year = c(2012, 
2012, 2012, 2013, 2013, 2013, 2011, 2011, 2014, 2015), Product_ID = c("385X", 
"385X", "385X", "450X", "450X", "900X", "3700", "3700", "A11U", 
"2700"), Revenue = c(1, 2, 3, 34, 34, 6, 7, 88, 9, 100), Quantity = c(1, 
2, 3, 8, 8, 6, 7, 8, 9, 40), Location1 = c("MA", "NY", "WA", 
"NY", "WA", "NY", "IL", "IL", "MN", "CA")), .Names = c("PO_ID", 
"SO_ID", "F_Year", "Product_ID", "Revenue", "Quantity", "Location1"
), row.names = c(NA, 10L), class = "data.frame")

DF_Lookup

dput(DF_Lookup)

structure(list(PO_ID = c("P1234", "P1234", "P1234", "P2345", 
"P2345", "P3456", "P4567"), SO_ID = c("S1", "S2", "S2", "S3", 
"S4", "S7", "S10"), F_Year = c(2012, 2013, 2013, 2011, 2011, 
2014, 2015), Product_ID = c("385X", "450X", "900X", "3700", "3700", 
"A11U", "2700"), Revenue = c(50, 70, 35, 100, -50, 50, 100), 
    Quantity = c(3, 20, 20, 20, -10, 20, 40)), .Names = c("PO_ID", 
"SO_ID", "F_Year", "Product_ID", "Revenue", "Quantity"), row.names = c(NA, 
7L), class = "data.frame")

第一次尝试：

策略： - 使用 Join 覆盖 DF_Lookup 中 DFI 中的条目

DF_Generated <- DFI %>% 
  left_join(DF_Lookup,by = c("PO_ID", "SO_ID", "F_Year", "Product_ID")) %>%
  dplyr::group_by(PO_ID, SO_ID, F_Year, Product_ID) %>%
  dplyr::mutate(Count = n()) %>%
  dplyr::ungroup()%>%
  dplyr::mutate(Revenue = Revenue.y/Count, Quantity = Quantity.y/Count) %>%
  dplyr::select(PO_ID:Product_ID,Location1,Revenue,Quantity)

预期输出：

dput(DF_Generated)
structure(list(PO_ID = c("P1234", "P1234", "P1234", "P1234", 
"P1234", "P1234", "P2345", "P2345", "P3456", "P4567"), SO_ID = c("S1", 
"S1", "S1", "S2", "S2", "S2", "S3", "S4", "S7", "S10"), F_Year = c(2012, 
2012, 2012, 2013, 2013, 2013, 2011, 2011, 2014, 2015), Product_ID = c("385X", 
"385X", "385X", "450X", "450X", "900X", "3700", "3700", "A11U", 
"2700"), Location1 = c("MA", "NY", "WA", "NY", "WA", "NY", "IL", 
"IL", "MN", "CA"), Revenue = c(16.6666666666667, 16.6666666666667, 
16.6666666666667, 35, 35, 35, 100, -50, 50, 100), Quantity = c(1, 
1, 1, 10, 10, 20, 20, -10, 20, 40)), class = c("tbl_df", "tbl", 
"data.frame"), row.names = c(NA, -10L), .Names = c("PO_ID", "SO_ID", 
"F_Year", "Product_ID", "Location1", "Revenue", "Quantity"))

挑战：这适用于较小的数据集。我正在处理的原始数据有大约 9000 万条记录。所以，上面的代码需要永远。

第二次尝试： 因此，我只想更新那些 $ 和单位超出 +/-10% 范围的行。

这是我的代码：

#Find out whether the numbers are within +/-10% range.
DF_Mod<-DFI %>%
  dplyr::group_by(PO_ID, SO_ID, F_Year, Product_ID) %>%
  dplyr::summarise(Rev_agg = sum(Revenue), Qty_agg = sum(Quantity)) %>%
  left_join(DF_Lookup) %>%
  dplyr::rowwise() %>%
  #check for +/- 10% confidence interval
  dplyr::mutate(Compute = ifelse((abs(Rev_agg-Revenue)/Revenue <=0.1) & (abs(Qty_agg-Quantity)/Quantity <=0.1),"N","Y")) %>%
  dplyr::rowwise() %>%
  dplyr::ungroup() %>%
  dplyr::select(PO_ID:Product_ID,Compute) %>%
  dplyr::right_join(DFI)

#Now, filter Compute == "Y" and then do the join with DF_Lookup.
DF_Generated_2 <- DF_Mod %>% 
  dplyr::filter(Compute == "Y") %>%  
  left_join(DF_Lookup,by = c("PO_ID", "SO_ID", "F_Year", "Product_ID")) %>%
  dplyr::group_by(PO_ID, SO_ID, F_Year, Product_ID) %>%
  dplyr::mutate(Count = n()) %>%
  dplyr::ungroup()%>%
  dplyr::mutate(Revenue = Revenue.y/Count, Quantity = Quantity.y/Count) %>%
  dplyr::select(PO_ID:Product_ID,Location1,Revenue,Quantity)

#Bind the rows
DF_Final <- rbind(DF_Generated_2,DFI[DF_Mod$Compute=="N",]) #Expected output

这里，DF_Final 确实是预期的输出。

问题：即使按照上述方法，由于涉及的连接太多，性能仍然很慢。无论如何我们可以加快这个过程吗？还有其他更好的方法来做我想做的事吗？

感谢您的想法。我花了一天的时间，仍然无处可去。我真的被困住了。

【问题讨论】：

您是否尝试过加入其他包中的功能？ stackoverflow.com/questions/4322219/…
@coletl - 谢谢。我不太熟悉使用data.table 和sqldf 进行连接。如果有人可以指导我，那真的会帮助我。

标签： r join dplyr tidyr

【解决方案1】：

我还没有在大型数据集上对此进行过测试，但它仍然可能是您所需要的（并且可能还有更快的方法来做到这一点）：

# load data table library
library(data.table)

# convert data frames to data tables
DFI <- data.table(DFI)
DF_Lookup <- data.table(DF_Lookup)

# left join
df <- merge(DFI, DF_Lookup, all.x = TRUE, by = c("PO_ID", "SO_ID", "F_Year", "Product_ID"))

# Calculate the strange quantity and revenue
df2 <- df[, list(Revenue = Revenue.y/.N, Quantity = Quantity.y/.N),
              by = list(PO_ID, SO_ID, F_Year, Product_ID)]

【讨论】：

【解决方案2】：

希望我能正确解释您的问题。

使用来自 DFI 的正确人口统计数据通过 data.table 执行查找

library(data.table)
setDT(DFI)
setDT(DF_Lookup)
cols <- c("PO_ID", "SO_ID", "F_Year", "Product_ID")
DF_Lookup[DFI, Location:=Location1, on=cols]

按 PO_ID、SO_ID、F_Year、Product_ID、位置计算平均收入和数量

DF_Lookup[, list(AveRevenue=mean(Revenue), AveQuantity=mean(Quantity)), 
    by=c(cols, "Location")]

您可能还想在其他包中探索更快的 mean 版本（搜索 SO）

【讨论】：