R - 一次读取多个数据文件答案

【问题标题】：R - reading multiple data files at onceR - 一次读取多个数据文件
【发布时间】：2018-04-04 13:57:37
【问题描述】：

我正在研究 R 中的并行处理，想知道我是否可以并行读取多个 txt 文件而不是按顺序读取。这样做的原因是我有一个闪亮的应用程序，我想减少加载时间，并且很大一部分来自加载文件。

目前情况：

Shipments_Raw <- read.delim('/srv/samba/share/SAP data/_zmrosales_ship_month.txt', fill = TRUE)
ShipmentsYear_Raw <- read.delim('/srv/samba/share/SAP data/_zmrosales_ship_year.txt', fill = TRUE)
Open_Raw <- read.delim('/srv/samba/share/SAP data/_zmrosales_wip.txt', fill = TRUE)
WIP_Short_Raw <- read.delim('/srv/samba/share/SAP data/_zmro_short.txt', fill = TRUE)
WIP_RTQT_Raw <- read.delim('/srv/samba/share/SAP data/_zmro_sno_tasks_year.txt', fill = TRUE)
Invoiced_Raw <- read.delim('/srv/samba/share/SAP data/_zmrosales_inv.txt', fill = TRUE)

我见过并行运行的示例，但它们都以合并所有文件结束。我导入的每个文件，我想作为一个单独的数据框。

这里有一些例子：

How do you read in multiple .txt files into R?

https://www.r-bloggers.com/import-all-text-files-in-a-folder-with-parallel-execution/

理想情况（虽然我知道这不是代码）：

RunParallel {
    Shipments_Raw <- read.delim('/srv/samba/share/SAP data/_zmrosales_ship_month.txt', fill = TRUE)
    ShipmentsYear_Raw <- read.delim('/srv/samba/share/SAP data/_zmrosales_ship_year.txt', fill = TRUE)
    Open_Raw <- read.delim('/srv/samba/share/SAP data/_zmrosales_wip.txt', fill = TRUE)
    WIP_Short_Raw <- read.delim('/srv/samba/share/SAP data/_zmro_short.txt', fill = TRUE)
    WIP_RTQT_Raw <- read.delim('/srv/samba/share/SAP data/_zmro_sno_tasks_year.txt', fill = TRUE)
    Invoiced_Raw <- read.delim('/srv/samba/share/SAP data/_zmrosales_inv.txt', fill = TRUE)
}

下方评论后

    tic <- Sys.time()
Shipments_Raw <- read.delim('/srv/samba/share/SAP data/_zmrosales_ship_month.txt', fill = TRUE)
ShipmentsYear_Raw <- read.delim('/srv/samba/share/SAP data/_zmrosales_ship_year.txt', fill = TRUE)
Open_Raw <- read.delim('/srv/samba/share/SAP data/_zmrosales_wip.txt', fill = TRUE)
WIP_Short_Raw <- read.delim('/srv/samba/share/SAP data/_zmro_short.txt', fill = TRUE)
WIP_RTQT_Raw <- read.delim('/srv/samba/share/SAP data/_zmro_sno_tasks_year.txt', fill = TRUE)
Invoiced_Raw <- read.delim('/srv/samba/share/SAP data/_zmrosales_inv.txt', fill = TRUE)
toc <- Sys.time()
Sequential <- toc - tic


tic <- Sys.time()
file <- c("/srv/samba/share/SAP data//_zmrosales_ship_month.txt", 
          "/srv/samba/share/SAP data//_zmrosales_ship_year.txt", 
          "/srv/samba/share/SAP data//_zmrosales_inv.txt",
          "/srv/samba/share/SAP data//_zmrosales_wip.txt",
          "/srv/samba/share/SAP data//_zmro_short.txt",
          "/srv/samba/share/SAP data//_zmro_sno_tasks_year.txt")
x2 <- lapply(file, data.table::fread)

Shipments_Raw <- as.data.frame(x2[1])
ShipmentsYear_Raw <- as.data.frame(x2[2])
Invoiced_Raw <- as.data.frame(x2[3])
Open_Raw <- as.data.frame(x2[4])
WIP_Short_Raw <- as.data.frame(x2[5])
WIP_RTQT_Raw <- as.data.frame(x2[6])

toc <- Sys.time()
Lapply <- toc - tic

Sequential
Lapply

时间差异：

> Sequential
Time difference of 6.011156 secs
> Lapply
Time difference of 0.8015034 secs

【问题讨论】：

lapply(files, data.table::fread)
太棒了！我没有意识到这给了我元素中的数据集而不是合并它们。如果你把你的答案写在下面，我会把它标记为正确的

标签： r data.table lapply

【解决方案1】：

只需使用lapply 结合data.tables 超快fread：

lapply(files, data.table::fread)

【讨论】：