根据条件将多个 csv 文件拆分为块

Question

我有 400 多个非常大的 csv 文件（约百万行），它们的结构都相似：

1. 我只需要第 2 行和第 3 行的长标题
2. 第一次系列（始终以“Target1”开头）
3. 第二次系列（始终以“Target2”开头）

这是一个数据示例：

#multiple rows of header#
Target 1
Timestamp,X,Y,Z
1553972886851,0.017578,-0.003052,-0.971375
1553972886851,0.017883,-0.003662,-0.980408
1553972886851,0.016418,-0.003174,-0.977295
1553972886999,0.017151,-0.002808,-0.978088
1553972886999,0.016785,-0.003113,-0.977051
1553972886999,0.017883,-0.002197,-0.975830
1553972887096,0.017517,-0.003113,-0.976624
1553972887096,0.017883,-0.003113,-0.977966
1553972887096,0.017883,-0.002869,-0.978210
1553972887243,0.017151,-0.003113,-0.976135
1553972887243,0.018250,-0.003235,-0.975647
1553972887243,0.017273,-0.002991,-0.976257
1553972887340,0.018372,-0.003235,-0.977722
1553972887340,0.017761,-0.003235,-0.978027
Target 2
Timestamp,X,Y,Z
1553972886753,-0.411585,0.072409,-0.849848
1553972886753,-0.339177,-0.053354,-0.556402
1553972886753,-0.411585,-0.262957,-0.483994
1553972886855,-0.506860,-0.057165,-0.472561
1553972886855,-0.499238,-0.007622,-0.529726
1553972886855,-0.472561,-0.041921,-0.560213
1553972887002,-0.510671,-0.083841,-0.480183
1553972887002,-0.525915,-0.057165,-0.480183
1553972887002,-0.544969,-0.038110,-0.522104
1553972887098,-0.510671,-0.030488,-0.510671
1553972887098,-0.529726,-0.026677,-0.525915
1553972887098,-0.510671,-0.068598,-0.518293

我需要将每个 csv 文件分成这 3 个部分并相应地命名。

我设法完成了第 1 步）和第 3 步），但为第 2 步努力奋斗。

这是我为第 3 步所做的）：

fileNames <- basename(list.files(path = ".", all.files = FALSE, full.names = FALSE, recursive = TRUE, ignore.case = FALSE, include.dirs = FALSE))
extension <- "txt"
fileNumbers <- seq(fileNames)

for (fileNumber in fileNumbers) {

  newFileName <-  paste("Target2-", 
                        sub(paste("\\.", extension, sep = ""), "", fileNames[fileNumber]), 
                        ".", extension, sep = "")

  # read old data:

  Lines <- readLines(fileNames[fileNumber])
  ix <- which(Lines == "Target2")

  sample <- read.csv(fileNames[fileNumber],
                     header = TRUE,
                     sep = ",", skip= ix)

  # write old data to new files:
  write.table(sample, 
              newFileName,
              append = FALSE,
              quote = FALSE,
              sep = ",",
              row.names = FALSE,
              col.names = TRUE)

}

我很确定这不是最直接的方法，而且我无法使用这种方法获得目标 1 和目标 2 之间的数据。另外，这超级慢，我想知道是否有更高效的内存方法？

Answer 1

foo = function(filename) {
  cat("\nprocessing", filename, "...")
  x = readLines(con = filename) 
  idx = grepl("^Target", x)
  x = split(x[!idx], cumsum(idx)[!idx])[-1]
  invisible(lapply(seq_along(x), function(i) {
    write.table(
      x = x[[i]], 
      file = sub("\\.csv", paste0("_", i, ".csv"), filename),
      append =FALSE,
      row.names = FALSE,
      quote = FALSE, 
      col.names = FALSE)
  }))
}

files = list.files(path = "path/to/files", pattern = ".+\\.csv$")
lapply(files, foo)