R - 向量内存耗尽（达到限制？）嵌套循环的内存问题？

Question

我目前正在尝试编写一个 R 脚本来导入我创建的与数据集相关的各种文件。这涉及根据我如何组织文件的目录和名称，使用几个嵌套的 for 循环读取大量 .txt 文件。

我可以很好地运行最里面的循环（虽然有点慢）。但是，尝试运行第二个循环或任何其他循环会产生以下错误：

Error: vector memory exhausted (limit reached?)

我相信这可能与R如何处理内存有关？我正在用 Rstuidio 运行 R。我也尝试了here 发布的解决方案，但没有成功

'R
 R version 3.5.1 (2018-07-02) -- "Feather Spray"
 Platform: x86_64-apple-darwin15.6.0 (64-bit)

代码如下

subjects <- 72
loop1_names <- as.character(list('a','b','c'))
loop2_names <- as.character(list('one','two','three'))
loop3_names <- as.character(list('N1','N2'))
loop4_names<- as.character(list('choice1','choice2','choice3'))
i<-1;j<-1;

loop3.subset<- data.frame
for(k in 1:length(loop3_names)){

  loop4.subset<- data.frame()#Data frame for handling each set of loop 4 values
  for(l in 1:length(loop4_names)){

            #Code for extracting the variables for each measure

            measures.path <- file.path(results_fldr, 'amp_measures',loop1_names[i],loop2_names[j],'mont',loop3_names[k])
            measures.data <- read.table(file.path(measures.path, paste(paste(loop1_names[i],loop2_names[j],loop3_names[k],loop4_names[l],sep = '_'),'.txt',sep = '')), header = T, nrows = subjects)

            #Get rid of the IDs, we'll add those back in later
            col_idx_ID <- grep('ID', names(measures.data))
            measures.data <- as.data.frame(measures.data[,-col_idx_ID])# make sure when trimming to keep the measures as a data frame
            names(measures.data) <- c(paste(loop1_names[i],loop2_names[j],loop3_names[k],loop4_names[l],sep = '_'))#Add a label to the data

            #Now combine this data with the other data in the loop4 subset data frame
            if(l == 1){
              loop4.subset <- measures.data
            } else {
              loop4.subset <- merge(erp.subset,measures.data)
            }

          }#End l/loop 4
          if(k == 1){
            loop3.subset <- loop4.subset
          } else {
            freq.subset <- merge(loop3.subset,loop4.subset)
          }

        }#End k/loop 3

Answer 1

通常我建议您只将部分数据读入内存，然后将部分合并写入磁盘。在下面的示例中，我当然无法运行，因为我没有您的文件。我在每个 i, j 循环后写入磁盘，然后完成后有 9 个文件。现在您将这 6 个文件合并到另一个循环中。如果您仍然有内存问题，请先执行“j”合并并将每个文件写入 3 个“i”文件，然后将其分成另外 2 个文件。然后，如果您无法合并这些文件，那么您的机器内存不足就是一个根本问题。

subjects <- 72
loop1_names <- as.character(list('a','b','c'))
loop2_names <- as.character(list('one','two','three'))
loop3_names <- as.character(list('N1','N2'))
loop4_names<- as.character(list('choice1','choice2','choice3'))

for(i in 1:length(loop1_names)) {
    for(j in 1:length(loop2_names)) {
        loop3.subset<- data.frame
        for(k in 1:length(loop3_names)){

            loop4.subset<- data.frame()
            for(l in 1:length(loop4_names)){

                ##Code for extracting the variables for each measure

                measures.path <- file.path(results_fldr,
                                           'amp_measures',
                                           loop1_names[i],
                                           loop2_names[j],
                                           'mont',
                                           loop3_names[k])
                measures.data <- read.table(file.path(measures.path, paste(paste(loop1_names[i],
                                                                                 loop2_names[j],
                                                                                 loop3_names[k],
                                                                                 loop4_names[l],
                                                                                 sep = '_'),'.txt',sep = '')),
                                            header = T, nrows = subjects)

                ##Get rid of the IDs, we'll add those back in later
                col_idx_ID <- grep('ID', names(measures.data))
                measures.data <- as.data.frame(measures.data[,-col_idx_ID])
                names(measures.data) <- c(paste(loop1_names[i],
                                                loop2_names[j],
                                                loop3_names[k],
                                                loop4_names[l],
                                                sep = '_'))

                ## Now combine this data with the other data in the loop4 subset data frame
                if(l == 1){
                    loop4.subset <- measures.data
                } else {
                    loop4.subset <- merge(erp.subset,measures.data)
                }

            }#End l/loop 4
            if(k == 1){
                loop3.subset <- loop4.subset
            } else {
                freq.subset <- merge(loop3.subset,loop4.subset)
            }
        }#End k/loop 3
        write.table(freq.subset, paste0(i, "_", j, ".txt"))
    }
}

## Now you have 6 files to read in a merge.
## Something like this:

df <- NULL
for(i in 1:length(loop1_names)) {
    for(j in 1:length(loop2_names)) {
        df1 <- read.table(paste0(i, "_", j, ".txt"))
        df <- merge(df, df1)
    }
}