每次迭代后强制 R 将循环输出写入列表答案

【问题标题】：Force R to write loop output to list after each iteration每次迭代后强制 R 将循环输出写入列表
【发布时间】：2022-01-07 20:49:03
【问题描述】：

在对形状应用一些函数后，我正在循环一个大型形状文件（xy 坐标）列表，我想将这些文件存储在 R 中的另一个列表中。

list_adj <- list()
for(ii in 1:length(data_list)) {

  ij <- names(data_list)[ii]
  M <- data_list[[ii]]

  fancy_obj <- function(){do_fancy_things}

  list_adj[[ij]] <- fancy_obj

  Sys.sleep(2)
}

我面临的问题是，R 不断循环遍历所有形状文件，而没有在每次迭代后将它们写入 list_adj 变量。相反，R 尝试在所有迭代完成后将所有 1600 个形状文件写入列表。这会导致我的 for 循环一直崩溃。有没有办法在每次迭代后强制 R 写入列表？

【问题讨论】：

1.6k 文件是否比您的内存大？您想在并行加载其他文件的同时处理前几个文件吗？运行所有内容需要几分钟到几小时吗？如果是这样，我推荐R targets
否，但 Rstudio 显示循环期间内存峰值约为 4 GB。虽然仍然崩溃。如果我使用数据子集运行循环，则循环工作得很好。
是否因为内存不足而崩溃？你只有4GB的机器吗？操作系统会杀死需要太多内存的进程。
我有 8 GB，但 4 GB 似乎是分配给 R 的最大数量。运行循环大约需要 10 分钟。
请在帖子中添加一些示例和预期输出。此循环似乎内存效率低下，基于先前的基准 stackoverflow.com/questions/42393658/…，它可以重写为 purrr::map oneliner 或本身变得更高效，从而解决问题。

标签： r for-loop

【解决方案1】：

你可以使用purrr::map函数代替循环：

数据：

data_list <- list(sf1 = structure(list(geometry = structure(list(structure(c(2.3111662, 
48.840346), class = c("XY", "POINT", "sfg"))), class = c("sfc_POINT", 
"sfc"), precision = 0, bbox = structure(c(xmin = 2.3111662, ymin = 48.840346, 
xmax = 2.3111662, ymax = 48.840346), class = "bbox"), crs = structure(list(
    input = "WGS84", wkt = "GEOGCRS[\"WGS 84\",\n    DATUM[\"World Geodetic System 1984\",\n        ELLIPSOID[\"WGS 84\",6378137,298.257223563,\n            LENGTHUNIT[\"metre\",1]]],\n    PRIMEM[\"Greenwich\",0,\n        ANGLEUNIT[\"degree\",0.0174532925199433]],\n    CS[ellipsoidal,2],\n        AXIS[\"geodetic latitude (Lat)\",north,\n            ORDER[1],\n            ANGLEUNIT[\"degree\",0.0174532925199433]],\n        AXIS[\"geodetic longitude (Lon)\",east,\n            ORDER[2],\n            ANGLEUNIT[\"degree\",0.0174532925199433]],\n    ID[\"EPSG\",4326]]"), class = "crs"), n_empty = 0L)), row.names = c(NA, 
-1L), sf_column = "geometry", agr = structure(integer(0), .Label = c("constant", 
"aggregate", "identity"), class = "factor", .Names = character(0)), class = c("sf", 
"tbl_df", "tbl", "data.frame")), sf2 = structure(list(geometry = structure(list(
    structure(c(2.36571460340002, 2.3655857662182, 2.36547329046861, 
    2.36542516193229, 2.36528545258358, 2.36512653636567, 2.3648060350023, 
    2.36424260145439, 2.36387560297359, 2.36366713159424, 2.36334443947327, 
    2.36322695208547, 2.36317764205549, 2.36318437079076, 2.36321528700506, 
    2.36326729012258, 2.36331020083956, 48.8776400092826, 48.8774711513306, 
    48.8773023799338, 48.8772238828371, 48.8770522688551, 48.8768904454866, 
    48.8765622871995, 48.8759189703608, 48.8754916354644, 48.8752575997409, 
    48.8748871565497, 48.874683281842, 48.8744788703593, 48.8742630655848, 
    48.8740761681905, 48.8739487391069, 48.8738959065238), .Dim = c(17L, 
    2L), class = c("XY", "LINESTRING", "sfg"))), class = c("sfc_LINESTRING", 
"sfc"), precision = 0, bbox = structure(c(xmin = 2.36317764205549, 
ymin = 48.8738959065238, xmax = 2.36571460340002, ymax = 48.8776400092826
), class = "bbox"), crs = structure(list(input = "WGS84", wkt = "GEOGCRS[\"WGS 84\",\n    DATUM[\"World Geodetic System 1984\",\n        ELLIPSOID[\"WGS 84\",6378137,298.257223563,\n            LENGTHUNIT[\"metre\",1]]],\n    PRIMEM[\"Greenwich\",0,\n        ANGLEUNIT[\"degree\",0.0174532925199433]],\n    CS[ellipsoidal,2],\n        AXIS[\"geodetic latitude (Lat)\",north,\n            ORDER[1],\n            ANGLEUNIT[\"degree\",0.0174532925199433]],\n        AXIS[\"geodetic longitude (Lon)\",east,\n            ORDER[2],\n            ANGLEUNIT[\"degree\",0.0174532925199433]],\n    ID[\"EPSG\",4326]]"), class = "crs"), n_empty = 0L)), row.names = 1L, class = c("sf", 
"data.frame"), sf_column = "geometry", agr = structure(integer(0), .Label = c("constant", 
"aggregate", "identity"), class = "factor", .Names = character(0))))

代码：

library(tidyverse)
list_adj <- map(1:length(data_list), function(x) data_list[[x]] %>%
                  st_crs()) # Replace st_crs() by anything, that is for the example.

names(list_adj) <- map(1:length(data_list), function(x) names(data_list)[[x]])

输出：

$sf1
Coordinate Reference System:
  User input: WGS84 
  wkt:
GEOGCRS["WGS 84",
    DATUM["World Geodetic System 1984",
        ELLIPSOID["WGS 84",6378137,298.257223563,
            LENGTHUNIT["metre",1]]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433]],
    CS[ellipsoidal,2],
        AXIS["geodetic latitude (Lat)",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433]],
        AXIS["geodetic longitude (Lon)",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433]],
    ID["EPSG",4326]]

$sf2
Coordinate Reference System:
  User input: WGS84 
  wkt:
GEOGCRS["WGS 84",
    DATUM["World Geodetic System 1984",
        ELLIPSOID["WGS 84",6378137,298.257223563,
            LENGTHUNIT["metre",1]]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433]],
    CS[ellipsoidal,2],
        AXIS["geodetic latitude (Lat)",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433]],
        AXIS["geodetic longitude (Lon)",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433]],
    ID["EPSG",4326]]

【讨论】：

谢谢 MonJeanJean。我可以确认这适用于我的数据。我发现地图功能的使用不是很直观，但我会习惯的。 ;) 再次感谢您的帮助！

【解决方案2】：

R targets 允许您读取文件并在其上应用 R 函数。它会一个接一个地处理一个文件，并且不会将所有内容加载到一个可能导致内存不足错误的大 R 列表中：

创建文件_targets.R:

# load all libraries needed to read and process the shapefiles
library(targets)
library(sf)

# do fancy things
modify_shape <- function(x) {
  x
}

list(
  tar_target(
    shape_files, {
      list.files(pattern = ".shp$", recursive = TRUE, full.names = TRUE)
    },
    format = "file"
  ),
  tar_target(
    raw_shapes, {
      read_sf(shape_files)
    },
     pattern = map(shape_files)
  ),
  tar_target(
    processed_shapes, {
      modify_shape(shape_files)
    },
     pattern = map(raw_shapes)
  ),
)

运行管道

tar_make()

加载结果

tar_load(processed_shapes)
processed_shapes

【讨论】：