直接从 S3 读取 xarray go16 数据，无需下载到系统中

Question

直接从 S3 读取 xarray go16 数据，无需下载到系统中。问题是我无法连接 S3Files。我正在从 S3 中调用 24 个文件，并希望在时间范围内读取和提取这些文件的数据：

这是代码：

import datetime as dt
import xarray as xr
import fsspec
import s3fs

fs = fsspec.filesystem('s3', anon=True)

urls1=[]

for i in range (2):
    urls = [
        's3://' + f
        for f in fs.glob(f"s3://noaa-goes16/ABI-L2ACMC/2022/001/{i:02}/*.nc")
    ]
    urls1 = urls1+ urls

with fs.open(urls1[0]) as fileObj:
    ds = xr.open_dataset(fileObj, engine='h5netcdf')

但是，我遇到了I/O operation on closed file 的问题。

Answer 1

与 python 中的大多数文件对象接口类似，使用上下文管理器打开类似文件的对象会在退出时关闭文件。所以在下面的例子中：

# use fs.open to create an S3File object
with fs.open(urls1[0], mode="rb") as fileObj:
    # open the netcdf for reading, but don't load the data - instead, just
    # establish a lazy-load connection to the underlying S3File object
    ds = xr.open_dataset(fileObj, engine='h5netcdf')

# <--
# exit the context, thereby closing the S3File object

# attempt to access the data again, after the stream is closed
ds.load()  # raises IOError

相反，您应该在上下文管理器中加载所有数据：

with fs.open(urls1[0], mode="rb") as fileObj:
    with xr.open_dataset(fileObj, engine='h5netcdf') as ds:
        ds = ds.load()

或者，如果您打算在以后的代码中使用数据集而不加载：

fileObj = fs.open(urls1[0], mode="rb")
ds = xr.open_dataset(fileObj, engine='h5netcdf')

# other data operations

# be sure to close the connections when you're done
ds.close()
fileObj.close()