来自 Xarray 的往返 Zarr 数据

Question

使用xarray，我使用ds.to_zarr() 将数据集写入S3，然后xr.open_zarr() 以查看是否得到相同的数据集。

xarray 中的数据集如下所示：

<xarray.Dataset>
Dimensions:                     (nv: 2, reference_time: 11, time: 11, x: 4608, y: 3840)
Coordinates:
  * reference_time              (reference_time) datetime64[ns] 2018-04-01T18:00:00 ...
  * x                           (x) float64 -2.304e+06 -2.303e+06 -2.302e+06 ...
  * y                           (y) float64 -1.92e+06 -1.919e+06 -1.918e+06 ...
  * time                        (time) datetime64[ns] 2018-04-01T19:00:00 ...
Dimensions without coordinates: nv
Data variables:
    time_bounds                 (time, nv) datetime64[ns] dask.array<shape=(11, 2), chunksize=(1, 2)>
    ProjectionCoordinateSystem  (time) |S64 b'' b'' b'' b'' b'' b'' b'' b'' ...
    T2D                         (time, y, x) float64 dask.array<shape=(11, 3840, 4608), chunksize=(1, 3840, 4608)>

然后我将它写到zarr 使用：

fs = s3fs.S3FileSystem(anon=False)
d = s3fs.S3Map(f_zarr, s3=fs)
ds.to_zarr(store=d, mode='w')

当我尝试在使用中读回它时：

ds2 = xr.open_zarr(d)

我回来了：

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-17-8198db1c8578> in <module>()
----> 1 ds2 = xr.open_zarr(d)

/opt/conda/lib/python3.6/site-packages/xarray/backends/zarr.py in open_zarr(store, group, synchronizer, auto_chunk, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables)
    476 
    477         variables = OrderedDict([(k, maybe_chunk(k, v))
--> 478                                  for k, v in ds.variables.items()])
    479         return ds._replace_vars_and_dims(variables)
    480     else:

/opt/conda/lib/python3.6/site-packages/xarray/backends/zarr.py in <listcomp>(.0)
    476 
    477         variables = OrderedDict([(k, maybe_chunk(k, v))
--> 478                                  for k, v in ds.variables.items()])
    479         return ds._replace_vars_and_dims(variables)
    480     else:

/opt/conda/lib/python3.6/site-packages/xarray/backends/zarr.py in maybe_chunk(name, var)
    471                 token2 = tokenize(name, var._data)
    472                 name2 = 'zarr-%s' % token2
--> 473                 return var.chunk(chunks, name=name2, lock=None)
    474             else:
    475                 return var

/opt/conda/lib/python3.6/site-packages/xarray/core/variable.py in chunk(self, chunks, name, lock)
    820             data = indexing.ImplicitToExplicitIndexingAdapter(
    821                 data, indexing.OuterIndexer)
--> 822             data = da.from_array(data, chunks, name=name, lock=lock)
    823 
    824         return type(self)(self.dims, data, self._attrs, self._encoding,

/opt/conda/lib/python3.6/site-packages/dask/array/core.py in from_array(x, chunks, name, lock, asarray, fancy, getitem)
   1977     >>> a = da.from_array(x, chunks=(1000, 1000), lock=True)  # doctest: +SKIP
   1978     """
-> 1979     chunks = normalize_chunks(chunks, x.shape)
   1980     if name in (None, True):
   1981         token = tokenize(x, chunks)

/opt/conda/lib/python3.6/site-packages/dask/array/core.py in normalize_chunks(chunks, shape)
   1907             raise ValueError(
   1908                 "Chunks and shape must be of the same length/dimension. "
-> 1909                 "Got chunks=%s, shape=%s" % (chunks, shape))
   1910 
   1911     if shape is not None:

ValueError: Chunks and shape must be of the same length/dimension. Got chunks=(11, 64), shape=(11,)

如果我设置auto_chunk=False，我可以读取数据集：

ds2 = xr.open_zarr(d, auto_chunk=False)
ds2 

结果

<xarray.Dataset>
Dimensions:                     (nv: 2, reference_time: 11, time: 11, x: 4608, y: 3840)
Coordinates:
  * reference_time              (reference_time) datetime64[ns] 2018-04-01T18:00:00 ...
  * time                        (time) datetime64[ns] 2018-04-01T19:00:00 ...
  * x                           (x) float64 -2.304e+06 -2.303e+06 -2.302e+06 ...
  * y                           (y) float64 -1.92e+06 -1.919e+06 -1.918e+06 ...
Dimensions without coordinates: nv
Data variables:
    LWDOWN                      (time, y, x) float64 ...

但是我是否不了解分块以及 xarray、dask 和 zarr 应该一起工作的方式？

要让auto_chunk=True 工作，我需要做些什么不同的事情？

Answer 1

正如@mdurant 所建议的，变量ProjectionCoordinateSystem 和dtype=S64 导致了问题。由于我不需要这个非标准变量，只需用

删除它

ds.drop(['ProjectionCoordinateSystem']) 

在ds.to_zarr 解决问题之前，允许ds.open_zarr() 与默认autochunk=True 一起正常工作。

完整的笔记本在这里：https://gist.github.com/rsignell-usgs/4a54ea152d4e10a14deff516bf597015