问题现象:
最近在用pandas分析数据时,用hdf5存储结果,当我监听不同文件时,多个进程同时写入hdf5(save到不同group)时,报hdf5 AttributeError: 'UnImplemented' object has no attribute 'read',之后用ipython去查数据,报了以下错误
HDF5ExtError: HDF5 error back trace
 File "H5Dio.c", line 173, in H5Dread
   can't read data
 File "H5Dio.c", line 554, in H5D__read
   can't read data
 File "H5Dchunk.c", line 1875, in H5D__chunk_read
   unable to read raw data chunk
 File "H5Dchunk.c", line 2905, in H5D__chunk_lock
   data pipeline read failed
 File "H5Z.c", line 1372, in H5Z_pipeline
   filter returned failure during read
End of HDF5 error back trace
Problems reading records.

解决方案:
利用文件锁(fcntl模块)来保证数据的同步。
后来买了本书《Python和HDF5大数据应用》,里面介绍可以用mpio驱动来处理多进程的问题

相关文章:

  • 2021-08-31
  • 2021-12-04
  • 2021-07-25
  • 2021-06-22
  • 2021-06-16
  • 2021-08-06
  • 2021-06-13
  • 2022-01-08
猜你喜欢
  • 2022-12-23
  • 2021-09-15
  • 2021-12-07
  • 2021-12-14
  • 2021-06-06
  • 2021-07-27
  • 2021-06-21
相关资源
相似解决方案