hdf5文件读取速度突然变慢

Question

我正在尝试遍历 hdf5 文件。该文件由一系列“时间戳”组成，每个时间戳都有可变数量的“玩家”，我想从中获取数据。该文件非常大，有超过 17k 的时间戳，我观察到在经过数百次迭代后，获取“players = list(data[timestamp].keys())”所花费的时间从大约 0.0005 秒突然增加了很多大约为 0.05。

        with h5py.File(self.hdf5file, "r") as f:
            data = f['data']
            timestamps = list(data.keys())
            for timestamp in timestamps:
                start_time = time.time()
                players = list(data[timestamp].keys())
                end_time = time.time()
                print(end_time - start_time)

我不知道会发生什么以及如何解决它。

Answer 1

我怀疑还有比您的简单示例捕获的更多内容。当您只是读取键以获取组和数据集名称时，它不应显示性能下降。

为了演示，我编写了一个示例，该示例创建了一个模仿您的架构的文件。创建文件后，它会以只读模式关闭并重新打开，然后循环访问每个时间戳组和播放器数据集并打印访问时间。所有时间步长的响应时间始终很短。 [示例文件为 1.4 GB。如果您想测试更大的文件，请增加 arr 的大小。] 仅供参考，我是在具有 24 GB RAM 的 Windows 系统上运行的。

代码如下：

ntimes = 10000
nplayers = 10
hdf5file = 'SO_70466356.h5'
start_time = time.time()
with h5py.File(hdf5file, "w") as h5f:
    data = h5f.create_group('data')
    for t_cnt in range(ntimes):
        grp = data.create_group(f'time_{t_cnt:05}')
        for p_cnt in range(nplayers):
            arr = np.random.random(300*3).reshape(300,3)
            grp.create_dataset(f'player_{p_cnt:02}',data=arr)
print(f'HDF5 file creation time: {time.time()-start_time:.4f}')            

print(f'HDF5 file read times')            
with h5py.File(hdf5file, "r") as h5f:
    data = h5f['data']
    start_time = time.time()
    timestamps = list(data.keys())
    for cnt,timestamp in enumerate(timestamps):            
        players = list(data[timestamp].keys())
        if not (cnt+1)%1000 :
            print(f'For {timestamp}: {time.time()-start_time:.4f}')
            start_time = time.time()

上面的输出：

HDF5 file creation time: 29.7558
HDF5 file read times
At time_00999: 0.1766
At time_01999: 0.1736
At time_02999: 0.1650
At time_03999: 0.1872
At time_04999: 0.1716
At time_05999: 0.1740
At time_06999: 0.1716
At time_07999: 0.1716
At time_08999: 0.1720
At time_09999: 0.1872