为什么我不能在 Python 中打开 .h5 文件？

Question

我正在尝试打开一个 .h5 文件，但遇到操作系统错误。

import sys
sys.path.append('..') 
from unet3d.training import load_old_model
import tables
from train_model import config


model_file=config["model_file"] #config["model_file"] = os.path.abspath("mc_seg_model.h5")
hdf5_file=config["val_data_file"] #config['val_data_file'] = os.path.abspath("../data/val_data.h5")

model = load_old_model(model_file)

load_model函数如下：

import math
from functools import partial
import pdb
from keras import backend as K
from keras.callbacks import ModelCheckpoint, CSVLogger, LearningRateScheduler, ReduceLROnPlateau, EarlyStopping
from keras.models import load_model
import tensorflow_addons as tfa

def load_old_model(model_file):
#     pdb.set_trace()
    print("Loading pre-trained model")
    custom_objects = {'dice_coefficient_loss': dice_coefficient_loss, 'dice_coefficient': dice_coefficient,
                      'weighted_dice_coefficient': weighted_dice_coefficient,
                      'weighted_dice_coefficient_loss': weighted_dice_coefficient_loss}
    try:
        #from keras_contrib.layers import InstanceNormalization
        from tfa.layers import InstanceNormalization
        custom_objects["InstanceNormalization"] = InstanceNormalization
    except ImportError:
        pass
    try:
        return load_model(model_file, custom_objects=custom_objects)
    except ValueError as error:
        if 'InstanceNormalization' in str(error):
            raise ValueError(str(error) + "\n\nPlease install keras-contrib to use InstanceNormalization:\n"
                                          "'pip install git+https://www.github.com/keras-team/keras-contrib.git'")
        else:
            raise error

当我尝试加载模型时，它会引发以下操作系统错误，并且是“输入/输出错误”。

2021-06-16 14:31:38.354199: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Traceback (most recent call last):
  File "draft.py", line 35, in <module>
    model = load_old_model(model_file)
  File "../unet3d/training.py", line 50, in load_old_model
    return load_model(model_file, custom_objects=custom_objects)
  File "/share/apps/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/saving/save.py", line 182, in load_model
    return hdf5_format.load_model_from_hdf5(filepath, custom_objects, compile)
  File "/share/apps/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/saving/hdf5_format.py", line 173, in load_model_from_hdf5
    model_config = f.attrs.get('model_config')
  File "/share/apps/anaconda3/lib/python3.7/_collections_abc.py", line 660, in get
    return self[key]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/share/apps/anaconda3/lib/python3.7/site-packages/h5py/_hl/attrs.py", line 81, in __getitem__
    attr.read(arr, mtype=htype)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5a.pyx", line 355, in h5py.h5a.AttrID.read
  File "h5py/_proxy.pyx", line 58, in h5py._proxy.attr_rw
OSError: Unable to read attribute (file read failed: time = Wed Jun 16 14:31:42 2021
, filename = '/data/kfernando/brats20/demo_task3_mcmc/mc_seg_model.h5', file descriptor = 4, errno = 5, error message = 'Input/output error', buf = 0x56126c096440, total read size = 30352, bytes this sub-read = 30352, bytes actually read = 18446744073709551615, offset = 16384)

谁能告诉我是什么导致了这个错误？

Answer 1

根据您关于成功打开/关闭 h5py 的 cmets，您似乎拥有一个有效的 HDF5 文件。还有 2 个问题需要调查：1) 读取属性数据的问题，或 2) TensorFlow load_model() 函数中的错误。我对 TF 无能为力。然而，这里有一些代码递归地降低数据层次结构并输出所有属性和值。见下文：

def get_all_attrs(name, h5_obj):
    
    if isinstance(h5_obj,h5py.Group):
        print('\n{} is a Group'.format(name))
    elif isinstance(h5_obj,h5py.Dataset):
        print('\n{} is a Dataset'.format(name))
    
    print('number of attributes:',len( h5_obj.attrs.keys() ))
    for k in h5_obj.attrs.keys():
        print('{} => {}'.format(k, h5_obj.attrs[k]))

with h5py.File(file_path, 'r') as h5r:
    print('number of root level attributes:',len( h5r.attrs.keys() ))
    for k in h5r.attrs.keys():
        print('{} => {}'.format(k, h5r.attrs[k]))        
    h5r.visititems(get_all_attrs)

用你的 TF 文件运行它。它可能会在读取其中一个属性时发现错误。我的测试文件的示例输出如下所示：

number of root level attributes: 2
OS => Windows
User => Me

Base_Group is a Group
number of attributes: 2
Date => today
Time => now

Base_Group/default is a Dataset
number of attributes: 2
attr1 => 1.0
attr2 => 22.2

Group1 is a Group
number of attributes: 0

Group1/default1 is a Dataset
number of attributes: 0

这应该有助于确定错误的来源。如果 h5py 可以读取属性，则需要研究 TF load_data() 函数。如果您在读取属性时遇到错误……嗯，那是您的问题，但我不知道如何确定根本原因。