在 h5py 中保存 Inception_Resnet_v2 功能

Question

我目前正在尝试在 keras 中使用一些预训练的 ImageNet 网络来从图像中提取特征。为此，我将移除网络的顶层，根据每个网络要求对输入进行预处理，然后将输出保存在 hdf5 文件中。我使用了其他几个使用完全相同的方法和代码（仅切换网络）的预训练网络，它似乎工作得很好。但是，我正在努力工作的网络是“InceptionResNetV2”。我不相信我在网络上遇到任何问题，只是通过保存 - 我附上了一个稍微简化的代码版本。更改特征提取器中的模型和预处理中的模型意味着它可以完美运行 - 对于 vgg16、vgg19、resnet、inception 等 - 都很好。

db = h5py.File(hdf5_path, mode="w")

featuresDB = db.create_dataset("features", shape=features_shape, dtype="float")

images = [cv2.imread(path, 1) for path in image_paths[start:end]]

images = inception_resnet_v2.preprocess_input(images)

features = feature_extractor.extract(images)

featuresDB[start:end] = features

但是，这会产生以下错误。我试图将进入 featuresDB 的数据的 dtype 更改为 int，但这没有效果。任何建议表示赞赏！

File "h5py/utils.pyx", line 101, in h5py.utils.convert_tuple
TypeError: an integer is required

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "Extract_Features.py", line 111, in <module>
    extract_features(image_paths, hdf5_path=args["features_db"], 
feature_extractor=feature_extractor)

File "Extract_Features.py", line 83, in extract_features
    featuresDB = db.create_dataset("features", shape=features_shape, dtype="float")

File "/home/will/tensorflow/lib/python3.5/site-packages/h5py/_hl/group.py", line 
106, in create_dataset
    dsid = dataset.make_new_dset(self, shape, dtype, data, **kwds)

File "/home/will/tensorflow/lib/python3.5/site-packages/h5py/_hl/dataset.py", line 137, in make_new_dset
    sid = h5s.create_simple(shape, maxshape)

File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper

File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper

File "h5py/h5s.pyx", line 95, in h5py.h5s.create_simple

File "h5py/utils.pyx", line 103, in h5py.utils.convert_tuple
TypeError: Can't convert element 1 (None) to hsize_t

Answer 1

In [201]: f = h5py.File('test.h5','w')

我可以用这个表达式重现你的错误：

In [203]: ds = f.create_dataset('features', shape=(None,3), dtype=float)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
h5py/utils.pyx in h5py.utils.convert_tuple()

TypeError: an integer is required

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-203-a5733d841c5c> in <module>()
----> 1 ds = f.create_dataset('features', shape=(None,3), dtype=float)

~/.local/lib/python3.6/site-packages/h5py/_hl/group.py in create_dataset(self, name, shape, dtype, data, **kwds)
    104         """
    105         with phil:
--> 106             dsid = dataset.make_new_dset(self, shape, dtype, data, **kwds)
    107             dset = dataset.Dataset(dsid)
    108             if name is not None:

~/.local/lib/python3.6/site-packages/h5py/_hl/dataset.py in make_new_dset(parent, shape, dtype, data, chunks, compression, shuffle, fletcher32, maxshape, compression_opts, fillvalue, scaleoffset, track_times)
    135         sid = h5s.create(h5s.NULL)
    136     else:
--> 137         sid = h5s.create_simple(shape, maxshape)
    138 
    139 

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/h5s.pyx in h5py.h5s.create_simple()

h5py/utils.pyx in h5py.utils.convert_tuple()

TypeError: Can't convert element 0 (None) to hsize_t

None 可以在 maxshape 参数中使用，但 shape 不行 - 这是用于可调整大小的数据集：

In [204]: ds = f.create_dataset('features', shape=(10,3), maxshape=(None,3), dty
     ...: pe=float)
In [205]: ds
Out[205]: <HDF5 dataset "features": shape (10, 3), type "<f8">
In [206]: ds.resize((20,3))
In [207]: ds
Out[207]: <HDF5 dataset "features": shape (20, 3), type "<f8">

我没有使用过keras，但从其他 SO 问题和文档来看，None 允许在其shape 中使用。 None 不允许在 numpy 数组形状中。