CapsNet 模型评估与 K-Fold 验证 - 如何获得最佳模型和评估

Question

我试图实现CapsuleNet 对一些数字进行分类。所有图像均为RGB图像转灰度并调整为32 X 32，数据集有10分类输出。

X_train_all.shape：(2075, 32, 32, 1)
y_train_all.shape：(2075, 10)

以下是我目前尝试过的。

CapsNet 模型

首先，定义CapsNet 模型。以下是 Capsule Network 的主要架构，包括 PrimaryCaps 和 DigitCaps 等。

def CapsNet(input_shape, n_class, routings):

    x = layers.Input(shape=input_shape)

    # Layer 1
    conv1 = layers.Conv2D()(x)

    # Layer 2
    primarycaps = PrimaryCap()

    # Layer 3
    digitcaps = CapsuleLayer()(primarycaps)

    # Layer 4
    out_caps = Length(name='capsnet')(digitcaps)

    # Decoder network.
    y = layers.Input()
    masked_by_y = Mask()([digitcaps, y])  
    masked = Mask()(digitcaps)  

    # Shared Decoder model in training and prediction
    decoder = models.Sequential(name='decoder')
    decoder.add(layers.Dense(512, activation='relu', input_dim=16*n_class))
    decoder.add(layers.Dense(1024, activation='relu'))
    decoder.add(layers.Dense(np.prod(input_shape), activation='sigmoid'))
    decoder.add(layers.Reshape(target_shape=input_shape, name='out_recon'))

    # Models for training and evaluation (prediction)
    train_model = models.Model([x, y], [out_caps, decoder(masked_by_y)])
    evals_model = models.Model(x, [out_caps, decoder(masked)])

    return train_model, evals_model

实际训练

这只是返回train_model 和eval_model。下面是我实施的实际训练过程。

def train_caps(model, data, epoch_size_frac=1.0):

    # unpacking the data
    (x_train, y_train), (x_val, y_val) = data

    # compile the model
    model.compile (....)

    # --------------Begin Training with data augmentation --------------
    def train_generator (...)


    # Training with data augmentation. 
    history = model.fit_generator (...)

    return model

K 折交叉验证

现在为了训练模型并在其上拟合数据，我使用了 K-Fold 交叉验证方法。假设它是K-Fold = 5。像下面的代码，我们保存5折模型并保存权重。

cvscores = []

for train, val in kfold.split(X_train_all, y_train_all):

    print ('Fold: ', Fold)

    # define model
    model, eval_model = CapsNet ( ... )


    X_train = X_train_all[train]
    X_val = X_train_all[val]

    y_train = y_train_all[train]
    y_val = y_train_all[val]


#   train -
    train_caps( ... ) # calling actual training 


#     # Save each fold model
    model_name = 'Fold_'+ str(Fold) + '.h5'
    model.save(model_name)

    # evaluate the model
    scores = model.evaluate(X_val, y_val, verbose = 0)
    print("%s: %.2f%%" % (model.metrics_names[7], scores[3]*100))
    cvscores.append(scores[3] * 100) 

    Fold = Fold + 1

面临的问题 1

问题出现在评估部分。 scores = model.evaluate(X_val, y_val, verbose = 0) 它显示：

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-36-12f206477b39> in <module>()
----> 1 scores = model.evaluate(X_val, Y_val, verbose = 0)
      2 print("%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))

ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), but instead got the following list of 1 arrays: [array([[[[218.],
         [1.],
         [0.],
         ...,
         [1.],
         [1.],

还有其他方法可以评估模型性能并衡量分数吗？

面临的问题 2

在这种方法中，如何找到最佳模型？我在想，我们可以加载顶部（假设）2 weighted fold 模型并获得平均分值或类似的集成方法。以下是我迄今为止尝试过的。

def ensemble(models, model_input):

    Models_output = [model(model_input) for model in models]
    Avg = keras.layers.average(Models_output)

    model_Ensemble = Model(inputs = model_input, outputs = Avg, name = 'ens')
    model_Ensemble.compile( ... )

    return modelEnsemble

并加载我们得到K-Fold交叉验证方法的保存加权。

import keras

model_1, eval_model_1 = CapsNet(input_shape=[32, 32, 1],
                n_class=10,
                routings=3)

model_2, eval_model_2 = CapsNet()

models = []

# Load weights 
model_1.load_weights('Fold_1.h5')
model_1.name = 'model_1'
models.append(model_1)

model_2.load_weights('Fold_2.h5')
model_2.name = 'model_2'
models.append(model_2)

model_input = Input(shape=models[0].input_shape[1:])
ensemble_model = ensemble(models, model_input)

这会引发以下错误。我知道，我在这里遗漏了一些东西，但不知道如何管理它。

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-37-8cec3f452a69> in <module>()
      4 model_1, eval_model_1 = CapsNet(input_shape=[32, 32, 1],
      5                 n_class=10,
----> 6                 routings=3)
      7 
      8 model_2, eval_model_2 = CapsNet(input_shape=[32, 32, 1],

<ipython-input-31-d96b4a5e15ad> in CapsNet(input_shape, n_class, routings)
     44 
     45     # Shared Decoder model in training and prediction
---> 46     decoder = models.Sequential(name='decoder')
     47     decoder.add(layers.Dense(512, activation='relu', input_dim=16*n_class))
     48     decoder.add(layers.Dense(1024, activation='relu'))

AttributeError: 'list' object has no attribute 'Sequential'

如果我对我的问题或我遇到的问题做一个简短的总结 - 我无法使用 model.evaluate(...,...) 方法评估模型性能。并进一步得到这个归因错误。

非常感谢任何帮助或建议。谢谢。

Answer 1

train_model 的以下行输入是两个数组的列表。

train_model = models.Model([x, y], [out_caps, decoder(masked_by_y)])

但是当您在 K-Fold 交叉验证部分中调用 model.evaluate 时，您将单个 numpy 数组作为输入（X_val）传递。

scores = model.evaluate(X_val, y_val, verbose = 0)

在这种情况下，我还想知道您是要使用训练模型还是验证模型。从您调用该方法的方式以及您评估模型的意图，可以推断您可能想要调用 eval_model 的评估方法。

scores = eval_model.evaluate(X_val, y_val, verbose = 0)