为什么我在训练期间有很好的 val_acc，但在相同的图像上总是错误的手动预测

Question

我使用val_acc=0.97 和model.fit_generator 训练了我的 CNN 网络模型。

这是最后一个 epoch 的输出，证明验证准确度很高：

199/200 [============================>.] - ETA: 1s - loss: 0.1563 - acc: 0.9563
200/200 [==============================] - 306s 2s/step - loss: 0.1556 - acc: 0.9565 - val_loss: 0.1402 - val_acc: 0.9691

Epoch 00005: val_acc improved from 0.96701 to 0.96907, saving model to /home/sergorl/cars/color_weights.hdf5

但是，当我使用我在训练期间使用的相同验证数据集，但只测试一张图像时，对于我验证集中的每张图像，我总是得到错误的预测标签，并且预测概率看起来像一个均匀分布。

我阅读了以下链接： Wrong prediction on images

Why is Keras training well but returning wrong predictions?

Keras Val_acc is good but prediction for same data is poor

但我没有找到解决办法！

from keras.models import Sequential,Model,load_model
from keras.optimizers import SGD
from keras.layers import BatchNormalization, Lambda, Input, Dense, Convolution2D, MaxPooling2D, AveragePooling2D, ZeroPadding2D, Dropout, Flatten, merge, Reshape, Activation
from keras.layers.merge import Concatenate
from keras.preprocessing.image import ImageDataGenerator
from keras.callbacks import ModelCheckpoint
import os
import cv2
import numpy as np


class CarColorNet:

    def __init__(self, numClasses=6, imageWidth=256, imageHeight=256):

        self.classes = {}
        self.numClasses = numClasses
        self.imageWidth = imageWidth
        self.imageHeight = imageHeight

        input_image = Input(shape=(self.imageWidth, self.imageHeight, 3))

        # ------------------------------------ TOP BRANCH ------------------------------------
        # first top convolution layer
        top_conv1 = Convolution2D(filters=48, kernel_size=(11, 11), strides=(4, 4),
                                  input_shape=(self.imageWidth, self.imageHeight, 3), activation='relu')(input_image)
        top_conv1 = BatchNormalization()(top_conv1)
        top_conv1 = MaxPooling2D(pool_size=(3, 3), strides=(2, 2))(top_conv1)

        # second top convolution layer
        # split feature map by half
        top_top_conv2 = Lambda(lambda x: x[:, :, :, :24])(top_conv1)
        top_bot_conv2 = Lambda(lambda x: x[:, :, :, 24:])(top_conv1)

        top_top_conv2 = Convolution2D(filters=64, kernel_size=(3, 3), strides=(1, 1), activation='relu',
                                      padding='same')(top_top_conv2)
        top_top_conv2 = BatchNormalization()(top_top_conv2)
        top_top_conv2 = MaxPooling2D(pool_size=(3, 3), strides=(2, 2))(top_top_conv2)

        top_bot_conv2 = Convolution2D(filters=64, kernel_size=(3, 3), strides=(1, 1), activation='relu',
                                      padding='same')(top_bot_conv2)
        top_bot_conv2 = BatchNormalization()(top_bot_conv2)
        top_bot_conv2 = MaxPooling2D(pool_size=(3, 3), strides=(2, 2))(top_bot_conv2)

        # third top convolution layer
        # concat 2 feature map
        top_conv3 = Concatenate()([top_top_conv2, top_bot_conv2])
        top_conv3 = Convolution2D(filters=192, kernel_size=(3, 3), strides=(1, 1), activation='relu',
                                  padding='same')(top_conv3)

        # fourth top convolution layer
        # split feature map by half
        top_top_conv4 = Lambda(lambda x: x[:, :, :, :96])(top_conv3)
        top_bot_conv4 = Lambda(lambda x: x[:, :, :, 96:])(top_conv3)

        top_top_conv4 = Convolution2D(filters=96, kernel_size=(3, 3), strides=(1, 1), activation='relu',
                                      padding='same')(top_top_conv4)
        top_bot_conv4 = Convolution2D(filters=96, kernel_size=(3, 3), strides=(1, 1), activation='relu',
                                      padding='same')(top_bot_conv4)

        # fifth top convolution layer
        top_top_conv5 = Convolution2D(filters=64, kernel_size=(3, 3), strides=(1, 1), activation='relu',
                                      padding='same')(top_top_conv4)
        top_top_conv5 = MaxPooling2D(pool_size=(3, 3), strides=(2, 2))(top_top_conv5)

        top_bot_conv5 = Convolution2D(filters=64, kernel_size=(3, 3), strides=(1, 1), activation='relu',
                                      padding='same')(top_bot_conv4)
        top_bot_conv5 = MaxPooling2D(pool_size=(3, 3), strides=(2, 2))(top_bot_conv5)

        # ------------------------------------ TOP BOTTOM ------------------------------------
        # first bottom convolution layer
        bottom_conv1 = Convolution2D(filters=48, kernel_size=(11, 11), strides=(4, 4),
                                     input_shape=(224, 224, 3), activation='relu')(input_image)
        bottom_conv1 = BatchNormalization()(bottom_conv1)
        bottom_conv1 = MaxPooling2D(pool_size=(3, 3), strides=(2, 2))(bottom_conv1)

        # second bottom convolution layer
        # split feature map by half
        bottom_top_conv2 = Lambda(lambda x: x[:, :, :, :24])(bottom_conv1)
        bottom_bot_conv2 = Lambda(lambda x: x[:, :, :, 24:])(bottom_conv1)

        bottom_top_conv2 = Convolution2D(filters=64, kernel_size=(3, 3), strides=(1, 1), activation='relu',
                                         padding='same')(bottom_top_conv2)
        bottom_top_conv2 = BatchNormalization()(bottom_top_conv2)
        bottom_top_conv2 = MaxPooling2D(pool_size=(3, 3), strides=(2, 2))(bottom_top_conv2)

        bottom_bot_conv2 = Convolution2D(filters=64, kernel_size=(3, 3), strides=(1, 1), activation='relu',
                                         padding='same')(bottom_bot_conv2)
        bottom_bot_conv2 = BatchNormalization()(bottom_bot_conv2)
        bottom_bot_conv2 = MaxPooling2D(pool_size=(3, 3), strides=(2, 2))(bottom_bot_conv2)

        # third bottom convolution layer
        # concat 2 feature map
        bottom_conv3 = Concatenate()([bottom_top_conv2, bottom_bot_conv2])
        bottom_conv3 = Convolution2D(filters=192, kernel_size=(3, 3), strides=(1, 1), activation='relu',
                                     padding='same')(bottom_conv3)

        # fourth bottom convolution layer
        # split feature map by half
        bottom_top_conv4 = Lambda(lambda x: x[:, :, :, :96])(bottom_conv3)
        bottom_bot_conv4 = Lambda(lambda x: x[:, :, :, 96:])(bottom_conv3)

        bottom_top_conv4 = Convolution2D(filters=96, kernel_size=(3, 3), strides=(1, 1), activation='relu',
                                         padding='same')(bottom_top_conv4)
        bottom_bot_conv4 = Convolution2D(filters=96, kernel_size=(3, 3), strides=(1, 1), activation='relu',
                                         padding='same')(bottom_bot_conv4)

        # fifth bottom convolution layer
        bottom_top_conv5 = Convolution2D(filters=64, kernel_size=(3, 3), strides=(1, 1), activation='relu',
                                         padding='same')(bottom_top_conv4)
        bottom_top_conv5 = MaxPooling2D(pool_size=(3, 3), strides=(2, 2))(bottom_top_conv5)

        bottom_bot_conv5 = Convolution2D(filters=64, kernel_size=(3, 3), strides=(1, 1), activation='relu',
                                         padding='same')(bottom_bot_conv4)
        bottom_bot_conv5 = MaxPooling2D(pool_size=(3, 3), strides=(2, 2))(bottom_bot_conv5)

        # ---------------------------------- CONCATENATE TOP AND BOTTOM BRANCH ------------------------------------
        conv_output = Concatenate()([top_top_conv5, top_bot_conv5, bottom_top_conv5, bottom_bot_conv5])

        # Flatten
        flatten = Flatten()(conv_output)

        # Fully-connected layer
        FC_1 = Dense(units=4096, activation='relu')(flatten)
        FC_1 = Dropout(0.6)(FC_1)
        FC_2 = Dense(units=4096, activation='relu')(FC_1)
        FC_2 = Dropout(0.6)(FC_2)
        output = Dense(units=self.numClasses, activation='softmax')(FC_2)

        self.model = Model(inputs=input_image, outputs=output)
        sgd = SGD(lr=1e-3, decay=1e-6, momentum=0.9, nesterov=True)
        # sgd = SGD(lr=0.01, momentum=0.9, decay=0.0005, nesterov=True)
        self.model.compile(optimizer=sgd, loss='categorical_crossentropy', metrics=['accuracy'])

    def train(self,
              pathToTrainSet,
              pathToValidSet,
              pathToSaveModel,
              epochs=7,
              batchSize=32,
              stepsPerEpoch=200,
              validationSteps=1000):

        fileOfWeights = 'color_weights.hdf5'
        checkpoint = ModelCheckpoint(os.path.join(pathToSaveModel, fileOfWeights),
                                     monitor='val_acc', verbose=1,
                                     save_best_only=True, mode='max')

        checkpoint2 = ModelCheckpoint(os.path.join(pathToSaveModel, fileOfWeights),
                                     monitor='val_loss', verbose=1,
                                     save_best_only=True, mode='max')

        trainDataGen = ImageDataGenerator(rescale=1.0/255, shear_range=0.2,
                                          zoom_range=0.3, horizontal_flip=True)

        validDataGen = ImageDataGenerator(rescale=1.0/255)

        trainSet = trainDataGen.flow_from_directory(
                pathToTrainSet,
                target_size=(self.imageWidth, self.imageHeight),
                batch_size=batchSize,
                class_mode='categorical'
        )

        self.classes = {v: k for k, v in trainSet.class_indices.items()}

        np.save(os.path.join(pathToSaveModel, 'class_index.npy'), self.classes)

        validSet = validDataGen.flow_from_directory(
                pathToValidSet,
                target_size=(self.imageWidth, self.imageHeight),
                batch_size=batchSize,
                class_mode='categorical'
        )

        self.model.fit_generator(
            trainSet,
            steps_per_epoch=stepsPerEpoch,
            epochs=epochs,
            validation_data=validSet,
            validation_steps=validationSteps//batchSize,
            callbacks=[checkpoint, checkpoint2])

        print('============================ Saving is here ============================')
        self.model.save(os.path.join(pathToSaveModel, 'car_color_net.h5'))

    @staticmethod
    def load(pathToModel, pathToClassIndexes):

        model = load_model(pathToModel)

        layers = model.layers
        inputShape, outputShape = layers[0].input_shape, layers[-1].output_shape,

        imageWidth, imageHeight = inputShape[1], inputShape[2]
        numClasses = outputShape[1]

        net = CarColorNet(numClasses, imageWidth, imageHeight)
        net.classes = np.load(os.path.join(pathToClassIndexes, 'class_index.npy')).item()

        return net

    def predictOneImage(self, pathToImage):

        frame = cv2.imread(pathToImage)
        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        frame = cv2.resize(frame, (self.imageWidth, self.imageHeight))

        frame = np.expand_dims(frame, axis=0)

        # cv2.imshow("boxed", frame[0, :, :, :])
        # cv2.waitKey(0)

        frame = np.asarray(frame, dtype='float32')
        img = frame/255

        probs = self.model.predict(img)
        ind = probs.argmax(axis=-1)[0]

        return self.classes[ind]


if __name__ == '__main__':

    pathToTrainSet = '/home/sergorl/cars/train'
    pathToValidSet = '/home/sergorl/cars/valid'
    pathToSaveModel = '/home/sergorl/cars'

    ## Train net
    # net = CarColorNet(numClasses=6)
    # net.train(pathToTrainSet, pathToValidSet, pathToSaveModel)

    # Test on all images from validSet
    net = CarColorNet.load(os.path.join(pathToSaveModel, 'car_color_net.h5'), pathToSaveModel)

    count, countTrueLabels = 0, 0

    for dirpath, _dirnames, filenames in os.walk(pathToValidSet):
        trueLabel = dirpath.split('/')[-1]

        for file in filenames:

            label = net.predictOneImage(os.path.join(dirpath, file))

            print(trueLabel, label)

            if label == trueLabel:
                countTrueLabels += 1

            count += 1

    print('rate is {0:.2f}'.format(float(countTrueLabels) / float(count) * 100))

如果我有一个好的val_acc=0.97，我会期待相同的结果（或几乎相同），测试验证集中的每个图像。但总是有错误的预测！

train完后马上跑网，发现学习很好：

if __name__ == '__main__':

    pathToTrainSet = '/home/sergorl/cars/train'
    pathToValidSet = '/home/sergorl/cars/valid'
    pathToSaveModel = '/home/sergorl/cars'

    # Train net
    net = CarColorNet(numClasses=6)
    net.train(pathToTrainSet, pathToValidSet, pathToSaveModel)

    # Test on all images from validSet
    count, countTrueLabels = 0, 0

    for dirpath, _dirnames, filenames in os.walk(pathToValidSet):
        trueLabel = dirpath.split('/')[-1]

        for file in filenames:

            label = net.predictOneImage(os.path.join(dirpath, file))

            print(trueLabel, label)

            if label == trueLabel:
                countTrueLabels += 1

            count += 1

    print('rate is {0:.2f}'.format(float(countTrueLabels) / float(count) * 100))

所以看来问题出在model.save 内部，而且看起来保存不起作用！。我在 git 上发现了很多相关的问题，例如：

https://github.com/keras-team/keras/issues/4875

https://github.com/keras-team/keras/issues/4904

但我不知道如何用Python 3.7.3 和keras 2.0.0 修复它

Answer 1

你能否分享更多关于这个问题的信息，比如你得到的输出是什么。从代码中我可以看到您正在训练 6 个类并使用分类交叉熵，因此理想情况下，您应该得到一个包含 6 个值的数组，每个值 bw 0 和 1，并且该数组中最高值的索引应该是输出。