在 keras 中的预训练密集层之间添加 dropout 层答案

【问题标题】：Add dropout layers between pretrained dense layers in keras在 keras 中的预训练密集层之间添加 dropout 层
【发布时间】：2020-03-07 05:45:32
【问题描述】：

keras.applications 中有一个在 imagenet 上预训练的 VGG16 模型。

from keras.applications import VGG16
model = VGG16(weights='imagenet')

此模型具有以下结构。


Layer (type)                     Output Shape          Param #     Connected to                     
====================================================================================================
input_1 (InputLayer)             (None, 3, 224, 224)   0                                            
____________________________________________________________________________________________________
block1_conv1 (Convolution2D)     (None, 64, 224, 224)  1792        input_1[0][0]                    
____________________________________________________________________________________________________
block1_conv2 (Convolution2D)     (None, 64, 224, 224)  36928       block1_conv1[0][0]               
____________________________________________________________________________________________________
block1_pool (MaxPooling2D)       (None, 64, 112, 112)  0           block1_conv2[0][0]               
____________________________________________________________________________________________________
block2_conv1 (Convolution2D)     (None, 128, 112, 112) 73856       block1_pool[0][0]                
____________________________________________________________________________________________________
block2_conv2 (Convolution2D)     (None, 128, 112, 112) 147584      block2_conv1[0][0]               
____________________________________________________________________________________________________
block2_pool (MaxPooling2D)       (None, 128, 56, 56)   0           block2_conv2[0][0]               
____________________________________________________________________________________________________
block3_conv1 (Convolution2D)     (None, 256, 56, 56)   295168      block2_pool[0][0]                
____________________________________________________________________________________________________
block3_conv2 (Convolution2D)     (None, 256, 56, 56)   590080      block3_conv1[0][0]               
____________________________________________________________________________________________________
block3_conv3 (Convolution2D)     (None, 256, 56, 56)   590080      block3_conv2[0][0]               
____________________________________________________________________________________________________
block3_pool (MaxPooling2D)       (None, 256, 28, 28)   0           block3_conv3[0][0]               
____________________________________________________________________________________________________
block4_conv1 (Convolution2D)     (None, 512, 28, 28)   1180160     block3_pool[0][0]                
____________________________________________________________________________________________________
block4_conv2 (Convolution2D)     (None, 512, 28, 28)   2359808     block4_conv1[0][0]               
____________________________________________________________________________________________________
block4_conv3 (Convolution2D)     (None, 512, 28, 28)   2359808     block4_conv2[0][0]               
____________________________________________________________________________________________________
block4_pool (MaxPooling2D)       (None, 512, 14, 14)   0           block4_conv3[0][0]               
____________________________________________________________________________________________________
block5_conv1 (Convolution2D)     (None, 512, 14, 14)   2359808     block4_pool[0][0]                
____________________________________________________________________________________________________
block5_conv2 (Convolution2D)     (None, 512, 14, 14)   2359808     block5_conv1[0][0]               
____________________________________________________________________________________________________
block5_conv3 (Convolution2D)     (None, 512, 14, 14)   2359808     block5_conv2[0][0]               
____________________________________________________________________________________________________
block5_pool (MaxPooling2D)       (None, 512, 7, 7)     0           block5_conv3[0][0]               
____________________________________________________________________________________________________
flatten (Flatten)                (None, 25088)         0           block5_pool[0][0]                
____________________________________________________________________________________________________
fc1 (Dense)                      (None, 4096)          102764544   flatten[0][0]                    
____________________________________________________________________________________________________
fc2 (Dense)                      (None, 4096)          16781312    fc1[0][0]                        
____________________________________________________________________________________________________
predictions (Dense)              (None, 1000)          4097000     fc2[0][0]                        
====================================================================================================
Total params: 138,357,544
Trainable params: 138,357,544
Non-trainable params: 0
____________________________________________________________________________________________________

我想在密集层（fc1、fc2 和预测）之间使用 dropout 层微调这个模型，同时保持模型的所有预训练权重不变。我知道可以使用model.layers 单独访问每一层，但我还没有找到如何在现有层之间添加新层的任何地方。

这样做的最佳做法是什么？

【问题讨论】：

标签： python keras

【解决方案1】：

我自己通过Keras functional API找到了答案

from keras.applications import VGG16
from keras.layers import Dropout
from keras.models import Model

model = VGG16(weights='imagenet')

# Store the fully connected layers
fc1 = model.layers[-3]
fc2 = model.layers[-2]
predictions = model.layers[-1]

# Create the dropout layers
dropout1 = Dropout(0.85)
dropout2 = Dropout(0.85)

# Reconnect the layers
x = dropout1(fc1.output)
x = fc2(x)
x = dropout2(x)
predictors = predictions(x)

# Create a new model
model2 = Model(input=model.input, output=predictors)

model2 有我想要的 dropout 层

____________________________________________________________________________________________________
Layer (type)                     Output Shape          Param #     Connected to                     
====================================================================================================
input_1 (InputLayer)             (None, 3, 224, 224)   0                                            
____________________________________________________________________________________________________
block1_conv1 (Convolution2D)     (None, 64, 224, 224)  1792        input_1[0][0]                    
____________________________________________________________________________________________________
block1_conv2 (Convolution2D)     (None, 64, 224, 224)  36928       block1_conv1[0][0]               
____________________________________________________________________________________________________
block1_pool (MaxPooling2D)       (None, 64, 112, 112)  0           block1_conv2[0][0]               
____________________________________________________________________________________________________
block2_conv1 (Convolution2D)     (None, 128, 112, 112) 73856       block1_pool[0][0]                
____________________________________________________________________________________________________
block2_conv2 (Convolution2D)     (None, 128, 112, 112) 147584      block2_conv1[0][0]               
____________________________________________________________________________________________________
block2_pool (MaxPooling2D)       (None, 128, 56, 56)   0           block2_conv2[0][0]               
____________________________________________________________________________________________________
block3_conv1 (Convolution2D)     (None, 256, 56, 56)   295168      block2_pool[0][0]                
____________________________________________________________________________________________________
block3_conv2 (Convolution2D)     (None, 256, 56, 56)   590080      block3_conv1[0][0]               
____________________________________________________________________________________________________
block3_conv3 (Convolution2D)     (None, 256, 56, 56)   590080      block3_conv2[0][0]               
____________________________________________________________________________________________________
block3_pool (MaxPooling2D)       (None, 256, 28, 28)   0           block3_conv3[0][0]               
____________________________________________________________________________________________________
block4_conv1 (Convolution2D)     (None, 512, 28, 28)   1180160     block3_pool[0][0]                
____________________________________________________________________________________________________
block4_conv2 (Convolution2D)     (None, 512, 28, 28)   2359808     block4_conv1[0][0]               
____________________________________________________________________________________________________
block4_conv3 (Convolution2D)     (None, 512, 28, 28)   2359808     block4_conv2[0][0]               
____________________________________________________________________________________________________
block4_pool (MaxPooling2D)       (None, 512, 14, 14)   0           block4_conv3[0][0]               
____________________________________________________________________________________________________
block5_conv1 (Convolution2D)     (None, 512, 14, 14)   2359808     block4_pool[0][0]                
____________________________________________________________________________________________________
block5_conv2 (Convolution2D)     (None, 512, 14, 14)   2359808     block5_conv1[0][0]               
____________________________________________________________________________________________________
block5_conv3 (Convolution2D)     (None, 512, 14, 14)   2359808     block5_conv2[0][0]               
____________________________________________________________________________________________________
block5_pool (MaxPooling2D)       (None, 512, 7, 7)     0           block5_conv3[0][0]               
____________________________________________________________________________________________________
flatten (Flatten)                (None, 25088)         0           block5_pool[0][0]                
____________________________________________________________________________________________________
fc1 (Dense)                      (None, 4096)          102764544   flatten[0][0]                    
____________________________________________________________________________________________________
dropout_1 (Dropout)              (None, 4096)          0           fc1[0][0]                        
____________________________________________________________________________________________________
fc2 (Dense)                      (None, 4096)          16781312    dropout_1[0][0]                  
____________________________________________________________________________________________________
dropout_2 (Dropout)              (None, 4096)          0           fc2[1][0]                        
____________________________________________________________________________________________________
predictions (Dense)              (None, 1000)          4097000     dropout_2[0][0]                  
====================================================================================================
Total params: 138,357,544
Trainable params: 138,357,544
Non-trainable params: 0
____________________________________________________________________________________________________

【讨论】：

如何在这个函数式 API 中添加 BatchNormalization？
@Malathi 上次我在 Keras 工作是不久前的事，但我相信应该是类似的。确保批量标准化的初始参数具有均值 0 和标准 1（我不确定这是否是 Keras 中的默认值）。如果您有问题，您可以随时提出新问题，希望比我更新的人可以回答您。
如果您使用 Keras Sequential API，您对这个问题的解决方案会是什么样子？

【解决方案2】：

这是一个保留在 Keras“顺序 API”中的解决方案。

您可以遍历层并按顺序将它们添加到更新的 Sequential 模型中。使用 if 子句在您选择的层之后添加 Dropouts。

from tensorflow.keras.applications import VGG16
from tensorflow.keras.layers import Dropout
from tensorflow.keras.models import Sequential

model = VGG16(weights='imagenet')

# check structure and layer names before looping
model.summary()

# loop through layers, add Dropout after layers 'fc1' and 'fc2'
updated_model = Sequential()
for layer in model.layers:
    updated_model.add(layer)
    if layer.name in ['fc1', 'fc2']:
        updated_model.add(Dropout(.2))

model = updated_model

# check structure
model.summary()

【讨论】：

当我测试你的方法时，我每次都得到相同的预测。你能验证你的答案吗？
@random9 会继续的。您能否进一步澄清这个问题。每次使用预训练模型进行预测时，您都会得到相同的预测？这应该是可以预料的，因为权重没有改变，并且（AFAIK）预测操作中没有随机元素。顺序模型仅在模型训练时自动应用 dropout，因此在预测过程中不应随机丢弃节点（参考：stackoverflow.com/questions/47787011/…）。
抱歉，我说得够精确。我的意思是输出激活是完全相同的。但是每个预测之间应该有小的差异。
@random9 感谢您的澄清。为什么在同一示例上进行多次预测时，最终层的激活值会有所不同？这是您在使用未修改的 VGG-16 模型进行预测时发现的吗？
@random9 节点的丢弃只发生在训练过程中（参见我之前分享的链接）。因此，在预测前向传递期间，网络的行为就像没有随机操作的正常前馈（或卷积等）网络。（更多参考：tensorflow.org/tutorials/keras/overfit_and_underfit#add_dropout）