如何将标量值和二进制值插入到 keras 中的层（最后一层）？

Question

我正在尝试修改实现 here 的 network。该网络使用胸片images作为输入，将其分为14类（13类疾病，未发现）。该网络不将患者的年龄和性别作为输入。所以我也想向网络提供这些信息。总之网络的最后3层是这样的：

bn (BatchNormalization)         (None, 7, 7, 1024)   4096        conv5_block16_concat[0][0]       
__________________________________________________________________________________________________
avg_pool (GlobalAveragePooling2 (None, 1024)         0           bn[0][0]                         
__________________________________________________________________________________________________
predictions (Dense)             (None, 14)           14350       avg_pool[0][0]  

所以到目前为止我所做的如下： 我只是使用model_vgg16.layers.pop() 弹出最后一个密集层。

然后根据expexted网络变成：

bn (BatchNormalization)         (None, 7, 7, 1024)   4096        conv5_block16_concat[0][0]       
__________________________________________________________________________________________________
avg_pool (GlobalAveragePooling2 (None, 1024)         0           bn[0][0] 

我知道我可以使用以下方法添加图层：

new_layer = Dense(14, activation='softmax', name='my_dense')

inp = model.input
out = new_layer(model.layers[-1].output)

model2 = Model(inp, out)

但我不知道如何添加一个层，该层从前一层获取输入以及 1 个标量值（年龄 [0:100]）和一个二进制值性别 [0:1]。

那么我如何添加最后一层，该层从前一层获取输入以及 1 个标量值和 1 个二进制值？

编辑：我使用的基本模型是 DenseNet121。最后的一些层看起来像这样：

编辑 我加载模型的方式如下：

cp = ConfigParser()
cp.read(config_file)

# default config
output_dir = cp["DEFAULT"].get("output_dir")
base_model_name = cp["DEFAULT"].get("base_model_name")
class_names = cp["DEFAULT"].get("class_names").split(",")
image_source_dir = cp["DEFAULT"].get("image_source_dir")
image_dimension = cp["TRAIN"].getint("image_dimension")

output_weights_name = cp["TRAIN"].get("output_weights_name")
weights_path = os.path.join(output_dir, output_weights_name)
best_weights_path = os.path.join(output_dir, f"best_{output_weights_name}")
model_weights_path = best_weights_path

model_factory = ModelFactory()
model = model_factory.get_model(
    class_names,
    model_name=base_model_name,
    use_base_weights=False,
    weights_path=model_weights_path)

现在模型在变量model 中。 然后按照建议我做

x = model.output
flat1 = Flatten()(x)

并得到这个错误：

ValueError: Input 0 is in compatible with layer flatten_27: expected min_ndim=3, found ndim=2

当我使用model.layers.pop()删除最后一层后重复同样的事情时

我仍然遇到同样的错误？即使我花了几个小时也无法克服这个问题。那么如何做到这一点呢？

Answer 1

试试这个

from keras.applications.densenet import DenseNet121
from keras.layers import Dense, GlobalAveragePooling2D,concatenate

input_image = Input(shape=(224, 224, 3))
# normalize age
input_age_and_gender = Input(shape=(2,))
base_model = DenseNet121(input_tensor=input_image, weights='imagenet', include_top=False)
x = base_model.output
encoded_image = GlobalAveragePooling2D()(x)
out = concatenate([encoded_image,input_age_and_gender])
output = Dense(14, activation='softmax')(out)
model = Model([input_image,input_age_and_gender],output)

Answer 2

你可以有一个多输入模型。

所以不要只使用this:

img_input = Input(shape=input_shape)

base_model = base_model_class(
    include_top=False,
    input_tensor=img_input,
    input_shape=input_shape,
    weights=base_weights,
    pooling="avg")
x = base_model.output
predictions = Dense(len(class_names), activation="sigmoid", name="predictions")(x)
model = Model(inputs=img_input, outputs=predictions)

我不确定您的 base_model 在那里是什么样子。但是为了检查以下内容，第一个输入是虚构的，第二个输入的形状应该是您的 age_gender_df.values 的形状：

input1 = Input(shape=(64,64,1))
conv11 = Conv2D(32, kernel_size=4, activation='relu')(input1)
pool11 = MaxPooling2D(pool_size=(2, 2))(conv11)
conv12 = Conv2D(16, kernel_size=4, activation='relu')(pool11)
pool12 = MaxPooling2D(pool_size=(2, 2))(conv12)
flat1 = Flatten()(pool12)
# INSTEAD OF THE ABOVE INPUT I WROTE YOU CAN USE YOUR BASE MODEL

input2 = Input(shape=(2,2)) # HERE THIS SHOULD BE THE SHAPE OF YOUR AGE/GENDER DF
layer = Dense(10, activation='relu')(input2)
flat2 = Flatten()(layer)

merge = concatenate([flat1, flat2])
# interpretation model
hidden1 = Dense(10, activation='relu')(merge)
hidden2 = Dense(10, activation='relu')(hidden1)
output = Dense(14, activation='linear')(hidden2)
model = Model(inputs=[input1, input2], outputs=output)

总结

__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_30 (InputLayer)           (None, 64, 64, 1)    0                                            
__________________________________________________________________________________________________
conv2d_23 (Conv2D)              (None, 61, 61, 32)   544         input_30[0][0]                   
__________________________________________________________________________________________________
max_pooling2d_23 (MaxPooling2D) (None, 30, 30, 32)   0           conv2d_23[0][0]                  
__________________________________________________________________________________________________
conv2d_24 (Conv2D)              (None, 27, 27, 16)   8208        max_pooling2d_23[0][0]           
__________________________________________________________________________________________________
input_31 (InputLayer)           (None, 2, 2)         0                                            
__________________________________________________________________________________________________
max_pooling2d_24 (MaxPooling2D) (None, 13, 13, 16)   0           conv2d_24[0][0]                  
__________________________________________________________________________________________________
dense_38 (Dense)                (None, 2, 10)        30          input_31[0][0]                   
__________________________________________________________________________________________________
flatten_23 (Flatten)            (None, 2704)         0           max_pooling2d_24[0][0]           
__________________________________________________________________________________________________
flatten_24 (Flatten)            (None, 20)           0           dense_38[0][0]                   
__________________________________________________________________________________________________
concatenate_9 (Concatenate)     (None, 2724)         0           flatten_23[0][0]                 
                                                                 flatten_24[0][0]                 
__________________________________________________________________________________________________
dense_39 (Dense)                (None, 10)           27250       concatenate_9[0][0]              
__________________________________________________________________________________________________
dense_40 (Dense)                (None, 10)           110         dense_39[0][0]                   
__________________________________________________________________________________________________
dense_41 (Dense)                (None, 14)           154         dense_40[0][0]                   
==================================================================================================
Total params: 36,296
Trainable params: 36,296
Non-trainable params: 0

可视化：

编辑：

在你的情况下，我认为模型应该如下所示：

img_input = Input(shape=input_shape)

base_model = base_model_class(
    include_top=False,
    input_tensor=img_input,
    input_shape=input_shape,
    weights=base_weights,
    pooling="avg")
x = base_model.output
flat1 = Flatten()(x)

input2 = Input(shape=(2,2)) # HERE THIS SHOULD BE THE SHAPE OF YOUR AGE/GENDER DF
layer = Dense(10, activation='relu')(input2)
flat2 = Flatten()(layer)

merge = concatenate([flat1, flat2])
# interpretation model
hidden1 = Dense(10, activation='relu')(merge)
hidden2 = Dense(10, activation='relu')(hidden1)
output = Dense(14, activation='linear')(hidden2)
model = Model(inputs=[img_input, input2], outputs=output)

Answer 3

问题的解决方法是先去掉最后一层（预测层）

model_original.layers.pop()

然后定义另一个模型，它是原始模型的副本，除了最后一层

model2 = keras.Model(model_original.input, model_original.layers[-1].output)

然后定义包括年龄的输入

age = layers.Input(shape=(1,))

接下来，年龄输入和之前定义的网络的最后一层使用连接起来

x = model2.output concatenated = layers.concatenate([x, age])

在最后一步中，在连接之后添加一个预测层以完成网络

output = Dense(14, activation='linear')(concatenated) model3 = keras.Model(inputs=[model_original.input, age], outputs=output)

所以设计的最后几层看起来像这样：