Keras 模型训练 - 是否可以通过具有 1 个或多个输入的生成器答案

【问题标题】：Keras Model Training - Is it possible to pass an generator with 1 or more inputsKeras 模型训练 - 是否可以通过具有 1 个或多个输入的生成器
【发布时间】：2021-08-09 15:19:44
【问题描述】：

我目前正在研究视觉问答主题。我做了一个模型如下：

Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_3 (InputLayer)            [(None, 224, 224, 3) 0                                            
__________________________________________________________________________________________________
block1_conv1 (Conv2D)           (None, 224, 224, 64) 1792        input_3[0][0]                    
__________________________________________________________________________________________________
block1_conv2 (Conv2D)           (None, 224, 224, 64) 36928       block1_conv1[0][0]               
__________________________________________________________________________________________________
block1_pool (MaxPooling2D)      (None, 112, 112, 64) 0           block1_conv2[0][0]               
__________________________________________________________________________________________________
block2_conv1 (Conv2D)           (None, 112, 112, 128 73856       block1_pool[0][0]                
__________________________________________________________________________________________________
block2_conv2 (Conv2D)           (None, 112, 112, 128 147584      block2_conv1[0][0]               
__________________________________________________________________________________________________
block2_pool (MaxPooling2D)      (None, 56, 56, 128)  0           block2_conv2[0][0]               
__________________________________________________________________________________________________
block3_conv1 (Conv2D)           (None, 56, 56, 256)  295168      block2_pool[0][0]                
__________________________________________________________________________________________________
block3_conv2 (Conv2D)           (None, 56, 56, 256)  590080      block3_conv1[0][0]               
__________________________________________________________________________________________________
block3_conv3 (Conv2D)           (None, 56, 56, 256)  590080      block3_conv2[0][0]               
__________________________________________________________________________________________________
block3_conv4 (Conv2D)           (None, 56, 56, 256)  590080      block3_conv3[0][0]               
__________________________________________________________________________________________________
block3_pool (MaxPooling2D)      (None, 28, 28, 256)  0           block3_conv4[0][0]               
__________________________________________________________________________________________________
block4_conv1 (Conv2D)           (None, 28, 28, 512)  1180160     block3_pool[0][0]                
__________________________________________________________________________________________________
block4_conv2 (Conv2D)           (None, 28, 28, 512)  2359808     block4_conv1[0][0]               
__________________________________________________________________________________________________
block4_conv3 (Conv2D)           (None, 28, 28, 512)  2359808     block4_conv2[0][0]               
__________________________________________________________________________________________________
block4_conv4 (Conv2D)           (None, 28, 28, 512)  2359808     block4_conv3[0][0]               
__________________________________________________________________________________________________
block4_pool (MaxPooling2D)      (None, 14, 14, 512)  0           block4_conv4[0][0]               
__________________________________________________________________________________________________
block5_conv1 (Conv2D)           (None, 14, 14, 512)  2359808     block4_pool[0][0]                
__________________________________________________________________________________________________
block5_conv2 (Conv2D)           (None, 14, 14, 512)  2359808     block5_conv1[0][0]               
__________________________________________________________________________________________________
block5_conv3 (Conv2D)           (None, 14, 14, 512)  2359808     block5_conv2[0][0]               
__________________________________________________________________________________________________
block5_conv4 (Conv2D)           (None, 14, 14, 512)  2359808     block5_conv3[0][0]               
__________________________________________________________________________________________________
block5_pool (MaxPooling2D)      (None, 7, 7, 512)    0           block5_conv4[0][0]               
__________________________________________________________________________________________________
flatten_1 (Flatten)             (None, 25088)        0           block5_pool[0][0]                
__________________________________________________________________________________________________
input_4 (InputLayer)            [(None, 20)]         0                                            
__________________________________________________________________________________________________
repeat_vector_1 (RepeatVector)  (None, 20, 25088)    0           flatten_1[0][0]                  
__________________________________________________________________________________________________
embedding_1 (Embedding)         (None, 20, 50)       901900      input_4[0][0]                    
__________________________________________________________________________________________________
concatenate_1 (Concatenate)     (None, 20, 25138)    0           repeat_vector_1[0][0]            
                                                                 embedding_1[0][0]                
__________________________________________________________________________________________________
bidirectional_1 (Bidirectional) (None, 20, 50)       5032800     concatenate_1[0][0]              
__________________________________________________________________________________________________
global_max_pooling1d_1 (GlobalM (None, 50)           0           bidirectional_1[0][0]            
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 18037)        919887      global_max_pooling1d_1[0][0]     
==================================================================================================

你可以在这里找到关于 VQA 的原始论文：http://arxiv.org/pdf/1512.02167.pdf

总结一下，我有一个有 2 个输入的模型

一个预先训练的 VGG19，可以拍摄图像
和一个接受标记化问题的嵌入式层。

作为输出，我们有一个带有最终密集层的双向 LSTM，可以给出问题的答案。

训练数据如下：

          img_path                              question                         answer
103 train2014/COCO_train2014_000000262171.jpg   How many people are on the boat?    5   
104 train2014/COCO_train2014_000000262171.jpg   What color are the leaves?  green
105 train2014/COCO_train2014_000000262171.jpg   What type of watercraft is that?    raft    
131 train2014/COCO_train2014_000000262180.jpg   What is the fruit?  banana
132 train2014/COCO_train2014_000000262180.jpg   Is this a good dessert?

我的问题是这个：我无法对内存中的所有图像进行充电，我想知道是否可以使用生成器来拟合模型以动态生成图像 + 标记化问题?

我想做类似的事情：

h = model_VQA.fit([X_train_img_generator, X_train_question], y_train_answer, epochs = 15, batch_size = 32)

其中：X_train_question 是标记化的问题，X_train_img_generator 是图像生成器。

--> 但它不起作用，有没有办法正确处理这个问题？

--------- 2021 年 6 月 2 日编辑

好的，我现在更新了我的问题的答案，并更正了一些关于输入图像大小现在为 480x640x3 的问题

__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_8 (InputLayer)            [(None, 480, 640, 3) 0                                            
__________________________________________________________________________________________________
block1_conv1 (Conv2D)           (None, 480, 640, 64) 1792        input_8[0][0]                    
__________________________________________________________________________________________________
block1_conv2 (Conv2D)           (None, 480, 640, 64) 36928       block1_conv1[0][0]               
__________________________________________________________________________________________________
block1_pool (MaxPooling2D)      (None, 240, 320, 64) 0           block1_conv2[0][0]               
__________________________________________________________________________________________________
block2_conv1 (Conv2D)           (None, 240, 320, 128 73856       block1_pool[0][0]                
__________________________________________________________________________________________________
block2_conv2 (Conv2D)           (None, 240, 320, 128 147584      block2_conv1[0][0]               
__________________________________________________________________________________________________
block2_pool (MaxPooling2D)      (None, 120, 160, 128 0           block2_conv2[0][0]               
__________________________________________________________________________________________________
block3_conv1 (Conv2D)           (None, 120, 160, 256 295168      block2_pool[0][0]                
__________________________________________________________________________________________________
block3_conv2 (Conv2D)           (None, 120, 160, 256 590080      block3_conv1[0][0]               
__________________________________________________________________________________________________
block3_conv3 (Conv2D)           (None, 120, 160, 256 590080      block3_conv2[0][0]               
__________________________________________________________________________________________________
block3_conv4 (Conv2D)           (None, 120, 160, 256 590080      block3_conv3[0][0]               
__________________________________________________________________________________________________
block3_pool (MaxPooling2D)      (None, 60, 80, 256)  0           block3_conv4[0][0]               
__________________________________________________________________________________________________
block4_conv1 (Conv2D)           (None, 60, 80, 512)  1180160     block3_pool[0][0]                
__________________________________________________________________________________________________
block4_conv2 (Conv2D)           (None, 60, 80, 512)  2359808     block4_conv1[0][0]               
__________________________________________________________________________________________________
block4_conv3 (Conv2D)           (None, 60, 80, 512)  2359808     block4_conv2[0][0]               
__________________________________________________________________________________________________
block4_conv4 (Conv2D)           (None, 60, 80, 512)  2359808     block4_conv3[0][0]               
__________________________________________________________________________________________________
block4_pool (MaxPooling2D)      (None, 30, 40, 512)  0           block4_conv4[0][0]               
__________________________________________________________________________________________________
block5_conv1 (Conv2D)           (None, 30, 40, 512)  2359808     block4_pool[0][0]                
__________________________________________________________________________________________________
block5_conv2 (Conv2D)           (None, 30, 40, 512)  2359808     block5_conv1[0][0]               
__________________________________________________________________________________________________
block5_conv3 (Conv2D)           (None, 30, 40, 512)  2359808     block5_conv2[0][0]               
__________________________________________________________________________________________________
block5_conv4 (Conv2D)           (None, 30, 40, 512)  2359808     block5_conv3[0][0]               
__________________________________________________________________________________________________
block5_pool (MaxPooling2D)      (None, 15, 20, 512)  0           block5_conv4[0][0]               
__________________________________________________________________________________________________
flatten_7 (Flatten)             (None, 153600)       0           block5_pool[0][0]                
__________________________________________________________________________________________________
input_quest (InputLayer)        [(None, None)]       0                                            
__________________________________________________________________________________________________
repeat_vector_7 (RepeatVector)  (None, 20, 153600)   0           flatten_7[0][0]                  
__________________________________________________________________________________________________
embedding_7 (Embedding)         (None, 20, 50)     540700      input_quest[0][0]                
__________________________________________________________________________________________________
concatenate_7 (Concatenate)     (None, 20, 153650)   0           repeat_vector_7[0][0]            
                                                                 embedding_7[0][0]                
__________________________________________________________________________________________________
bidirectional_7 (Bidirectional) (None, 20, 22)       13522256    concatenate_7[0][0]              
__________________________________________________________________________________________________
global_max_pooling1d_7 (GlobalM (None, 22)           0           bidirectional_7[0][0]            
__________________________________________________________________________________________________
dense_7 (Dense)                 (None, 7465)         171695      global_max_pooling1d_7[0][0]     
==================================================================================================

数据集为：

def load(file_path):

    img = tf.io.read_file(file_path)
    img = tf.image.decode_png(img, channels=3)
    img = tf.image.convert_image_dtype(img, tf.float32)
    img = preprocess_input(img)

    #img = tf.image.resize(img, size=(224, 224))
    img /= 255.
    img = tf.expand_dims(img, axis = 0)

    return img

x1 = tf.data.Dataset.from_tensor_slices(X_img_train).map(lambda xx: load(xx))
x2 = tf.data.Dataset.from_tensor_slices(X_train_rnn_pad)
y = tf.data.Dataset.from_tensor_slices(answer_tr)
dataset = tf.data.Dataset.zip(((x1, x2), y))


h = model_VQA.fit(x = dataset, batch_size = 32, shuffle = True, epochs = 15)

但我收到以下错误：

ValueError: Dimension 0 in both shapes must be equal, but are 1 and 20. Shapes are [1,20] and [20,1]. for '{{node model_8/concatenate_7/concat}} = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32](model_8/repeat_vector_7/Tile, model_8/embedding_7/embedding_lookup/Identity_1, model_8/concatenate_7/concat/axis)' with input shapes: [1,20,153600], [20,1,50], [] and with computed input tensors: input[2] = <2>.

我猜这与嵌入部分的输入形状有关，但我没有错过什么

我的输入数据形状是

X_train_rnn_pad = (53607,20), 和 answer_tr = (53607, 7465)

【问题讨论】：

标签： tensorflow keras input

【解决方案1】：

是的，您应该使用tf.data.Datasets API 之类的东西

如果你有一个 2 输入模型，你会做这样的事情：

x = tf.data.Dataset.from_tensor_slices((img_path_array, questions_array))
y = tf.data.Dataset.from_tensor_slices(answer_array)
dataset = tf.data.Dataset.zip((x, y)).shuffle(50)

之后，您可以使用.map method 来加载您的图像并根据需要应用数据增强。

然后，您只需执行以下操作：

h = model.fit(dataset)

使用此 API 可避免在内存中加载图像。

【讨论】：

我知道我试一试！
自从我研究数据集以来已经过了一段时间，但我很难理解我是如何处理这一切的。我不明白你如何映射到生成器
[这是答案中的完整示例] (stackoverflow.com/questions/63636427/…)。我建议你学习如何使用它，一开始可能需要一些时间。它是一个强大的工具，有助于提高可扩展性。
好的，我会尽力理解它。让你知道我的进步
我还有一个问题：ValueError：两个形状中的维度 0 必须相等，但分别是 1 和 20。形状是 [1,20] 和 [20,1]。 for '{{node model_8/concatenate_7/concat}} = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32](model_8/repeat_vector_7/Tile, model_8/embedding_7/embedding_lookup/Identity_1, model_8/concatenate_7/concat/axis) ' 输入形状：[1,20,153600], [20,1,50], [] 和计算输入张量：输入[2] = 。我猜这是由于输入，但无法弄清楚为什么... [查看编辑]