如何合并两个没有预定义输入大小的 keras 模型

Question

我正在尝试使用连接层将 VGG16 模型、LSTM 模型和张量输入合并为一个

LSTM 代码

这是一个简单的 LSTM 模型

VOCAB_SIZE=1000
encoder = tf.keras.layers.experimental.preprocessing.TextVectorization(max_tokens=VOCAB_SIZE)
encoder.adapt(np.array(df['caption']))

def build_lstm():
  model = Sequential(
      [
       encoder,
       Embedding(input_dim=len(encoder.get_vocabulary()), output_dim=64, mask_zero=True),
       Bidirectional(tf.keras.layers.LSTM(64)),
       Dense(80, activation='relu'),
      ]
  )
  return model

VGG16 代码

这是 Keras 中的 VGG16 实现

def build_vgg16():
  model = Sequential(
      [
       Input(shape=(224,224,3)),
       Conv2D(filters=64,kernel_size=(3,3),padding='same', activation='relu'),
       Conv2D(filters=64,kernel_size=(3,3),padding='same', activation='relu'),
       MaxPool2D(pool_size=(2,2),strides=(2,2)),
       Conv2D(filters=128, kernel_size=(3,3), padding='same', activation='relu'),
       Conv2D(filters=128, kernel_size=(3,3), padding='same', activation='relu'),
       MaxPool2D(pool_size=(2,2),strides=(2,2)),
       Conv2D(filters=256, kernel_size=(3,3), padding='same', activation='relu'),
       Conv2D(filters=256, kernel_size=(3,3), padding='same', activation='relu'),
       Conv2D(filters=256, kernel_size=(3,3), padding='same', activation='relu'),
       MaxPool2D(pool_size=(2,2),strides=(2,2)),
       Conv2D(filters=512, kernel_size=(3,3), padding='same', activation='relu'),
       Conv2D(filters=512, kernel_size=(3,3), padding='same', activation='relu'),
       Conv2D(filters=512, kernel_size=(3,3), padding='same', activation='relu'),
       MaxPool2D(pool_size=(2,2),strides=(2,2)),
       Conv2D(filters=512, kernel_size=(3,3), padding='same', activation='relu'),
       Conv2D(filters=512, kernel_size=(3,3), padding='same', activation='relu'),
       Conv2D(filters=512, kernel_size=(3,3), padding='same', activation='relu'),
       MaxPool2D(pool_size=(2,2),strides=(2,2)),
       Flatten(),
       Dense(512, activation='relu'),
       Dense(256, activation='relu'),
       Dense(128, activation='relu'),
       Dense(80, activation='relu'),
      ]
  )
  return model

线性码

这个模型应该合并 VGG16、LSTM 和一个 80 大小的张量

def build_linear(classes, vgg16, lstm):
  input = Input(shape=(80))
  concat = Concatenate()([vgg16.output, lstm.output, input])
  linear = Flatten()(concat)
  linear = Dense(128, activation='relu')(linear)
  linear = Dense(64, activation='relu')(linear)
  linear = Dense(32, activation='relu')(linear)
  linear = Dense(classes, activation='softmax')(linear)
  return Model([vgg16.input, lstm.input, input], linear)

主要代码

vgg16 = build_vgg16()
lstm = build_lstm()

CLASSES = 6

linear = build_linear(CLASSES, vgg16, lstm)

但是我得到了这个错误

AttributeError                            Traceback (most recent call last)
<ipython-input-291-0fbdec0644b4> in <module>()
      4 CLASSES = 6
      5 
----> 6 linear = build_linear(CLASSES, vgg16, lstm)

1 frames
<ipython-input-197-1bb0f30479b3> in build_linear(classes, vgg16, lstm)
      1 def build_linear(classes, vgg16, lstm):
      2   input = Input(shape=(80))
----> 3   concat = Concatenate()([vgg16.output, lstm.output, input])
      4   linear = Flatten()(concat)
      5   linear = Dense(128, activation='relu')(linear)

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/functional.py in output(self)
    307       RuntimeError: if called in Eager mode.
    308     """
--> 309     return self._nested_outputs
    310 
    311   @property

AttributeError: 'Sequential' object has no attribute '_nested_outputs'

如何合并这两个模型？另一个question 建议添加输入层，但我不太明白在LSTM模型中应用它

Answer 1

显然问题出在编码器上，它给出了不同长度的输出。解决方案是从序列模型中移除编码器并给它预定义的输出序列长度，并添加具有编码器输出长度形状的输入层

新的 LSTM 代码

VOCAB_SIZE=1000
encoder = tf.keras.layers.experimental.preprocessing.TextVectorization(max_tokens=VOCAB_SIZE, output_sequence_length=VOCAB_SIZE)
encoder.adapt(np.array(df['caption']))

def build_lstm():
  model = Sequential(
      [
       Input(shape=(len(encoder.get_vocabulary()))),
       Embedding(input_dim=len(encoder.get_vocabulary()), output_dim=64, mask_zero=True),
       Bidirectional(tf.keras.layers.LSTM(64)),
       Dense(80, activation='relu'),
      ]
  )
  return model

最后将编码器放入训练循环中

训练循环代码

def train(epochs, model, dataset):
  for epoch in range(epochs):
    for step, (x, text, y) in enumerate(dataset):
      encoded = encoder(text)
      loss_val = train_step(x, y)