Keras TimeDistributed 用于多输入情况？

Question

我们的模型描述

在我们的模型中，我想时间分配low_level_model到LSTM上层来做一个层次模型。 low_level_model 通过聚合来自区域序列的结果及其 visit_id 找到客户访问的隐藏表示。每个区域序列都经过 CNN 和注意力层，结果与每次访问的嵌入向量连接。

据我所知，TimeDistributed 包装器可用于制作分层模型，因此我尝试用两个不同的输入包装我们的low_level_model。但似乎该库不支持多输入案例。这是我们的代码。

# Get 1st input
visit_input = keras.Input((1,))
visit_emb = visit_embedding_layer(visit_input)
visit_output = Reshape((-1,))(visit_emb)

# Get 2nd input - Shallow model
areas_input = keras.Input((10,))
areas_emb = area_embedding_layer(areas_input)
areas_cnn = Conv1D(filters=200, kernel_size=5,
               padding='same', activation='relu', strides=1)(areas_emb)
areas_output = simple_attention(areas_cnn, areas_cnn)

# Concat two results from 1st and 2nd input
v_a_emb_concat = Concatenate()([visit_output, areas_output])

# Define this model as low_level_model
low_level_model = keras.Model(inputs=[areas_input, visit_input], outputs=v_a_emb_concat)

# Would like to use the result of this low_level_model as inputs for higher-level LSTM layer.
# Therefore, wrap this model by TimeDistributed layer
encoder = TimeDistributed(low_level_model)

# New input with step-size 5 (Consider 5 as the number of previous data)
all_visit_input = keras.Input((5, 1))
all_areas_input = keras.Input((5, 10))

# This part raises AssertionError (assert len(input_shape) >= 3)
all_areas_rslt = encoder(inputs=[all_visit_input, all_areas_input])
all_areas_lstm = LSTM(64, return_sequences=False)(all_areas_rslt)
logits = Dense(365, activation='softmax')(all_areas_lstm)

# Model define (Multi-input ISSUE HERE!)
self.model = keras.Model(inputs=[all_visit_input, all_areas_input], outputs=logits)

self.model.compile(optimizer=keras.optimizers.Adam(0.001),
                   loss=custom_loss_function)

# Get data
self.train_data = data.train_data_generator_hist()
self.test_data = data.test_data_generator_hist()

# Fit
self.history = self.model.fit_generator(
generator=self.train_data,
steps_per_epoch=train_data_size//FLAGS.batch_size,
epochs=FLAGS.train_epochs]
)

错误信息

错误信息如下。

File "/home/dmlab/sundong/revisit/survival-revisit-code/survrev.py", line 163, in train_test
all_areas_rslt = encoder(inputs=[all_visit_input, all_areas_input])
File "/home/dmlab/ksedm1/anaconda3/envs/py36/lib/python3.6/site-packages/keras/engine/base_layer.py", line 431, in __call__
self.build(unpack_singleton(input_shapes))
File "/home/dmlab/ksedm1/anaconda3/envs/py36/lib/python3.6/site-packages/keras/layers/wrappers.py", line 195, in build
assert len(input_shape) >= 3
AssertionError

我尝试过的

1) 我读了这个keras issue，但无法清楚地弄清楚如何制作转发多个输入的技巧。

2) 我检查了 TimeDistribute 的代码在我只使用单个输入时是否有效（例如，areas_input）。修改后的代码示例如下。

3) 现在尝试关注 [previous question]。 (Keras TimeDistributed layer with multiple inputs)

# Using only one input 
areas_input = keras.Input((10,))
areas_emb = area_embedding_layer(areas_input)
areas_cnn = Conv1D(filters=200, kernel_size=5,
           padding='same', activation='relu', strides=1)(areas_emb)
areas_output = simple_attention(areas_cnn, areas_cnn)

# Define this model as low_level_model
low_level_model = keras.Model(inputs=areas_input, outputs=areas_output)

# Would like to use the result of this low_level_model as inputs for higher-level LSTM layer.
# Therefore, wrap this model by TimeDistributed layer
encoder = TimeDistributed(low_level_model)

# New input with step-size 5 (Consider 5 as the number of previous data)
all_areas_input = keras.Input((5, 10))

# No Error
all_areas_rslt = encoder(inputs=all_areas_input)
all_areas_lstm = LSTM(64, return_sequences=False)(all_areas_rslt)
logits = Dense(365, activation='softmax')(all_areas_lstm)

# Model define (Multi-input ISSUE HERE!)
self.model = keras.Model(inputs=all_areas_input, outputs=logits)

self.model.compile(optimizer=keras.optimizers.Adam(0.001),
               loss=custom_loss_function)

# Get data
self.train_data = data.train_data_generator_hist()
self.test_data = data.test_data_generator_hist()

# Fit
self.history = self.model.fit_generator(
generator=self.train_data,
steps_per_epoch=train_data_size//FLAGS.batch_size,
epochs=FLAGS.train_epochs]
)

提前感谢您分享解决此问题的技巧。

Answer 1

总之，我通过完全获取输入解决了这个问题，并使用Lambda 层划分这些输入。 TimeDistributed 只能接受单个输入，这就是原因。这是我的代码 sn-ps。

single_input = keras.Input((1+10),))
visit_input = Lambda(lambda x: x[:, 0:1])(single_input)
areas_input = Lambda(lambda x: x[:, 1: ])(single_input)
...
low_level_model = keras.Model(inputs=single_input, outputs=concat)

encoder = TimeDistributed(low_level_model)
multiple_inputs = keras.Input((5, 11)))
all_areas_rslt = encoder(inputs=multiple_inputs)
all_areas_lstm = LSTM(64, return_sequences=False)(all_areas_rslt)
logits = Dense(365, activation='softmax')(all_areas_lstm)

Answer 2

我收到了相同的错误消息，并将其追溯到与原始发布者相同的问题和 Github 问题。通过使用keras.layers.RepeatVector，我能够使用TimeDistributed 输出层解决多个输入的问题。下面是我的例子：

core_input_1 = Input(shape=(self.core_timesteps, self.core_input_1_dim), name='core_input_1')
core_branch_1 = BatchNormalization(momentum=0.0, name='core_1_bn')(core_input_1)
core_branch_1 = LSTM(self.core_nodes[0], activation='relu', name='core_1_lstm_1', return_sequences=True)(core_branch_1)
core_branch_1 = LSTM(self.core_nodes[1], activation='relu', name='core_1_lstm_2')(core_branch_1)

core_input_2 = Input(shape=(self.core_timesteps, self.core_input_2_dim), name='core_input_2')
core_branch_2 = BatchNormalization(momentum=0.0, name='core_2_bn')(core_input_2)
core_branch_2 = LSTM(self.core_nodes[0], activation='relu', name='core_2_lstm_1', return_sequences=True)(core_branch_2)
core_branch_2 = LSTM(self.core_nodes[1], activation='relu', name='core_2_lstm_2')(core_branch_2)

merged = Concatenate()([core_branch_1, core_branch_2])

full_branch = RepeatVector(self.output_timesteps)(merged)        
full_branch = LSTM(self.core_nodes[1], activation='relu', name='final_lstm', return_sequences=True)(full_branch)

full_branch = TimeDistributed(Dense(self.output_dim, name='td_dense', activation='relu'))(full_branch)
full_branch = TimeDistributed(Dense(self.output_dim, name='td_dense'))(full_branch)

model = Model(inputs=[core_input_1, core_input_2], outputs=full_branch, name='full_model')

我发布了完整的示例，以便其他人可以看到什么是可能的，但解决方案的关键部分是：

1. return_sequences = False 在 Concat 之前的层中。
2. 如果在 concat 之后使用 LSTM 层，return_sequences = True 将馈送到TimeDistributed 层。
3. RepeatVector 层参数必须与TimeDistributed 层输出中的时间步数相同。

我只在与我的问题相关的架构中测试了这个解决方案，所以我不确定它在 TimeDistributed 层的其他用例中的局限性。但它是一个很好的解决方案，我在任何讨论这个问题的帖子中都找不到。