如何在 TensorFlow 中堆叠的 LSTM 单元之间添加 dropout 层？答案

【问题标题】：How to add dropout layer in between stacked LSTM cells in TensorFlow?如何在 TensorFlow 中堆叠的 LSTM 单元之间添加 dropout 层？
【发布时间】：2017-03-21 09:29:12
【问题描述】：

我可以创建一个包含两个分层 LSTM 的 RNN 网络，如下所示：

lstm_cell1 = tf.nn.rnn_cell.BasicLSTMCell(50)
lstm_cell2 = tf.nn.rnn_cell.BasicLSTMCell(100)
lstm_net = tf.nn.rnn_cell.MultiRNNCell([lstm_cell1, lstm_cell2])

但现在我还想在每个 lstm 单元之后包含 dropout 层。类似的，

tf.nn.rnn_cell.MultiRNNCell([tf.nn.dropout(lstm_cell1, 0.8), tf.nn.dropout(lstm_cell2, 0.8)])

我如何做到这一点？

【问题讨论】：

如果你想删除相应的行，我相信你可以做类似 tf.nn.dropout( [lstm_cell1, lstm_cell2], .8) 的事情。
不。这是行不通的。可能是因为lstm_cell1 或lstm_cell2 不是张量。

标签： tensorflow lstm recurrent-neural-network

【解决方案1】：

lstm_cell = tf.contrib.rnn.BasicLSTMCell(n_hidden_units)    
lstm_dropout = tf.contrib.rnn.DropoutWrapper(lstm_cell,input_keep_prob=keep_prob, output_keep_prob=keep_prob)
lstm_layers = tf.contrib.rnn.MultiRNNCell([lstm_dropout]* 2)

【讨论】：

【解决方案2】：

这是我的堆叠 LSTM 辍学代码。 Mashood Tanveer 的答案已经足够好了，但我想补充一点，对于 MultiRNNCell，你最好不要使用[cell]*num_layer。这是因为[cell]*num_layer 会将一个 LSTM 实例堆叠到一个列表中，这可能会导致维度不匹配。除非您确实知道它将输出什么维度，否则我建议您使用这样的代码。

[tf.contrib.rnn.BasicLSTMCell(hidden_size) for _ in range(num_layers)]

fw_lstms = []
for _ in range(num_layers):
    cell = tf.contrib.rnn.BasicLSTMCell(hidden_size) 
    cell = tf.nn.rnn_cell.DropoutWrapper(cell, output_keep_prob=1-drop_prob)
    fw_lstms.append(cell)

bw_lstms = []
for _ in range(num_layers):
    cell = tf.contrib.rnn.BasicLSTMCell(hidden_size) 
    cell = tf.nn.rnn_cell.DropoutWrapper(cell, output_keep_prob=1-drop_prob, 
                                         input_keep_prob=1-drop_prob)
    bw_lstms.append(cell)    
#bw_lstms = [tf.contrib.rnn.BasicLSTMCell(hidden_size) for _ in range(num_layers)]

fw_init_state_ls = [lstm.zero_state(batch_size, tf.float32) for lstm in fw_lstms]
bw_init_state_ls = [lstm.zero_state(batch_size, tf.float32) for lstm in bw_lstms]

outputs, final_states_fw, final_states_bw = tf.contrib.rnn.stack_bidirectional_dynamic_rnn(cells_fw = fw_lstms, cells_bw = bw_lstms,
   inputs = inputs,
   initial_states_fw = fw_init_state_ls, 
   initial_states_bw = bw_init_state_ls)
bi_final_state = tf.concat([final_states_fw[-1][1], final_states_bw[-1][1]], 1)

【讨论】：