堆叠 LSTM 单元的所有输出的总和 - Tensorflow

Question

在这里您可能会看到 TensorFlow 中多个堆叠 LSTM 单元的标准实现

with tf.name_scope("RNN_layers"):
    def lstm_cell():
        lstm = tf.contrib.rnn.LayerNormBasicLSTMCell(lstm_size)
        return lstm
    cell = tf.contrib.rnn.MultiRNNCell([lstm_cell() for _ in range(num_layers)])

with tf.name_scope("RNN_init_state"):
    initial_state = cell.zero_state(batch_size, tf.float32)

with tf.name_scope("RNN_forward"):
    outputs, state = tf.nn.dynamic_rnn(cell, inputs, initial_state=initial_state)

这对于大量任务非常有效。然而，除此之外，一些专家建议将沿num_layers 方向的所有单元格的总输出作为最终输出，而不仅仅是最后一个单元格的输出。。 p>

在下图中，要求为y_t=h_t^1+h_t^2+h_t^3

在 TensorFlow 中实现这一点最聪明的方法是什么？

Answer 1

您从tf.nn.dynamic_rnn 获得的outputs 张量是所有单元格的输出列表。如果您想计算它们的总和，只需在outputs 上调用tf.reduce_sum：

n_steps = 2
n_inputs = 3
n_neurons = 5
X = tf.placeholder(dtype=tf.float32, shape=[None, n_steps, n_inputs])

basic_cell = tf.nn.rnn_cell.BasicRNNCell(num_units=n_neurons)
outputs, states = tf.nn.dynamic_rnn(basic_cell, X, dtype=tf.float32)
# outputs = [?, n_steps, n_neurons], e.g. outputs from all cells

sum = tf.reduce_sum(outputs, axis=1)
# sum = [?, n_neurons]

如果是MultiRNNCell，它将是最后一层输出的总和，这也是您通常想要的。

---

更新：

对隐藏层中的张量求和会更加困难，因为 tensorflow MultiRNNCell 对每个单元的输出重复使用相同的张量，因此隐藏层永远不会暴露在 RNN 之外。

最简单的解决方案是编写您自己的MultiRNNCell 来总结每一层的输出，而不是只记住最后一个。您可以这样做：

from tensorflow.python.util import nest

class MyMultiRNNCell(tf.nn.rnn_cell.MultiRNNCell):
  def call(self, inputs, state):
    cur_state_pos = 0
    cur_inp = inputs
    new_states = []
    new_outputs = []
    for i, cell in enumerate(self._cells):
      with tf.variable_scope("cell_%d" % i):
        if self._state_is_tuple:
          if not nest.is_sequence(state):
            raise ValueError("Expected state to be a tuple of length %d, but received: %s" %
                (len(self.state_size), state))
          cur_state = state[i]
        else:
          cur_state = tf.slice(state, [0, cur_state_pos], [-1, cell.state_size])
          cur_state_pos += cell.state_size
        cur_inp, new_state = cell(cur_inp, cur_state)
        new_states.append(new_state)
        new_outputs.append(cur_inp)

    new_states = (tuple(new_states) if self._state_is_tuple else
                  tf.concat(new_states, 1))
    new_outputs_sum = tf.reduce_sum(new_outputs, axis=0)

    return new_outputs_sum, new_states