TensorFlow LSTM：关于批处理和状态的混淆答案

【问题标题】：TensorFlow LSTM: confusion about batching and statesTensorFlow LSTM：关于批处理和状态的混淆
【发布时间】：2016-11-11 22:02:24
【问题描述】：

看着官方的 TensorFlow RNN tutorial 和 full code，我有点困惑数据是如何在 epoch 中划分的。首先，我不明白run_epoch()函数中状态变量的使用。在 main() 函数中，在 epochs 的循环中，我们调用：

def run_epoch(session, model, eval_op=None, verbose=False):
  """Runs the model on the given data."""
  start_time = time.time()
  costs = 0.0
  iters = 0
  state = session.run(model.initial_state)

  fetches = {
      "cost": model.cost,
      "final_state": model.final_state,
  }
  if eval_op is not None:
    fetches["eval_op"] = eval_op

  for step in range(model.input.epoch_size):
    feed_dict = {}
    for i, (c, h) in enumerate(model.initial_state):
      feed_dict[c] = state[i].c
      feed_dict[h] = state[i].h

    vals = session.run(fetches, feed_dict)
    cost = vals["cost"]
    state = vals["final_state"]

    costs += cost
    iters += model.input.num_steps

    if verbose and step % (model.input.epoch_size // 10) == 10:
      print("%.3f perplexity: %.3f speed: %.0f wps" %
            (step * 1.0 / model.input.epoch_size, np.exp(costs / iters),
             iters * model.input.batch_size / (time.time() - start_time)))

  return np.exp(costs / iters)

什么是state 变量，为什么我们在每一步都枚举和覆盖model.initial_state？

另外，查看reader.py 文件，以下拆分数据：

def ptb_producer(raw_data, batch_size, num_steps, name=None):
  """Iterate on the raw PTB data.

  This chunks up raw_data into batches of examples and returns Tensors that
  are drawn from these batches.

  Args:
    raw_data: one of the raw data outputs from ptb_raw_data.
    batch_size: int, the batch size.
    num_steps: int, the number of unrolls.
    name: the name of this operation (optional).

  Returns:
    A pair of Tensors, each shaped [batch_size, num_steps]. The second element
    of the tuple is the same data time-shifted to the right by one.

  Raises:
    tf.errors.InvalidArgumentError: if batch_size or num_steps are too high.
  """
  with tf.name_scope(name, "PTBProducer", [raw_data, batch_size, num_steps]):
    raw_data = tf.convert_to_tensor(raw_data, name="raw_data", dtype=tf.int32)

    data_len = tf.size(raw_data)
    batch_len = data_len // batch_size
    data = tf.reshape(raw_data[0 : batch_size * batch_len],
                      [batch_size, batch_len])

    epoch_size = (batch_len - 1) // num_steps
    assertion = tf.assert_positive(
        epoch_size,
        message="epoch_size == 0, decrease batch_size or num_steps")
    with tf.control_dependencies([assertion]):
      epoch_size = tf.identity(epoch_size, name="epoch_size")

    i = tf.train.range_input_producer(epoch_size, shuffle=False).dequeue()
    x = tf.slice(data, [0, i * num_steps], [batch_size, num_steps])
    y = tf.slice(data, [0, i * num_steps + 1], [batch_size, num_steps])
    return x, y

为什么我们要分批分步拆分数据并将两者混合？这有点令人困惑。为什么不分批或分步迭代整个数据集？

【问题讨论】：

标签： python tensorflow lstm

【解决方案1】：

两者完全不同。 num_steps 参数控制发生反向传播的输入数量。通过一次性处理更大的数据，分批拆分数据提供了一种有效的实现方式。如果您直接查看批处理输入，这会有点令人困惑。将 batch_size 设置为 1 并检查输入的外观。这就像只使用 num_steps。训练 RNN 需要更长的时间。

【讨论】：