TensorFlow + Batch Gradient Descent：每批0%的准确率，但网络收敛？

Question

我目前是 TensorFlow 和深度学习的初学者，我正在尝试使用 ReLU 激活函数用于隐藏层，softmax 用于创建一个非常简单的 2 层神经网络输出层。特别是，我在著名的 notMNIST 数据集上进行训练，该数据集与 MNIST 具有完全相同的形状，但具有更困难的示例。这就是我解决它的方法（使用 TensorFlow v1.0.0）：

batch_size = 128
hidden_nodes = 1024

graph = tf.Graph()
with graph.as_default():
    tf_train_dataset = tf.placeholder(tf.float32,
                                    shape=(batch_size, image_size * image_size))
    tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))
    tf_valid_dataset = tf.constant(valid_dataset)
    tf_test_dataset = tf.constant(test_dataset)

    weights_ih = tf.Variable(tf.truncated_normal([image_size * image_size, hidden_nodes]))
    biases_ih = tf.Variable(tf.ones([hidden_nodes])/10)
    weights_ho = tf.Variable(tf.truncated_normal([hidden_nodes, num_labels]))
    biases_ho = tf.Variable(tf.zeros([num_labels]))

    logits = tf.matmul(tf_train_dataset, weights_ih) + biases_ih
    hidden_layer_output = tf.nn.relu(logits)

    output = tf.matmul(hidden_layer_output, weights_ho) + biases_ho
    loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=output, labels=tf_train_labels))

    optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)

    train_prediction = tf.nn.softmax(hidden_layer_output)
    valid_prediction = tf.nn.softmax(tf.matmul(
        tf.nn.relu(tf.matmul(tf_valid_dataset, weights_ih) + biases_ih),
                   weights_ho) + biases_ho)
    test_prediction = tf.nn.softmax(tf.matmul(
        tf.nn.relu(tf.matmul(tf_test_dataset, weights_ih) + biases_ih),
                   weights_ho) + biases_ho)

以这种方式使用简单的跑步者：

num_steps = 5000

def accuracy(predictions, labels):
    return (100.0 * np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1)) / predictions.shape[0])

with tf.Session(graph=graph) as sess:
    tf.global_variables_initializer().run()
    print("Initialized")
    for step in range(num_steps):
        offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
        batch_data = train_dataset[offset:(offset + batch_size), :]
        batch_labels = train_labels[offset:(offset + batch_size), :]
        feed_dict = {tf_train_dataset: batch_data, tf_train_labels: batch_labels}
        _, l, predictions =\
            sess.run([optimizer, loss, train_prediction], feed_dict=feed_dict)
        if (step % 500 == 0):
            print("Minibatch loss at step %d: %f" % (step, l))
            print("Minibatch accuracy: %.1f%%" % accuracy(predictions, batch_labels))
            print("Validation accuracy: %.1f%%" % accuracy(valid_prediction.eval(),
                                                           valid_labels))
    print("Test accuracy: %.1f%%" % accuracy(test_prediction.eval(), test_labels))

我得到了这些结果：

Initialized
Minibatch loss at step 0: 281.805603
Minibatch accuracy: 0.0%
Validation accuracy: 21.9%
Minibatch loss at step 500: 18.725670
Minibatch accuracy: 0.0%
Validation accuracy: 81.0%
Minibatch loss at step 1000: 13.720121
Minibatch accuracy: 0.0%
Validation accuracy: 81.2%
Minibatch loss at step 1500: 16.521467
Minibatch accuracy: 0.0%
Validation accuracy: 81.3%
Minibatch loss at step 2000: 4.905802
Minibatch accuracy: 0.0%
Validation accuracy: 80.7%
Minibatch loss at step 2500: 1.040669
Minibatch accuracy: 0.0%
Validation accuracy: 82.4%
Minibatch loss at step 3000: 2.731811
Minibatch accuracy: 0.0%
Validation accuracy: 80.6%
Minibatch loss at step 3500: 1.011298
Minibatch accuracy: 0.0%
Validation accuracy: 81.9%
Minibatch loss at step 4000: 1.432833
Minibatch accuracy: 0.0%
Validation accuracy: 82.7%
Minibatch loss at step 4500: 0.934623
Minibatch accuracy: 0.0%
Validation accuracy: 82.5%
Test accuracy: 89.6%

可以看出，小批量准确率总是0%，但小批量损失正在下降，验证准确率正在上升。该模型似乎“有效”，但我认为正在发生的其他事情表明存在更大的问题。 500 epochs 后的突然跳跃也很可疑。由于我对此没有太多的直觉，所以我尝试了各种肤浅的东西，比如改变学习率和批量大小，但他们没有对永远 0% 的准确率做任何事情。

如果在 TensorFlow 方面更有经验的人能告诉我这可能是什么原因，我将不胜感激，这样我就可以学会在将来避免它。

提前致谢！

Answer 1

试试

train_prediction = tf.nn.softmax(output)

而不是

train_prediction = tf.nn.softmax(hidden_layer_output)

它应该可以工作。

顺便说一句：我不会把 logit 称为你所说的 logits。您的输出将被称为 logits，但这只是一个命名问题...