Tensorflow 梯度为 0，权重不更新

Question

在使用 Keras 一段时间后，我正在尝试学习 TensorFlow，并且我正在尝试为 CIFAR-10 分类构建一个 ConvNet。但是，我认为我误解了 TensorFlow API 中的某些内容，因为即使在 1 层网络模型中权重也不会更新。

模型代码如下：

num_epochs = 10 
batch_size = 64

# Shape of mu and std is correct: (1, 32, 32, 3)
mu = np.mean(X_train, axis=0, keepdims=True)
sigma = np.std(X_train, axis=0, keepdims=True)

# Placeholders for data & normalization
# (normalisation does not help)
data = tf.placeholder(np.float32, shape=(None, 32, 32, 3), name='data')
labels = tf.placeholder(np.int32, shape=(None,), name='labels')
data = (data - mu) / sigma

# flatten
flat = tf.reshape(data, shape=(-1, 32 * 32 * 3))
dense1 = tf.layers.dense(inputs=flat, units=10)
predictions = tf.nn.softmax(dense1)

onehot_labels = tf.one_hot(indices=labels, depth=10)

# Tried sparse_softmax_cross_entropy_with_logits as well
loss = tf.losses.softmax_cross_entropy(onehot_labels=onehot_labels, logits=predictions)
loss = tf.reduce_mean(loss)

# Learning rate does not matter as the weights are not updating!
optimizer = tf.train.AdamOptimizer(learning_rate=0.01).minimize(loss)
loss_history = []

with tf.Session() as session:
    tf.global_variables_initializer().run()
    tf.local_variables_initializer().run()

    for epochs in range(10):
        print("Epoch:", epochs)
        # Load tiny batches-
        for batch in iterate_minibatches(X_train.astype(np.float32)[:10], y_train[:10], 5):
            inputs, target = batch
            feed_dict = {data: inputs, labels: target}
            loss_val, _ = session.run([loss, optimizer], feed_dict=feed_dict)
            grads = tf.reduce_sum(tf.gradients(loss, dense1)[0])
            grads = session.run(grads, {data: inputs, labels: target})
            print("Loss:", loss_val, "Grads:", grads)

代码产生以下输出：

Epoch: 0
Loss: 2.46115 Grads: -1.02031e-17
Loss: 2.46041 Grads: 0.0
Epoch: 1
Loss: 2.46115 Grads: 0.0
Loss: 2.26115 Grads: 0.0
Epoch: 2
Loss: 2.46115 Grads: 0.0
Loss: 2.26115 Grads: 0.0
Epoch: 3
Loss: 2.46115 Grads: 0.0
Loss: 2.26115 Grads: 0.0
Epoch: 4
Loss: 2.46115 Grads: 0.0
Loss: 2.26115 Grads: 0.0
Epoch: 5
Loss: 2.46115 Grads: 0.0
Loss: 2.26115 Grads: 0.0
Epoch: 6
Loss: 2.46115 Grads: 0.0
Loss: 2.26115 Grads: 0.0
Epoch: 7
Loss: 2.46115 Grads: 0.0
Loss: 2.26115 Grads: 0.0
Epoch: 8
Loss: 2.46115 Grads: 0.0
Loss: 2.26115 Grads: 0.0
Epoch: 9
Loss: 2.46115 Grads: 0.0
Loss: 2.26115 Grads: 0.0

看起来模型可能会以某种方式重置其权重或完全停止学习。 我也尝试过稀疏 softmax 交叉熵损失，但没有任何帮助。

Answer 1

您对输出应用了两次 softmax，一次是在 tf.nn.softmax 中，另一次是在应用 softmax_cross_entropy 时。这可能会破坏网络中的任何学习能力。