Tensorflow：LSTM 和 GradientDescentOptimizer 的梯度为零答案

【问题标题】：Tensorflow: gradients are zero for LSTM and GradientDescentOptimizerTensorflow：LSTM 和 GradientDescentOptimizer 的梯度为零
【发布时间】：2018-03-14 22:30:04
【问题描述】：

由 GradientDescentOptimizer 为 LSTM 网络计算的梯度始终为零。即使在第一步它们也是零，所以，我认为这不是梯度消失的问题。 AdamOptimizer 也会出现同样的问题。

我已将输入减少到时间序列的一个点，并将标签（预期输出）标记到下一个点，并为神经网络预测附加信息，希望找到梯度为零的根本原因。即使在这个最小的设置中，渐变也是零。

我已经阅读了类似的问题，但没有回答可以帮助我。

import tensorflow as tf
import numpy as np
from tensorflow.contrib import rnn

def input_placeholder_sequence(input_placeholder, sequence_length, batch_size):
    input_placeholder = tf.transpose(input_placeholder, name="transpose_input")

    print("input_placeholder shape: " + str(input_placeholder.get_shape()))

    input_placeholder = tf.split(input_placeholder, np.repeat(batch_size, sequence_length), axis=0)

    print("input_placeholder_sequence shape: " + str(np.shape(input_placeholder)))
    print("input_placeholder_in_sequence shape: " + str(input_placeholder[0].get_shape()))

    return input_placeholder

def train_model(input_placeholder, sequence_length, batch_size, output_size):
    input_placeholder = input_placeholder_sequence(input_placeholder, sequence_length, batch_size)
    rnn_cell = rnn.BasicLSTMCell(output_size, name="hidden_layer")

    hidden_outputs, states = rnn.static_rnn(rnn_cell, input_placeholder, dtype=tf.float32)

    print("hidden_outputs shape: " + str(np.shape(hidden_outputs)))
    print("hidden_outputs last shape: " + str(hidden_outputs[-1].get_shape()))

    result = tf.concat(hidden_outputs, 0, name="concat")

    print("result shape: " + str(result.get_shape()))

    result = tf.transpose(result, name="transpose_result")

    print("result shape transposed: " + str(result.get_shape()))

    return result, rnn_cell, states

def main():
    input = [[1448949600], [3], [0.70089], [0.70089], [0.70086], [0.70089], [0.70071], [0.70071], [0.7007], [0.70071]]
    label = [[1448949660], [10], [0.70086], [0.7009], [0.70084], [0.70092], [0.7007], [0.70071], [0.70067], [0.70073], [0], [0], [0], [1]]

    print("input shape: " + str(np.shape(input)))
    print("label shape: " + str(np.shape(label)))

    input_size = np.shape(input)[0]
    output_size = np.shape(label)[0]
    batch_size = 1
    sequence_length = 1
    learning_rate = 0.01

    print("input_size: " + str(input_size))
    print("output_size: " + str(output_size))
    print("batch_size: " + str(batch_size))
    print("sequence_length: " + str(sequence_length))

    input_placeholder = tf.placeholder(tf.float32, (input_size, sequence_length * batch_size), "input")

    prediction_operation, rnn_cell, states = train_model(input_placeholder, sequence_length, batch_size, output_size)

    label_placeholder = tf.placeholder(tf.float32, (output_size, batch_size), "label")

    global_step = tf.Variable(0, name='global_step',trainable=False)

    cost = tf.norm(tf.subtract(prediction_operation, label_placeholder))
    optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
    gradients = optimizer.compute_gradients(cost)
    minimizer = optimizer.apply_gradients(gradients, global_step)

    init = tf.global_variables_initializer()

    with tf.Session() as session:
        session.run(init)

        _, prediction, loss, grads, weights, gl_step \
            = session.run([minimizer, prediction_operation, cost, gradients, rnn_cell.weights, global_step],
                          feed_dict={input_placeholder: input, label_placeholder: label})
        print("loss: " + str(loss))
        print("prediction: " + str(prediction))

        print("rnn weights and biases: " + str(weights))
        print("rnn weights and biases shape: " + str(np.shape(weights)))
        print("rnn weights shape: " + str(np.shape(weights[0])))
        print("rnn biases shape: " + str(np.shape(weights[1])))

        print("rnn weights and biases sum: " + str(np.sum(np.abs(weights[0])) + np.sum(np.abs(weights[1]))))

        print("gradients and variables: " + str(grads))
        print("gradients weights: " + str(grads[0][0]))
        print("gradients biases: " + str(grads[1][0]))
        print("gradients sum: " + str(np.sum(np.abs(grads[0][0])) + np.sum(np.abs(grads[1][0]))))

        print("global steps: " + str(gl_step))

if __name__ == "__main__":
    main()

【问题讨论】：

您是否尝试过使用 tfdbg 查看网络中出现零的位置？
我签入了 tfdbg。 BasicLSTMCell 的偏差和状态为零，但我会说这是意料之中的。

标签： python tensorflow machine-learning lstm recurrent-neural-network

【解决方案1】：

实现是正确的。输入并将结果标记为消失的梯度。

如果输入和标签如下，则梯度不为零。

input = [[3], [0.70089], [0.70089], [0.70086], [0.70089], [0.70071], [0.70071], [0.7007], [0.70071]]
label = [[10], [0.70086], [0.7009], [0.70084], [0.70092], [0.7007], [0.70071], [0.70067], [0.70073], [0], [0], [0], [1]]

【讨论】：