RNN分类不断输出相同的值

Question

我正在使用 rnn 做一些分类工作，并成功完成一项任务。但是当我在另一个任务上使用类似的模型时，奇怪的事情发生了。 这是一些信息。上面的值是预测，另一个是目标。

Step 147, learning rate is 0.050000000000000, cost is 0.333333
[[ 1.00000000e+00 1.94520349e-16 5.00660735e-10 8.93992450e-11
 6.57709234e-11 2.75211902e-11]]
[[ 0. 0. 0. 0. 0. 1.]]
Step 148, learning rate is 0.050000000000000, cost is 0.333333
[[ 1.00000000e+00 2.51522596e-16 6.98772706e-10 1.32924283e-10
 2.06628145e-10 1.63214553e-10]]
[[ 0. 0. 0. 1. 0. 0.]]
Step 149, learning rate is 0.050000000000000, cost is 1.07511e-18
[[ 1.00000000e+00 6.98618693e-16 2.44663956e-09 2.75078210e-10
 4.09978718e-10 4.69938033e-10]]
[[ 1. 0. 0. 0. 0. 0.]]

似乎所有输出都收敛到相同的值。换句话说，对于每个输入，模型都会输出相同的预测而不考虑成本。

为了提供更多信息，这是我的模型结构：

class SequenceClassification:

def __init__(self, data, target, dropout, learning_rate,num_hidden=2500, num_layers=2):
    self.data = data
    self.target = target
    self.dropout = dropout
    self.learning_rate = learning_rate
    self._num_hidden = num_hidden
    self._num_layers = num_layers
    self.prediction
    self.precision
    self.optimize 

@lazy_property
def prediction(self):
    # Recurrent network.
    network = tf.nn.rnn_cell.BasicLSTMCell(self._num_hidden)
    network = tf.nn.rnn_cell.DropoutWrapper(network, output_keep_prob = self.dropout)
    network = tf.nn.rnn_cell.MultiRNNCell([network]*self._num_layers)
    output, _ = tf.nn.dynamic_rnn(network, data, dtype=tf.float32)
    # Select last output.
    output = tf.transpose(output, [1, 0, 2])
    print(output.get_shape())
    last = tf.gather(output, int(output.get_shape()[0]) - 1)
    # Softmax layer.
    weight, bias = self._weight_and_bias(
        self._num_hidden, int(self.target.get_shape()[1]))
    prediction = tf.nn.softmax(tf.matmul(last, weight) + bias)
    return prediction

@lazy_property
def cost(self):
    #cross_entropy = -tf.reduce_sum(self.target * tf.log(self.prediction+1e-10))
    #loss =cross_entropy
    loss = tf.reduce_mean(tf.square(self.target - self.prediction))
    return loss

@lazy_property
def optimize(self):
    optimizer = tf.train.RMSPropOptimizer(self.learning_rate)
    return optimizer.minimize(self.cost), self.cost, self.prediction

@lazy_property
def precision(self):
    correct = tf.equal(
        tf.argmax(self.target, 1), tf.argmax(self.prediction, 1))
    return tf.reduce_mean(tf.cast(correct, tf.float32))

@staticmethod
def _weight_and_bias(in_size, out_size):
    weight = tf.get_variable("W", shape=[in_size, out_size],
       initializer=tf.contrib.layers.xavier_initializer())
    bias = tf.get_variable("B", shape=[out_size],
       initializer=tf.contrib.layers.xavier_initializer())
    return weight, bias

并且输入的形状是[datanum, maxstep, vectorsize]，我用零将它们填充成相同的大小。

我不明白会发生什么，因为它在前一个任务上效果很好。 此外，当我使用 DL4J 时，这个分类任务运行良好： 这是模型：

MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
        .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT).iterations(1)
        .updater(Updater.RMSPROP)
        .regularization(true).l2(1e-5)
        .weightInit(WeightInit.XAVIER)
        .gradientNormalization(GradientNormalization.ClipElementWiseAbsoluteValue).gradientNormalizationThreshold(1.0)
        .learningRate(0.08)
        .dropOut(0.5)
        .list(2)
        .layer(0, new GravesBidirectionalLSTM.Builder().nIn(vectorSize).nOut(1800)
                .activation("tanh").build())
        .layer(1, new RnnOutputLayer.Builder().activation("softmax")
                .lossFunction(LossFunctions.LossFunction.MCXENT).nIn(1800).nOut(6).build())
        .pretrain(false).backprop(true).build();

MultiLayerNetwork net = new MultiLayerNetwork(conf);
net.init();

感谢任何建议。

Answer 1

这个问题可能是由于“批量标准化”造成的。当您评估您的模型时，您应该禁用批量标准化。