使用 tensorflow 训练时，全连接神经网络的损失不会下降

Question

我正在使用 tensorflow 来训练我自己的全连接网络，但是在前几次迭代中出现显着下降后，网络的损失不再改变，并且损失一直徘徊在 4.3 左右。不知道哪里有问题。改变学习率似乎没有帮助。

我在数据集中使用的样本输入（代码中命名为'feat'）是一个长度为13294的稀疏向量，其中只有大约5个位置有效，其余的赋值为1。一批train_x看起来喜欢：

[[1 1 1 1 1 1... - 96...1 1 1 1... - 84...1 1 1 1... - 56...1 1 1 1]
 [1 1 1 1 1... - 47...1 1 1 1 1... - 52...1 1 1 1 1.......1 1 1 1 1]
 ...
]

样本的label是单个值，值在0到137之间。一批train_y的样子：

[
28
28
110
34
...
]

我有 26816 个训练样本用于训练。

使用的代码如下所示

"""Neural network applied with tensroflow.

"""

from __future__ import print_function
import tensorflow as tf
import numpy as np
from scipy.sparse import coo_matrix

file_wifi_feat = 'wifi_feat.npy'
file_shop_label = 'shop_label.npy'
num_shops = 137
num_wifis = 13294
num_hidden_1 = 8192
num_hidden_2 = 2048
num_hidden_3 = 512
num_hidden_4 = 128
num_hidden_5 = 64


class BatchReader:
    def __init__(self, feat, label):
        self.shuffle = True
        self.feat = []
        self.label = []
        self.batch_offset = 0
        self._load_data(feat, label)

    def _load_data(self, feat, label):
        self.feat = np.load(feat)
        self.label = np.load(label)

    def next_batch(self, batch_size):
        start = self.batch_offset
        self.batch_offset += batch_size
        if self.batch_offset > self.feat.shape[0]:
            perm = np.arange(self.feat.shape[0])
            np.random.shuffle(perm)
            self.feat = self.feat[perm]
            self.label = self.label[perm]
            start = 0
            self.batch_offset = batch_size
        end = self.batch_offset
        batch_feat = np.array([m.toarray()[0] for m in self.feat[start:end]])
        batch_feat[np.where(batch_feat == 0)] = 1
        batch_label = self.label[start:end]
        return batch_feat, batch_label


def weight_variable(shape):
    """weight_variable generates a weight variable of a given shape."""
    initial = tf.truncated_normal(shape, stddev=0.1)
    return tf.get_variable(name='weights', initializer=initial)


def bias_variable(shape):
    """bias_variable generates a bias variable of a given shape."""
    initial = tf.constant(0.1, shape=shape)
    return tf.get_variable(name='bias', initializer=initial)


def main(argv=None):
    batch_reader = BatchReader(file_wifi_feat, file_shop_label)

    feat_ph = tf.placeholder(tf.float32, [None, num_wifis])
    label_ph = tf.placeholder(tf.int32, [None])

    with tf.variable_scope('h1'):
        weight = weight_variable([num_wifis, num_hidden_1])
        bias = bias_variable([num_hidden_1])
        L1 = tf.nn.relu(tf.matmul(feat_ph, weight) + bias)

    with tf.variable_scope('h2'):
        weight = weight_variable([num_hidden_1, num_hidden_2])
        bias = bias_variable([num_hidden_2])
        L2 = tf.nn.relu(tf.matmul(L1, weight) + bias)

    with tf.variable_scope('h3'):
        weight = weight_variable([num_hidden_2, num_hidden_3])
        bias = bias_variable([num_hidden_3])
        L3 = tf.nn.relu(tf.matmul(L2, weight) + bias)

    with tf.variable_scope('h4'):
        weight = weight_variable([num_hidden_3, num_hidden_4])
        bias = bias_variable([num_hidden_4])
        L4 = tf.nn.relu(tf.matmul(L3, weight) + bias)

    with tf.variable_scope('h5'):
        weight = weight_variable([num_hidden_4, num_hidden_5])
        bias = bias_variable([num_hidden_5])
        L5 = tf.nn.relu(tf.matmul(L4, weight) + bias)

    with tf.variable_scope('hypo'):
        weight = weight_variable([num_hidden_5, num_shops])
        bias = bias_variable([num_shops])
        hypothesis = tf.matmul(L5, weight) + bias

    loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(logits=hypothesis, labels=label_ph))
    correct_prediction = tf.equal(tf.argmax(hypothesis, 1), tf.cast(label_ph, tf.int64))
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

    global_step = tf.Variable(0, trainable=False)
    starter_learning_rate = 0.1
    learning_rate = tf.train.exponential_decay(starter_learning_rate, global_step, 20, 0.96, staircase=True)
    optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss, global_step=global_step)

    config = tf.ConfigProto()
    config.gpu_options.allow_growth = True
    sess = tf.Session(config=config)
    sess.run(tf.initialize_all_variables())

    for itr in xrange(100001):
        feat, label = batch_reader.next_batch(256)
        feed_dict = {feat_ph: feat, label_ph: label}
        sess.run(optimizer, feed_dict=feed_dict)
        # hypothesis_val = sess.run(hypothesis, feed_dict=feed_dict)
        if itr % 10 == 0:
            loss_val, accuracy_val, learning_rate_val = sess.run([loss, accuracy, learning_rate], feed_dict=feed_dict)
            print('Step %d, loss %g, accuracy %g, learning_rate %g' % (itr, loss_val, accuracy_val, learning_rate_val))


if __name__ == '__main__':
    tf.app.run()

输出看起来是这样的（即使运行到第 10000 步，loss 也没有太大变化）：

Step 0, loss 1.41158e+09, accuracy 0.03125, learning_rate 0.1
Step 10, loss 4.68047, accuracy 0.0273438, learning_rate 0.1
Step 20, loss 4.54852, accuracy 0.0234375, learning_rate 0.096
Step 30, loss 4.22673, accuracy 0.0546875, learning_rate 0.096
Step 40, loss 4.36984, accuracy 0.0390625, learning_rate 0.09216
Step 50, loss 4.26286, accuracy 0.0546875, learning_rate 0.09216
Step 60, loss 4.4269, accuracy 0.0546875, learning_rate 0.0884736
Step 70, loss 4.21976, accuracy 0.105469, learning_rate 0.0884736
Step 80, loss 4.39736, accuracy 0.0546875, learning_rate 0.0849346
Step 90, loss 4.32979, accuracy 0.0820312, learning_rate 0.0849346
Step 100, loss 4.38875, accuracy 0.078125, learning_rate 0.0815373
Step 110, loss 4.37169, accuracy 0.0898438, learning_rate 0.0815373
...

Answer 1

首先，在第 1 次迭代中获得的损失表明您的网络初始化非常糟糕。初始损失不应大于 10，但在您的情况下为 1e9。将网络初始化的 std 至少降低一个数量级。一般来说，您不应该手动初始化变量，而是使用众所周知的启发式方法，例如 Xavier 初始化程序（准备在 TF 中使用）。

第二件事是数据规范化——根据 sn-p 提供的数据量很大，确保每个特征维度的均值为 0，标准为 1。这非常重要，尤其是对于可能会“死”的 relu 激活信号太大。

最后 - 不应该从复杂的架构开始，为什么要从 5 个隐藏层和复杂的学习率计划开始？这些是应该根据需要添加的内容，而不是默认使用。只需从固定学习率（甚至是默认学习率）和小型网络（例如 1-2 个隐藏层）开始，就可以避免上述许多问题。一旦这还不够 - 更深入/更高级的方法是一个好主意，但从它开始会更难理解为什么事情会变得糟糕。

Answer 2

我认为你的 global_step 出错了。

通常 tf.Variable 在网络中用于自动更新训练阶段的值。此外，您将 global_step 初始化为 tf.Variable，但将其设置为不可训练并将值设置为 0。

您可以在文档中找到有关变量的更多信息：https://www.tensorflow.org/programmers_guide/variables

希望能帮到你。