成本函数不规律地变化

Question

背景

我正在使用 tensorflow 为多类分类问题设计一个神经网络解决方案。输入数据由 16 个特征和 6000 个训练示例组成，从 17 列（16 个特征+1 个标签）和 6000 行（训练示例）的 csv 文件中读取).我决定将 16 个神经元作为输入层，16 个神经元在隐藏层，16 个神经元在输出层（因为它是 16 类分类）。这是我的实现代码-

import tensorflow as tf
x=tf.placeholder(tf.float32,shape=[None,16])
y_=tf.placeholder(tf.float32,shape=[None,16])
def weight_variable(shape):
    initial=tf.truncated_normal(shape,stddev=0.1,dtype=tf.float32)
    return tf.Variable(initial)
def bias_variable(shape):
    initial=tf.constant(0.1,shape=shape)
    return tf.Variable(initial)
def read_from_csv(filename_queue):
    reader=tf.TextLineReader()
    key,value=reader.read(filename_queue)
    record_defaults=[[1.], [1.], [1.], [1.], [1.],[1.], [1.], [1.], [1.], [1.],[1.], [1.], [1.], [1.], [1.],[1.],[1.]]
    col1,col2,col3,col4,col5,col6,col7,col8,col9,col10,col11,col12,col13,col14,col15,col16,col17=tf.decode_csv(value,record_defaults=record_defaults)
    features = tf.pack([col1, col2, col3, col4,col5,col6,col7,col8,col9,col10,col11,col12,col13,col14,col15,col16])
    labels=tf.pack([col17])
    return features,labels
def input_pipeline(filenames,batch_size,num_epochs=None):
    filename_queue=tf.train.string_input_producer([filenames],num_epochs=num_epochs,shuffle=True)
    features,labels=read_from_csv(filename_queue)
    min_after_dequeue=100
    capacity=300
    feature_batch,label_batch=tf.train.shuffle_batch([features,labels],batch_size=batch_size,capacity=capacity,min_after_dequeue=min_after_dequeue)
    return feature_batch,label_batch

x,y_=input_pipeline('csvnew1.csv',20,300)
#input layer
W_1=weight_variable([16,16])
b_1=bias_variable([16])
y_1=tf.nn.relu(tf.matmul(x,W_1)+b_1)

#hidden layer
W_2=weight_variable([16,16])
b_2=bias_variable([16])
y_2=tf.nn.softmax(tf.matmul(y_1,W_2)+b_2)


cross_entropy=tf.reduce_mean(-tf.reduce_sum(y_*tf.log(y_2),reduction_indices=[1]))
train_step=tf.train.GradientDescentOptimizer(0.01).minimize(cross_entropy)
correct_prediction=tf.equal(tf.argmax(y_2,1),tf.argmax(y_,1))
accuracy=tf.reduce_mean(tf.cast(correct_prediction,tf.float32))
summary_cross=tf.scalar_summary('cost',cross_entropy)
summaries = tf.merge_all_summaries()

init_op = tf.initialize_all_variables()

# Create a session for running operations in the Graph.
sess = tf.Session()
summary_writer = tf.train.SummaryWriter('stats', sess.graph)


# Initialize the variables (like the epoch counter).
sess.run(init_op)
sess.run(tf.initialize_local_variables())
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(sess=sess, coord=coord)

count=0
try:
    while not coord.should_stop():

         #print("training....")
    #summary_writer.add_summary(sess.run(summaries), count)



    sess.run(train_step)
    if count in range(300,90000,300):
        print(sess.run(cross_entropy))
    count=count+1
except tf.errors.OutOfRangeError:
    print('Done training -- epoch limit reached')
finally:
    # When done, ask the threads to stop.
    coord.request_stop()

# Wait for threads to finish.
coord.join(threads)
sess.close()

问题

这里的问题是，当我在训练期间打印成本函数而不是普遍下降的趋势时，它会随机且不规则地增加和减少。我正在粘贴完整的代码，因为它看起来像是我无法找到的实现问题。（变化学习率是徒劳的）。

编辑：将学习率降低到 10^-12 会产生以下成本（仍然不稳定）

201.928, 173.078, 144.212, 97.6255, 133.125, 164.19, 208.571, 208.599, 188.594, 244.078, 237.414, 224.085, 224.1, 206.36, 217.457, 244.083, 246.309, 268.496, 248.517, 272.924, 228.551, 239.637, 301.759,....

我在每 300 个计数后打印成本，因为 1 个批次 = 20 个示例，6000/20=300 个计数 1 个 epoch，之后权重会更新。

Answer 1

当您看到使用梯度下降时成本函数增加时，您应该尝试降低学习率参数。尝试反复将学习率降低 1/10，直到看到损失单调减少。