模型精度不变，保持为零

Question

我已经建立了一个有两个隐藏层的神经网络。对于前两个隐藏层，我使用了 ReLU 激活，而对于最后一层，我使用了 sigmoid 函数。当我启动模型时，损失函数减小（正确），但精度保持为零。

Epoch: 9/150 Train Loss: 6.1869 Train Acc: 0.0005
Epoch: 9/150 Validation Loss: 6.4013 Validation Acc: 0.0000
Epoch: 17/150 Train Loss: 3.5452 Train Acc: 0.0005
Epoch: 17/150 Validation Loss: 3.7929 Validation Acc: 0.0000
Epoch: 25/150 Train Loss: 2.1594 Train Acc: 0.0005
Epoch: 25/150 Validation Loss: 2.2964 Validation Acc: 0.0000
Epoch: 34/150 Train Loss: 1.4753 Train Acc: 0.0005
Epoch: 34/150 Validation Loss: 1.5603 Validation Acc: 0.0000
Epoch: 42/150 Train Loss: 1.1325 Train Acc: 0.0005
Epoch: 42/150 Validation Loss: 1.2386 Validation Acc: 0.0000
Epoch: 50/150 Train Loss: 0.9314 Train Acc: 0.0005
Epoch: 50/150 Validation Loss: 1.0469 Validation Acc: 0.0000
Epoch: 59/150 Train Loss: 0.8146 Train Acc: 0.0005
Epoch: 59/150 Validation Loss: 0.9405 Validation Acc: 0.0000
Epoch: 67/150 Train Loss: 0.7348 Train Acc: 0.0005
Epoch: 67/150 Validation Loss: 0.8703 Validation Acc: 0.0000
Epoch: 75/150 Train Loss: 0.6712 Train Acc: 0.0005
Epoch: 75/150 Validation Loss: 0.8055 Validation Acc: 0.0000
Epoch: 84/150 Train Loss: 0.6200 Train Acc: 0.0005
Epoch: 84/150 Validation Loss: 0.7562 Validation Acc: 0.0000
Epoch: 92/150 Train Loss: 0.5753 Train Acc: 0.0005
Epoch: 92/150 Validation Loss: 0.7161 Validation Acc: 0.0000
Epoch: 100/150 Train Loss: 0.5385 Train Acc: 0.0005
Epoch: 100/150 Validation Loss: 0.6819 Validation Acc: 0.0000
Epoch: 109/150 Train Loss: 0.5085 Train Acc: 0.0005
Epoch: 109/150 Validation Loss: 0.6436 Validation Acc: 0.0000
Epoch: 117/150 Train Loss: 0.4857 Train Acc: 0.0005
Epoch: 117/150 Validation Loss: 0.6200 Validation Acc: 0.0000
Epoch: 125/150 Train Loss: 0.4664 Train Acc: 0.0005
Epoch: 125/150 Validation Loss: 0.5994 Validation Acc: 0.0000
Epoch: 134/150 Train Loss: 0.4504 Train Acc: 0.0005
Epoch: 134/150 Validation Loss: 0.5788 Validation Acc: 0.0000
Epoch: 142/150 Train Loss: 0.4378 Train Acc: 0.0005
Epoch: 142/150 Validation Loss: 0.5631 Validation Acc: 0.0000
Epoch: 150/150 Train Loss: 0.4283 Train Acc: 0.0005
Epoch: 150/150 Validation Loss: 0.5510 Validation Acc: 0.0000
'./prova.ckpt'

我认为 ReLU 函数将梯度杀死为零。这可能是我准确的动机吗？

我可以尝试用不同组合的 softmax 来改变激活函数： 1. 仅使用 sigmoid 2.只使用softmax 3.使用ReLU和softmax 但情况没有改变。

为了构建神经网络，我遵循了 Kaggle 中的 Titanic 示例： https://www.kaggle.com/linxinzhe/tensorflow-deep-learning-to-solve-titanic

Answer 1

def split_valid_test_data(data, fraction=(1 - 0.8)):
    data_y=train_data.as_matrix(columns=[train_data.columns[25]])
    data_x = data.drop(["Premio"], axis=1)
    train_x, valid_x, train_y, valid_y = train_test_split(data_x, data_y, test_size=fraction)
    return train_x.values, train_y, valid_x, valid_y

train_x, train_y, valid_x, valid_y = split_valid_test_data(train_data)

print("train_x:{}".format(train_x.shape))
print("train_y:{}".format(train_y.shape))
print("train_y content:{}".format(train_y[:3]))

print("valid_x:{}".format(valid_x.shape))
print("valid_y:{}".format(valid_y.shape))

# 1st layer number of features (neurons)
n_hidden_1 = 50
# 2nd layer number of features (neurons)
n_hidden_2 = 50


##########################
Neural Network
##########################

from collections import namedtuple

def multilayer_perceptron():
    tf.reset_default_graph()
    inputs = tf.placeholder(tf.float32, shape=[None,train_x.shape[1]], name='inputs')
    y = tf.placeholder(tf.float32, shape=[None, 1], name='y')
    weights = {
    'h1': tf.Variable(tf.random_normal([train_x.shape[1], n_hidden_1])),
    'h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),
    'out': tf.Variable(tf.random_normal([n_hidden_2, 1]))
    }
    biases = {
    'b1': tf.Variable(tf.random_normal([n_hidden_1])),
    'b2': tf.Variable(tf.random_normal([n_hidden_2])),
    'out': tf.Variable(tf.random_normal([1]))
    }
    # Hidden layer con 50 neuroni e funzione di attivazione ReLU
    layer_1 = tf.add(tf.matmul(inputs, weights['h1']), biases['b1'], name='Layer_1_mat')
    layer_1 = tf.nn.relu(layer_1, name ='layer_1_relu')
    # Hidden layer with ReLU activation
    layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2'], name='Layer_2_mat')
    layer_2 = tf.nn.relu(layer_2, name ='vars')
    # Output layer with linear activation   
    out_layer = tf.matmul(layer_2, weights['out'], name ='out_layer') + biases['out'] 
    learning_rate = tf.placeholder(tf.float32, name = 'learning_rate')
    is_training=tf.Variable(True,dtype=tf.bool) 
    cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(labels=y,logits=out_layer, name='cross_entropy')
    cost = tf.reduce_mean(cross_entropy, name='cost')  
    with tf.control_dependencies(tf.get_collection(tf.GraphKeys.UPDATE_OPS)):   
         optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
    predicted = tf.nn.sigmoid(out_layer, name='predicted') 
    correct_pred = tf.equal(tf.round(predicted), y)
    accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32), name='accuracy')
    # Export the nodes 
    export_nodes = ['inputs', 'y', 'learning_rate','is_training', 'out_layer',
                    'cost', 'optimizer', 'predicted',  'accuracy'] 
    Graph = namedtuple('Graph', export_nodes)
    local_dict = locals()
    graph = Graph(*[local_dict[each] for each in export_nodes])
    return graph

pred1 = multilayer_perceptron()

#tf.add_to_collection('pred_func', pred1)

############################
#Batch
#############################

def get_batch(data_x,data_y,batch_size=300):
    batch_n=len(data_x)//batch_size
    for i in range(batch_n):
        batch_x=data_x[i*batch_size:(i+1)*batch_size]
        batch_y=data_y[i*batch_size:(i+1)*batch_size]
        yield batch_x,batch_y

epochs = 150
train_collect = 50
train_print=train_collect*2

learning_rate_value = 0.5 #0.0001
batch_size=150

x_collect = []
train_loss_collect = []
train_acc_collect = []
valid_loss_collect = []
valid_acc_collect = []

train_predict = train_data.drop(["Premio"], axis=1)    

saver = tf.train.Saver()

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    iteration=0
    for e in range(epochs):
        for batch_x,batch_y in get_batch(train_x,train_y,batch_size):
            iteration+=1
            feed = {pred1.inputs: train_x,
                    pred1.y: train_y,
                    pred1.learning_rate: learning_rate_value,
                    pred1.is_training:True
                   }
            train_loss, _, train_acc = sess.run([pred1.cost, pred1.optimizer, pred1.accuracy], feed_dict=feed)
            if iteration % train_collect == 0:
                x_collect.append(e)
                train_loss_collect.append(train_loss)
                train_acc_collect.append(train_acc)
                if iteration % train_print==0:
                     print("Epoch: {}/{}".format(e + 1, epochs),
                      "Train Loss: {:.4f}".format(train_loss),
                      "Train Acc: {:.4f}".format(train_acc))       
                feed = {pred1.inputs: valid_x,
                        pred1.y: valid_y,
                        pred1.is_training:False
                       }
                val_loss, val_acc = sess.run([pred1.cost, pred1.accuracy], feed_dict=feed)
                valid_loss_collect.append(val_loss)
                valid_acc_collect.append(val_acc) 
                if iteration % train_print==0:
                    print("Epoch: {}/{}".format(e + 1, epochs),
                      "Validation Loss: {:.4f}".format(val_loss),
                      "Validation Acc: {:.4f}".format(val_acc))
    saver.save(sess, "./prova.ckpt")

train_data.columns[25] 是我要预测的变量 Prime。

我有一个具有 56 个属性的数据集（包括双侧变量 Prime）。对于 df 编码，我使用热向量和二进制技术。数值变量我使用 MinMax Standardlization。