为什么在 Tensorflow nan 中丢失了我的简单 NN？

Question

我创建了这个简单的神经网络。但是，每次迭代中的损失总是'nan'。我以前用过这个算法，而且一直有效。

下面是带有相应输入的代码：

import pandas as pd
import numpy as np
import tensorflow as tf
from sklearn.metrics import accuracy_score
import sys

#load test and train sets
xtrain = pd.read_csv(r'xtrain.csv', decimal=',').astype('float32')
ytrain = pd.read_csv(r'ytrain.csv', decimal=',').astype('float32')
xtest = pd.read_csv(r'xtest.csv', decimal=',').astype('float32')
ytest = pd.read_csv(r'ytest.csv', decimal=',').astype('float32')

# Convert from pandas data frame
xtrain = xtrain.values
ytrain = ytrain.values

n_inputs = 43
n_hidden1 = 300
n_hidden2 = 200
n_outputs = 5

learning_rate = 0.001
training_epochs = 2

batch_size = 100
display_step = 1

X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int64, shape=(None, n_outputs), name="y")
training = tf.placeholder_with_default(False, shape=(), name='training')

with tf.name_scope("dnn"):
    he_init = tf.contrib.layers.variance_scaling_initializer()
    hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.relu, kernel_initializer=he_init, name="hidden1")
    hidden2 = tf.layers.dense(hidden1, n_hidden2, name="hidden2", activation=tf.nn.relu)
    logits = tf.layers.dense(hidden2, n_outputs, name="outputs")

# Reduce loss
with tf.name_scope("loss"):
    loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))


# Define optimizer
with tf.name_scope("train"):
    optimizer = tf.train.AdamOptimizer(learning_rate)
    training_op = optimizer.minimize(loss)

with tf.name_scope("eval"):
    correct_prediction = tf.equal(tf.argmax(logits, 1),tf.argmax(y, 1))
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))


init = tf.global_variables_initializer()
saver = tf.train.Saver()
avg_set = []
epoch_set = []

accuracy_summary = tf.summary.scalar('accuracy', accuracy)
cost_summary = tf.summary.scalar('loss', loss)

with tf.Session() as sess:
    sess.run(init)

    for epoch in range (training_epochs):
        avg_cost = 0.

        sess.run(training_op, feed_dict={X: xtrain, y: ytrain})

        avg_cost += sess.run(loss, feed_dict={X: xtrain, y: ytrain})
        acc_train = accuracy.eval(feed_dict={X: xtrain, y: ytrain})
        acc_test = accuracy.eval(feed_dict={X: xtest, y: ytest})

        if epoch % display_step == 0:
            print("Epoch: ", '%04d' % (epoch + 1), "loss = ", "{:.9f}".format(avg_cost), "accuracy-train = ",
                  "{:.9f}".format(acc_train), "accuracy-test = ", "{:.9f}".format(acc_test))
            avg_set.append(avg_cost)
            epoch_set.append(epoch + 1)
    print("Training phase finished")

    save_path = saver.save(sess, "./my_model_final.ckpt")

运行代码时，我得到以下结果：

 Epoch:  0001 loss =  nan accuracy-train =  0.073428266 accuracy-test =  0.071483918
Epoch:  0002 loss =  nan accuracy-train =  0.073428266 accuracy-test =  0.071483918
Epoch:  0003 loss =  nan accuracy-train =  0.073428266 accuracy-test =  0.071483918
Epoch:  0004 loss =  nan accuracy-train =  0.073428266 accuracy-test =  0.071483918
Epoch:  0005 loss =  nan accuracy-train =  0.073428266 accuracy-test =  0.071483918
Epoch:  0006 loss =  nan accuracy-train =  0.073428266 accuracy-test =  0.071483918
Epoch:  0007 loss =  nan accuracy-train =  0.073428266 accuracy-test =  0.071483918

有人知道为什么损失总是 nan 吗？多年来我一直试图找出错误，非常感谢您的帮助。

Answer 1

我发现了一些可疑的东西。为什么y 的类型是tf.int64 而不是tf.float32？当使用softmax_cross_entropy_with_logits 时，标签应该是一个概率分布，加起来等于1.0。如果标签是正确类的索引，那么您应该使用sparse_softmax_cross_entropy_with_logits。