无法让 TensorFlow 做一些琐碎的事情

Question

我有一个向量 x 并想计算一个向量 y 使得 y[j] = x[j]**2 使用 TensorFlow 指定的神经网络，如下所示。效果不太好，错误率很高。
我是不是做错了什么？
任何帮助将不胜感激

它的工作方式是首先在 Xtrain、Ytrain、Xtest 和 Ytest 中生成数据，然后创建占位符变量以启动 TensorFlow。
然后它指定三个隐藏层和一个输出层。然后它进行训练，并使用提要字典创建 Ytest 的预测 Ypred。

import numpy as np
import tensorflow as tf

n = 10
k = 1000
n_hidden = 10
learning_rate = .01
training_epochs = 100000

Xtrain = []
Ytrain = []
Xtest = []
Ytest = []

for i in range(0,k,1):
    X = np.random.randn(1,n)[0]
    Xtrain += [X]
    Ytrain += [Xtrain[-1]**2]
    X = np.random.randn(1,n)[0]
    Xtest += [X]
    Ytest += [Xtest[-1]**2]

x = tf.placeholder(tf.float64,shape = (k,n))
y = tf.placeholder(tf.float64,shape = (k,n))

W1 = tf.Variable(tf.random_normal((n,n_hidden),dtype = tf.float64))
b1 = tf.Variable(tf.random_normal((n_hidden,),dtype = tf.float64))
x_hidden1 = tf.nn.sigmoid(tf.matmul(x,W1) + b1)

W2 = tf.Variable(tf.random_normal((n,n_hidden),dtype = tf.float64))
b2 = tf.Variable(tf.random_normal((n_hidden,),dtype = tf.float64))
x_hidden2 = tf.nn.sigmoid(tf.matmul(x_hidden1,W2) + b2)

W3 = tf.Variable(tf.random_normal((n,n_hidden),dtype = tf.float64))
b3 = tf.Variable(tf.random_normal((n_hidden,),dtype = tf.float64))
x_hidden3 = tf.nn.sigmoid(tf.matmul(x_hidden1,W3) + b3)

W4 = tf.Variable(tf.random_normal((n,n_hidden),dtype = tf.float64))
b4 = tf.Variable(tf.random_normal((n_hidden,),dtype = tf.float64))
y_pred = tf.matmul(x_hidden3,W4) + b4

penalty = tf.reduce_sum(tf.abs((y - y_pred)))
train_op = tf.train.AdamOptimizer(learning_rate).minimize(penalty)

model = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(model)
    for i in range(0,training_epochs):
        sess.run(train_op,{x: Xtrain,y: Ytrain})

    Ypred = y_pred.eval(feed_dict = {x: Xtest})

Answer 1

这段代码做得更好。有人想进一步改进吗？

import numpy as np
import tensorflow as tf

n = 10
k = 1000
n_hidden = 50
learning_rate = .001
training_epochs = 100000

Xtrain = []
Ytrain = []
Xtest = []
Ytest = []

for i in range(0,k,1):
    X = np.random.randn(1,n)[0]
    Xtrain += [X]
    Ytrain += [Xtrain[-1]**2]
    X = np.random.randn(1,n)[0]
    Xtest += [X]
    Ytest += [Xtest[-1]**2]

x = tf.placeholder(tf.float64,shape = (k,n))
y = tf.placeholder(tf.float64,shape = (k,n))

W1 = tf.Variable(tf.random_normal((n,n_hidden),dtype = tf.float64))
b1 = tf.Variable(tf.random_normal((n_hidden,),dtype = tf.float64))
x_hidden1 = tf.nn.sigmoid(tf.matmul(x,W1) + b1)

W2 = tf.Variable(tf.random_normal((n_hidden,n_hidden),dtype = tf.float64))
b2 = tf.Variable(tf.random_normal((n_hidden,),dtype = tf.float64))
x_hidden2 = tf.nn.sigmoid(tf.matmul(x_hidden1,W2) + b2)

W3 = tf.Variable(tf.random_normal((n_hidden,n),dtype = tf.float64))
b3 = tf.Variable(tf.random_normal((n,),dtype = tf.float64))
y_pred = tf.matmul(x_hidden2,W3) + b3

penalty = tf.reduce_sum((y - y_pred)**2)
train_op = tf.train.AdamOptimizer(learning_rate).minimize(penalty)

model = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(model)
    for i in range(0,training_epochs):
        sess.run(train_op,{x: Xtrain,y: Ytrain})

    Ypred = y_pred.eval(feed_dict = {x: Xtest})

Answer 2

这里只是对您的代码进行一些简单的修改：

import numpy as np
import tensorflow as tf

n = 10
k = 1000
learning_rate = 1e-3
training_epochs = 100000

# It will be better for you to use PEP8 style

# None here will allow you to feed data with ANY k size
x = tf.placeholder(tf.float64, shape=(None, n))
y = tf.placeholder(tf.float64, shape=(None, n))

# Use default layer constructors
# from your implementation it uses another random initializer
out = tf.layers.dense(x, 100)
out = tf.layers.batch_normalization(out)
# ReLU is better than sigmoid, there are a lot of articles about it
out = tf.nn.relu(out)

out = tf.layers.dense(out, 200)
out = tf.layers.batch_normalization(out)
out = tf.nn.relu(out)

out = tf.layers.dense(out, n)

# total loss = mean L1 for samples
# each sample is a vector of 10 values, so you need to calculate
# sum along first axis, and them calculate mean of sums
l1 = tf.reduce_mean(tf.reduce_sum(tf.abs(y - out), axis=1))
train_op = tf.train.AdamOptimizer(learning_rate).minimize(l1)

model = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(model)
    for i in range(training_epochs):
        xs = np.random.randn(k, n)
        ys = xs ** 2
        _, l1_value = sess.run(
            [train_op, l1],
            feed_dict={x: xs, y: ys})
        if (i + 1) % 10000 == 0 or i == 0:
            print('Current l1({}/{}) = {}'.format(
                i + 1, training_epochs, l1_value))

    xs = np.random.randn(k, n)
    ys = xs ** 2
    test_l1 = sess.run(l1, feed_dict={x: xs, y: ys})
    print ('Total l1 at test = {}'.format(test_l1))

输出：

Current l1(1/100000) = 11.0853215657
Current l1(10000/100000) = 0.126037403282
Current l1(20000/100000) = 0.096445475666
Current l1(30000/100000) = 0.0719392853473
Current l1(40000/100000) = 0.0690671103719
Current l1(50000/100000) = 0.07661241544
Current l1(60000/100000) = 0.0743827124406
Current l1(70000/100000) = 0.0656016587469
Current l1(80000/100000) = 0.0675546809828
Current l1(90000/100000) = 0.0649035400487
Current l1(100000/100000) = 0.0583308788607
Total l1 at test = 0.0613149096968

使用其他一些架构、学习率、批量大小、时期数、损失函数等可以增强总惩罚

看起来架构可能会增加，然后你将能够长时间运行训练以获得1e-3。

有关其工作原理和操作方法的更多信息，请访问CS231 course。

附：这是关于数据馈送的一些假设：我测试的一些数据可能在训练过程中。因为任务很简单所以没关系，但最好保证没有任何训练样本会在测试集中。