如何在 tensorflow 0.11 中使用 RNN 编写多维回归预测器

Question

这是我实际尝试做的玩具版本。我在大量时间步长（150,000 步）上拥有非常高维的输入数据（2e05 到 5e06 维）。我知道我最终可能需要对状态进行一些嵌入/压缩（请参阅this question）。但是，让我们先把它放在一边。

以这个11维的玩具输入数据为例

t  Pattern
0  0,0,0,0,0,0,0,0,0,2,1 
1  0,0,0,0,0,0,0,0,2,1,0 
2  0,0,0,0,0,0,0,2,1,0,0 
n  ...

我希望 RNN 学会将当前时间步长与下一个时间步长相关联，这样如果输入 (x) 为 t0，则所需的输出 (y) 为 t1。

使用 RNN 的想法是，我一次只能为网络提供一个时间步长（由于我的真实数据的维数很大）。由于输入和输出的数量相同，我不确定基本的 RNN 是否合适。我看了一点 seq2seq 教程，但我不确定这个应用程序是否需要编码器/解码器，而且我无法使用我的玩具数据去任何地方。

以下是我所能想到的，但它根本没有收敛。我错过了什么？

import numpy as np
import tensorflow as tf

# Imports for loading CSV file
from tensorflow.python.platform import gfile 
import csv

# Input sequence
wholeSequence = [[0,0,0,0,0,0,0,0,0,2,1],
                 [0,0,0,0,0,0,0,0,2,1,0],
                 [0,0,0,0,0,0,0,2,1,0,0],
                 [0,0,0,0,0,0,2,1,0,0,0],
                 [0,0,0,0,0,2,1,0,0,0,0],
                 [0,0,0,0,2,1,0,0,0,0,0],
                 [0,0,0,2,1,0,0,0,0,0,0],
                 [0,0,2,1,0,0,0,0,0,0,0],
                 [0,2,1,0,0,0,0,0,0,0,0],
                 [2,1,0,0,0,0,0,0,0,0,0]]

data = np.array(wholeSequence[:-1], dtype=int) # all but last
target = np.array(wholeSequence[1:], dtype=int) # all but first
trainingSet = tf.contrib.learn.datasets.base.Dataset(data=data, target=target)
trainingSetDims = trainingSet.data.shape[1]

EPOCHS = 10000
PRINT_STEP = 1000

x_ = tf.placeholder(tf.float32, [None, trainingSetDims])
y_ = tf.placeholder(tf.float32, [None, trainingSetDims])

cell = tf.nn.rnn_cell.BasicRNNCell(num_units=trainingSetDims)

outputs, states = tf.nn.rnn(cell, [x_], dtype=tf.float32)
outputs = outputs[-1]

W = tf.Variable(tf.random_normal([trainingSetDims, 1]))     
b = tf.Variable(tf.random_normal([trainingSetDims]))

y = tf.matmul(outputs, W) + b

cost = tf.reduce_mean(tf.square(y - y_))
train_op = tf.train.RMSPropOptimizer(0.005, 0.2).minimize(cost)

with tf.Session() as sess:
    tf.initialize_all_variables().run()
    for i in range(EPOCHS):
        sess.run(train_op, feed_dict={x_:trainingSet.data, y_:trainingSet.target})
        if i % PRINT_STEP == 0:
            c = sess.run(cost, feed_dict={x_:trainingSet.data, y_:trainingSet.target})
            print('training cost:', c)

    response = sess.run(y, feed_dict={x_:trainingSet.data})
    print(response)

方法来自this thread。

最后我想使用 LSTM，重点是对序列进行建模，以便通过使用 t0 启动网络来重建整个序列的近似值，然后将预测反馈为下一个输入。

EDIT1

我现在看到成本降低了，因为我添加了以下代码，以便在训练之前将直方图输入数据重新调整为概率分布：

# Convert hist to probability distribution
wholeSequence = np.array(wholeSequence, dtype=float) # Convert to NP array.
pdfSequence = wholeSequence*(1./np.sum(wholeSequence)) # Normalize to PD.

data = pdfSequence[:-1] # all but last
target = pdfSequence[1:] # all but first

输出仍然没有像输入那样出现，所以我肯定遗漏了一些东西：

('training cost:', 0.49993864)
('training cost:', 0.0012213766)
('training cost:', 0.0010471855)
('training cost:', 0.00094231067)
('training cost:', 0.0008385859)
('training cost:', 0.00077578216)
('training cost:', 0.00071381911)
('training cost:', 0.00063783216)
('training cost:', 0.00061271922)
('training cost:', 0.00059178629)
[[ 0.02012676  0.02383044  0.02383044  0.02383044  0.02383044  0.02383044
   0.02383044  0.02383044  0.02383044  0.01642305  0.01271933]
 [ 0.02024871  0.02395239  0.02395239  0.02395239  0.02395239  0.02395239
   0.02395239  0.02395239  0.02395239  0.016545    0.01284128]
 [ 0.02013803  0.02384171  0.02384171  0.02384171  0.02384171  0.02384171
   0.02384171  0.02384171  0.02384171  0.01643431  0.0127306 ]
 [ 0.020188    0.02389169  0.02389169  0.02389169  0.02389169  0.02389169
   0.02389169  0.02389169  0.02389169  0.01648429  0.01278058]
 [ 0.02020025  0.02390394  0.02390394  0.02390394  0.02390394  0.02390394
   0.02390394  0.02390394  0.02390394  0.01649654  0.01279283]
 [ 0.02005926  0.02376294  0.02376294  0.02376294  0.02376294  0.02376294
   0.02376294  0.02376294  0.02376294  0.01635554  0.01265183]
 [ 0.02034193  0.02404562  0.02404562  0.02404562  0.02404562  0.02404562
   0.02404562  0.02404562  0.02404562  0.01663822  0.01293451]
 [ 0.02057907  0.02428275  0.02428275  0.02428275  0.02428275  0.02428275
   0.02428275  0.02428275  0.02428275  0.01687536  0.01317164]
 [ 0.02042386  0.02412754  0.02412754  0.02412754  0.02412754  0.02412754
   0.02412754  0.02412754  0.02412754  0.01672015  0.01301643]]

Answer 1

我放弃了直接使用tensowflow，最终使用Keras。以下是使用带有第二个密集层的单层 LSTM 学习上述玩具序列的代码：

import numpy as np

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM

# Input sequence
wholeSequence = [[0,0,0,0,0,0,0,0,0,2,1],
                 [0,0,0,0,0,0,0,0,2,1,0],
                 [0,0,0,0,0,0,0,2,1,0,0],
                 [0,0,0,0,0,0,2,1,0,0,0],
                 [0,0,0,0,0,2,1,0,0,0,0],
                 [0,0,0,0,2,1,0,0,0,0,0],
                 [0,0,0,2,1,0,0,0,0,0,0],
                 [0,0,2,1,0,0,0,0,0,0,0],
                 [0,2,1,0,0,0,0,0,0,0,0],
                 [2,1,0,0,0,0,0,0,0,0,0]]

# Preprocess Data: (This does not work)
wholeSequence = np.array(wholeSequence, dtype=float) # Convert to NP array.
data = wholeSequence[:-1] # all but last
target = wholeSequence[1:] # all but first

# Reshape training data for Keras LSTM model
# The training data needs to be (batchIndex, timeStepIndex, dimentionIndex)
# Single batch, 9 time steps, 11 dimentions
data = data.reshape((1, 9, 11))
target = target.reshape((1, 9, 11))

# Build Model
model = Sequential()  
model.add(LSTM(11, input_shape=(9, 11), unroll=True, return_sequences=True))
model.add(Dense(11))
model.compile(loss='mean_absolute_error', optimizer='adam')
model.fit(data, target, nb_epoch=2000, batch_size=1, verbose=2)