重塑输入标记化文本以预测 lstm rnn 中的情绪时出错

Question

我是神经网络的新手，一直在学习它在文本分析领域的应用，所以我在 python 中的应用中使用了 lstm rnn。

在尺寸为 20,000*1 的数据集上训练模型后（2000 表示文本，1 表示文本的情绪），我得到了 99% 的良好准确率，之后我验证了正在运行的模型很好（使用 model.predict() 函数）。

现在只是为了测试我的模型，我一直在尝试从数据框或包含一些文本的变量中提供随机文本输入，但我总是遇到重塑数组的错误，其中需要输入到 rnn 模型尺寸为 (1,30)。

但是当我将训练数据重新输入到模型中进行预测时，模型工作得非常好，为什么会发生这种情况？

link for the screenshot of error

link for image of model summary

training data

我只是被困在这里，任何类型的建议都将帮助我了解更多关于 rnn 的信息，我在此请求中附上了错误和 rnn 模型代码。

谢谢

问候

图沙尔·乌帕迪耶

    import numpy as np 
    import pandas as pd 
    import keras
    import sklearn

    from sklearn.feature_extraction.text import CountVectorizer
    from keras.preprocessing.text import Tokenizer
    from keras.preprocessing.sequence import pad_sequences
    from keras.models import Sequential
    from keras.layers import Dense, Embedding, LSTM
     from sklearn.model_selection import train_test_split
    from keras.utils.np_utils import to_categorical
    import re


    data=pd.read_csv('..../twitter_tushar_data.csv')
    max_fatures = 4000
    tokenizer = Tokenizer(num_words=max_fatures, split=' ')
    tokenizer.fit_on_texts(data['tweetText'].values)
    X = tokenizer.texts_to_sequences(data['tweetText'].values)
    X = pad_sequences(X)


    embed_dim = 128
    lstm_out = 196
    model = Sequential()
    keras.layers.core.SpatialDropout1D(0.2) #used to avoid overfitting
    model.add(Embedding(max_fatures, embed_dim,input_length = X.shape[1]))
    model.add(LSTM(196, recurrent_dropout=0.2, dropout=0.2))
   model.add(Dense(2,activation='softmax'))
   model.compile(loss = 'categorical_crossentropy', optimizer='adam',metrics 
   = ['accuracy'])
   print(model.summary())
   #splitting data in training and testing parts

   Y = pd.get_dummies(data['SA']).values
   X_train, X_test, Y_train, Y_test = train_test_split(X,Y, test_size = 
   0.30, random_state = 42)
   print(X_train.shape,Y_train.shape)
   print(X_test.shape,Y_test.shape)
   batch_size = 128
   model.fit(X_train, Y_train, epochs = 7, batch_size=batch_size, verbose = 
   2)


   validation_size = 3500

   X_validate = X_test[-validation_size:]
   Y_validate = Y_test[-validation_size:]
   X_test = X_test[:-validation_size]
   Y_test = Y_test[:-validation_size]
   score,acc = model.evaluate(X_test, Y_test, verbose = 2, batch_size = 128)
   print("score: %.2f" % (score))
   print("acc: %.2f" % (acc))


   pos_cnt, neg_cnt, pos_correct, neg_correct = 0, 0, 0, 0
   for x in range(len(X_validate)):
   result = 
 model.predict(X_validate[x].reshape(1,X_test.shape[1]),batch_size=1,verbose 
 = 2)[0]
 if np.argmax(result) == np.argmax(Y_validate[x]):
    if np.argmax(Y_validate[x]) == 0:
        neg_correct += 1
    else:
        pos_correct += 1

if np.argmax(Y_validate[x]) == 0:
    neg_cnt += 1
else:
    pos_cnt += 1
print("pos_acc", pos_correct/pos_cnt*100, "%")
print("neg_acc", neg_correct/neg_cnt*100, "%")

Answer 1

我得到了我的问题的解决方案，这只是正确标记输入的问题，谢谢！下面的代码用于预测不同的用户输入..

 text=np.array(['you are a pathetic awful movie'])
 print(text.shape)
 tk=Tokenizer(num_words=4000,lower=True,split=" ")
 tk.fit_on_texts(text)

prediction=model.predict(sequence.pad_sequences(tk.texts_to_sequences(text),
maxlen=max_review_length))
print(prediction)
print(np.argmax(prediction))