对 imdb 电影评论数据实施情感分析的问题

Question

我正在对 imdb 电影评论数据集进行情感分析，并在使用 LinearSVC() 进行预测时出现值错误。

# STOP IS FOR STOPWORDS

trainset,testset=dataloader(r'C:\Users\kkk\Desktop\nlp\aclImdb_v1\aclImdb')

trainset["text"]=trainset["text"].apply(lambda x:' '.join([word for word in x.split() if word not in (stop)] ))
trainset.iloc[1]["text"]
testset["text"]=testset["text"].apply(lambda x:' '.join([word for word in x.split() if word not in (stop)] ))
trainset["text"]=trainset["text"].apply(lambda x:x.lower())
replacebyspace=re.compile('[/(){}\[\]\|@,;]')
badwords=re.compile('[^0-9a-z #+_]')     
testset["text"]=testset["text"].apply(lambda x:re.sub(replacebyspace," ",x))
trainset["text"]=trainset["text"].apply(lambda x:re.sub(replacebyspace," ",x))
from sklearn.svm import LinearSVC
from sklearn.metrics import accuracy_score
from sklearn.feature_extraction.text import TfidfVectorizer
nltk.download('wordnet')
stemmer = nltk.stem.WordNetLemmatizer()
tokenizer = nltk.tokenize.TreebankWordTokenizer()
vectorizer = TfidfVectorizer(ngram_range=(1, 2))
trainfeatures=vectorizer.fit_transform(trainset["text"])
testfeatures=vectorizer.fit_transform(testset["text"])
model=LinearSVC()
model.fit(trainfeatures,trainset["sentiment"])
pred=model.predict(testfeatures)

我希望模型可以工作，但得到了错误

Traceback (most recent call last):

  File "<ipython-input-65-e537b07a6a6a>", line 3, in <module>
    pred=model.predict(testfeatures)

  File "C:\Users\kkk\Anaconda3\lib\site-packages\sklearn\linear_model\base.py", line 281, in predict
    scores = self.decision_function(X)

  File "C:\Users\kk\Anaconda3\lib\site-packages\sklearn\linear_model\base.py", line 262, in decision_function
    % (X.shape[1], n_features))

ValueError: X has 1860172 features per sample; expecting 1906325

Answer 1

替换这个

testfeatures=vectorizer.fit_transform(testset["text"])

与

testfeatures=vectorizer.transform(testset["text"])