【发布时间】:2018-12-28 21:31:14
【问题描述】:
我正在使用 Pandas 构建机器学习模型,但很难将我的模型应用于测试用户输入的数据。我的数据基本上是一个包含 2 列的数据框:文本和情绪。我希望能够预测用户输入的情绪。这是我的工作:
1。训练/测试模型
# reading dataset
df = pd.read_csv('dataset/dataset.tsv', sep='\t')
# splitting training/test set
test_size = 0.1
train_x, valid_x, train_y, valid_y = model_selection.train_test_split(df['text'], df['sentiment'], test_size=test_size)
# label encode the target variable (i.e. negative = 0, positive = 1)
encoder = preprocessing.LabelEncoder()
train_y = encoder.fit_transform(train_y)
valid_y = encoder.fit_transform(valid_y)
# create a count vectorizer object
count_vect = CountVectorizer(analyzer='word', token_pattern=r'\w{1,}')
count_vect.fit(df['text'])
# transform the training and validation data using count vectorizer object
xtrain_count = count_vect.transform(train_x)
xvalid_count = count_vect.transform(valid_x)
# function to train the model
def train_model(classifier, feature_vector_train, label, feature_vector_valid, name):
# fit the training dataset on the classifier
classifier.fit(feature_vector_train, label)
# save the trained model in the "models" folder
joblib.dump(classifier, 'models/' + name + '.pkl')
# predict the labels on validation dataset
predictions = classifier.predict(feature_vector_valid)
return metrics.accuracy_score(predictions, valid_y)
# Naive Bayes on Count Vectors
accuracy = train_model(naive_bayes.MultinomialNB(), xtrain_count, train_y, xvalid_count, 'NB-COUNT')
print("NB, Count Vectors: ", accuracy)
一切正常,准确率约为 80%
2。根据用户输入测试模型
然后我再次读取保存的模型,获取用户输入并尝试进行预测(用户输入现在被硬编码在input_text):
clf = joblib.load('models/NB-COUNT.pkl')
dataset_df = pd.read_csv('dataset/dataset.tsv', sep='\t')
input_text = 'stackoverflow is the best' # the sentence I want to predict the sentiment for
test_df = pd.Series(data=input_text)
count_vect = CountVectorizer(analyzer='word', token_pattern=r'\w{1,}')
count_vect.fit(dataset_df['text']) # fit the count vectorizer again so we can extract features from test_df
features = count_vect.transform(test_df)
result = clf.predict(features)[0]
print(result)
但我得到的错误是“尺寸不匹配”:
Traceback (most recent call last):
File "C:\Users\vdvax\iCloudDrive\Freelance\09. Arabic Sentiment Analysis\test.py", line 20, in <module>
result = clf.predict(features)[0]
File "C:\Python36\lib\site-packages\sklearn\naive_bayes.py", line 66, in predict
jll = self._joint_log_likelihood(X)
File "C:\Python36\lib\site-packages\sklearn\naive_bayes.py", line 725, in _joint_log_likelihood
return (safe_sparse_dot(X, self.feature_log_prob_.T) +
File "C:\Python36\lib\site-packages\sklearn\utils\extmath.py", line 135, in safe_sparse_dot
ret = a * b
File "C:\Python36\lib\site-packages\scipy\sparse\base.py", line 515, in __mul__
raise ValueError('dimension mismatch')
ValueError: dimension mismatch
【问题讨论】:
-
感谢您的回复。是否可以以某种方式保存计数矢量化器,以便稍后加载?因为训练好的模型被保存到一个文件中,然后加载到另一个脚本中。计数矢量化器是否有类似的可能?
-
您可以将所有预处理器堆叠到管道中,然后腌制就好了。
-
你能告诉我怎么做(代码部分)吗?我对管道不熟悉。
标签: python pandas machine-learning scikit-learn