分类器拟合和部分拟合的内存错误

Question

我正在尝试使用 scikit-learn 来预测输入文本字符串的值。我正在使用 HashingVectorizer 进行数据矢量化和 PassiveAggressiveClassifier 使用 partial_fit 进行学习（参考以下代码）：

from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np
from sklearn.naive_bayes import GaussianNB
from sklearn.multiclass import OneVsRestClassifier
from sklearn.svm import LinearSVC
from sklearn import metrics
from sklearn.metrics import zero_one_loss
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import PassiveAggressiveClassifier, SGDClassifier, Perceptron
from sklearn.pipeline import make_pipeline
from sklearn.externals import joblib
import pickle

a,r = [],[]

vectorizer = TfidfVectorizer()

with open('val', 'rb') as f:
    r = pickle.load(f)

with open('text', 'rb') as f:
    a = pickle.load(f)

L = (vectorizer.fit_transform(a))

training_set = L[:3250]
testing_set = L[3250:]

M = np.array(r)

training_result = M[:3250]
testing_result = M[3250:]

cls = np.unique(r)

model = PassiveAggressiveClassifier()

model.partial_fit(training_set, training_result, classes=cls)
print(model)
predicted = model.predict(testing_set)

print testing_result
print predicted

错误日志：

File "try.py", line 89, in <module>
    model.partial_fit(training_set, training_result, classes=cls)
  File "/usr/local/lib/python2.7/dist-packages/sklearn/linear_model/passive_aggressive.py", line 115, in partial_fit
    coef_init=None, intercept_init=None)
  File "/usr/local/lib/python2.7/dist-packages/sklearn/linear_model/stochastic_gradient.py", line 374, in _partial_fit
    coef_init, intercept_init)
  File "/usr/local/lib/python2.7/dist-packages/sklearn/linear_model/stochastic_gradient.py", line 167, in _allocate_parameter_mem
    dtype=np.float64, order="C")
MemoryError

我之前使用 CountVectorizer 和 Logical Regression 进行分类，并且没有问题。 但我的学习数据约为。数百万行，我想使用上述脚本实现 增量学习，这会导致每次执行时 Memory Error。

更新：

在循环中应用部分学习后，partial_fit 函数返回不匹配的特征数错误(ValueError: Number of features 8897 does not match previous data 9190.) 此外，即使我设置了最大特征属性，生成的预测也是不正确的。 有什么方法可以让 partial_fit 方法获取可变数量的特征？

执行输出：

(400, 8481)
(400, 9277)
Traceback (most recent call last):
  File "f9.py", line 65, in <module>
    training_set, training_result, classes=cls)
  File "/usr/local/lib/python2.7/dist-packages/sklearn/linear_model/passive_aggressive.py", line 115, in partial_fit
    coef_init=None, intercept_init=None)
  File "/usr/local/lib/python2.7/dist-packages/sklearn/linear_model/stochastic_gradient.py", line 379, in _partial_fit
    % (n_features, self.coef_.shape[-1]))
ValueError: Number of features 9277 does not match previous data 8481.

任何帮助将不胜感激。

谢谢！

Answer 1

内存错误来自内存中的数据过多。当您加载数据时，您的数量等于 N，然后当您进行 partial_fit 时，根据算法，它将存储更多数据，可能接近 N。

您不需要存储两倍的数据。尝试减小初始数据块的大小。将它分成几个部分，您将提供给partial_fit 方法。

您应该逐行读取文件以创建数据块，然后适应该块，刷新内存，然后再做一次

with open(path, "r", encoding='utf-8') as f:
    i = 0
    for line in f:
        % Create chunk of X line
        i ++
        arr.add(line)

        % Learn with partial_fit
        if (i == X):
            model.partial_fit()
            % Flush the last chunk 
            arr = []