scikit 学习转换多个文本特征

Question

我正在尝试将多个文本功能分类为一种状态。数据包括来自具有组件的不同服务器的消息（错误和警告），并将导致不同的状态。例如：

ServerName     Name     Description                               Severity   State
-------------- -------- ----------------------------------------- ---------- -------------
QWERT-XY-123   MySQL    Service not available on target machine   error      important
QWERT-XY-146   Oracle   Service caused an error                   warning    unimportant
...    

这是矢量化的一部分：

from sklearn.feature_extraction.text import HashingVectorizer

vectorizer = HashingVectorizer()

X_Servername = df["ServerName"].values
X_Name = df["Name"].values
X_Description = df["Description"].values
X_Severity = df["Severity"].values
y = df["State"].values

X_Servername = vectorizer.transform(X_Servername)
X_Name = vectorizer.transform(X_Name)
X_Description = vectorizer.transform(X_Description)

features=list(zip(X_Servername,X_Name,X_Description,X_Severity))

现在我想拟合模型：

from sklearn.svm import SVC

model = SVC(kernel = "linear", probability=True)
model.fit(features, y)

结果如下错误：

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-183-71455dd49f0b> in <module>()
  2 
  3 model = SVC(kernel = "linear", probability=True)
----> 4 model.fit(features, y)
  5 
  6 #print(model.score(X_test, y))

D:\Enviroment\Anaconda3\lib\site-packages\sklearn\svm\base.py in fit(self, X, y, sample_weight)
147         self._sparse = sparse and not callable(self.kernel)
148 
149 -->     X, y = check_X_y(X, y, dtype=np.float64, order='C', accept_sparse='csr')
150         y = self._validate_targets(y)
151 

D:\Enviroment\Anaconda3\lib\site-packages\sklearn\utils\validation.py in check_X_y(X, y, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, warn_on_dtype, estimator)
571     X = check_array(X, accept_sparse, dtype, order, copy, force_all_finite,
572                     ensure_2d, allow_nd, ensure_min_samples,
573 -->                 ensure_min_features, warn_on_dtype, estimator)
574     if multi_output:
575         y = check_array(y, 'csr', force_all_finite=True, ensure_2d=False,

D:\Enviroment\Anaconda3\lib\site-packages\sklearn\utils\validation.py in check_array(array, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, warn_on_dtype, estimator)
431                                       force_all_finite)
432     else:
433 -->     array = np.array(array, dtype=dtype, order=order, copy=copy)
434 
435         if ensure_2d:

ValueError: setting an array element with a sequence.

所以我的问题是关于如何使用 hashingvectorizer 的多个功能，或者是将所有功能放在一行中的唯一方法？

感谢您的帮助。

更新

失败者在于如何构建矢量化特征列表。而不是：

features=list(zip(X_Servername,X_Name,X_Description,X_Severity))

我现在使用这个函数，其中extracted 附加所有创建的矢量化值（X_ServerName、X_Name、...）：

def combine(extracted):
    if any(sparse.issparse(fea) for fea in extracted):
        stacked = sparse.hstack(extracted).tocsr()
        stacked = stacked.toarray()
    else:
        stacked = np.hstack(extracted)

    return stacked

Answer 1

请尝试以下代码：

from sklearn_pandas import DataFrameMapper, gen_features
from sklearn.feature_extraction.text import HashingVectorizer
from sklearn.preprocessing import LabelEncoder

cat_features = ["ServerName", "Name", "Description", "Severity"]
gf = gen_features(cat_features, [HashingVectorizer])
mapper = DataFrameMapper(gf)
cat_features_transformed = mapper.fit_transform(df)

target_name_encoded = LabelEncoder().fit_transform(df["State"])

from sklearn.svm import SVC

model = SVC(kernel = "linear", probability=True)
model.fit(cat_features_transformed, target_name_encoded)
SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
  decision_function_shape='ovr', degree=3, gamma='auto_deprecated',
  kernel='linear', max_iter=-1, probability=True, random_state=None,
  shrinking=True, tol=0.001, verbose=False)

### For test/prediction part ###

test_features_transformed = mapper.transform(df_test)
predictions = model.predict(test_features_transformed)

注意，您可能需要运行

pip install sklearn-pandas

如果您的机器上没有安装sklearn-pandas。

上述解决方案将允许您 (1) 将数据转换为合适的格式，然后 (2) 通过 transform 方法将相同的拟合转换应用于您的测试数据。

如果这有帮助，请告诉我们