在 sklearn 中使用 CountVectorizer 和 OneHotEncoder 的列转换答案

【问题标题】：Working With Column transformation for CountVectorizer and OneHotEncoder in sklearn在 sklearn 中使用 CountVectorizer 和 OneHotEncoder 的列转换
【发布时间】：2020-07-15 08:13:47
【问题描述】：

我有虚拟数据框，带有列文本和车辆，我想将 Countvectorizer 用于文本列，将 onehotencoding 用于车辆列

import pandas as pd 
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import make_column_transformer
from sklearn.feature_extraction.text import CountVectorizer

df  = pd.DataFrame([['how are you','car'],['good mrng have a nice day','bike'],['today is my best working day','cycle'],['hello','bike']], columns = ['text','vehicle'])

preprocess = make_column_transformer((CountVectorizer(), ['text']),(OneHotEncoder(), ['vehicle']))
preprocess.fit_transform(df)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-15-d7644861c938> in <module>()
----> 1 preprocess.fit_transform(df)

~\AppData\Roaming\Python\Python36\site-packages\sklearn\compose\_column_transformer.py in 
fit_transform(self, X, y)
469         self._validate_output(Xs)
470 
--> 471         return self._hstack(list(Xs))
472 
473     def transform(self, X):

~\AppData\Roaming\Python\Python36\site-packages\sklearn\compose\_column_transformer.py in 
_hstack(self, Xs)
526         else:
527             Xs = [f.toarray() if sparse.issparse(f) else f for f in Xs]
--> 528             return np.hstack(Xs)
529 
530 

C:\ProgramData\Anaconda3\lib\site-packages\numpy\core\shape_base.py in hstack(tup)
338         return _nx.concatenate(arrs, 0)
339     else:
--> 340         return _nx.concatenate(arrs, 1)
341 
342 

ValueError: all the input array dimensions except for the concatenation axis must match exactly

这个错误是因为两个变压器的输出不同

vect  = CountVectorizer()
vect.fit_transform(df['text'])
#op
<4x14 sparse matrix of type '<class 'numpy.int64'>'
with 15 stored elements in Compressed Sparse Row format>

encoder = OneHotEncoder(handle_unknown='ignore')
encoder.fit_transform(df['vehicle'].to_numpy().reshape(-1, 1)).toarray()

#op
 array([[0., 1., 0.],
   [1., 0., 0.],
   [0., 0., 1.],
   [1., 0., 0.]])

如何应用.to_numpy().reshape(-1,1)，或者有没有其他方法可以实现这个？？？

【问题讨论】：

我没有使用 make_column_transformer 的经验，但我相信您可以使用 FeatureUnion 做到这一点

标签： python machine-learning scikit-learn one-hot-encoding countvectorizer

【解决方案1】：

import pandas as pd 
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import make_column_transformer
from sklearn.feature_extraction.text import CountVectorizer

df  = pd.DataFrame([['how are you','car'],['good mrng have a nice day','bike'],['today is my best working day','cycle'],['hello','bike']], columns = ['text','vehicle'])

变化就在这里：

preprocess = make_column_transformer((CountVectorizer(), 'text'),(OneHotEncoder(), ['vehicle']))

不是在列表中传递“文本”，而是必须采用字符串格式。我相信这更像是一种安全机制，可以防止将多列传入一个 CountVectorizer。

【讨论】：