仅矢量化文本列并使用管道标准化数字列

Question

我正在尝试对文本列进行矢量化，然后对数值列进行标准化。以下是python脚本：

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.compose import ColumnTransformer
from sklearn.compose import make_column_transformer


numeric_features = ["sklearn_cosine_similarity"] # numeric feature

categorical_features = ["clean_text"] # text feature
categorical_transformer = CountVectorizer()

preprocess = make_column_transformer((CountVectorizer(), categorical_features),(StandardScaler(), numeric_features))

pipeline = Pipeline(
    [
        ("vect", preprocess),
        ("knnr", KNeighborsRegressor())
    ]
)

parameters = {
    "vect__countvectorizer__max_features" : [None, 50,100],
    "vect__countvectorizer__ngram_range" : [(1, 1)],  # unigrams
    "knnr__n_neighbors" : [3,4,5],
    "knnr__weights" :['uniform', 'distance'],
    "knnr__leaf_size" :[20],
    'knnr__metric' : ['euclidean']
}

grid_search = GridSearchCV(pipeline, parameters, verbose=4,scoring='neg_root_mean_squared_error',cv=3)

grid_search.fit(X_train[['clean_text','sklearn_cosine_similarity']].values, y_train.values)

type(X_train[['clean_text','sklearn_cosine_similarity']]) is pandas.core.frame.DataFrame

type(y_train) is pandas.core.frame.DataFrame

在运行 fit 时出现以下错误，

Fitting 3 folds for each of 18 candidates, totalling 54 fits
[CV 1/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=3, knnr__weights=uniform, vect__countvectorizer__max_features=None, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 2/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=3, knnr__weights=uniform, vect__countvectorizer__max_features=None, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 3/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=3, knnr__weights=uniform, vect__countvectorizer__max_features=None, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 1/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=3, knnr__weights=uniform, vect__countvectorizer__max_features=50, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 2/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=3, knnr__weights=uniform, vect__countvectorizer__max_features=50, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 3/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=3, knnr__weights=uniform, vect__countvectorizer__max_features=50, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 1/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=3, knnr__weights=uniform, vect__countvectorizer__max_features=100, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 2/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=3, knnr__weights=uniform, vect__countvectorizer__max_features=100, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 3/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=3, knnr__weights=uniform, vect__countvectorizer__max_features=100, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 1/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=3, knnr__weights=distance, vect__countvectorizer__max_features=None, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 2/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=3, knnr__weights=distance, vect__countvectorizer__max_features=None, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 3/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=3, knnr__weights=distance, vect__countvectorizer__max_features=None, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 1/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=3, knnr__weights=distance, vect__countvectorizer__max_features=50, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 2/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=3, knnr__weights=distance, vect__countvectorizer__max_features=50, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 3/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=3, knnr__weights=distance, vect__countvectorizer__max_features=50, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 1/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=3, knnr__weights=distance, vect__countvectorizer__max_features=100, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 2/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=3, knnr__weights=distance, vect__countvectorizer__max_features=100, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 3/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=3, knnr__weights=distance, vect__countvectorizer__max_features=100, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 1/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=4, knnr__weights=uniform, vect__countvectorizer__max_features=None, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 2/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=4, knnr__weights=uniform, vect__countvectorizer__max_features=None, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 3/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=4, knnr__weights=uniform, vect__countvectorizer__max_features=None, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 1/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=4, knnr__weights=uniform, vect__countvectorizer__max_features=50, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 2/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=4, knnr__weights=uniform, vect__countvectorizer__max_features=50, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 3/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=4, knnr__weights=uniform, vect__countvectorizer__max_features=50, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 1/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=4, knnr__weights=uniform, vect__countvectorizer__max_features=100, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 2/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=4, knnr__weights=uniform, vect__countvectorizer__max_features=100, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 3/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=4, knnr__weights=uniform, vect__countvectorizer__max_features=100, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 1/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=4, knnr__weights=distance, vect__countvectorizer__max_features=None, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 2/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=4, knnr__weights=distance, vect__countvectorizer__max_features=None, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 3/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=4, knnr__weights=distance, vect__countvectorizer__max_features=None, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 1/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=4, knnr__weights=distance, vect__countvectorizer__max_features=50, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 2/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=4, knnr__weights=distance, vect__countvectorizer__max_features=50, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 3/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=4, knnr__weights=distance, vect__countvectorizer__max_features=50, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 1/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=4, knnr__weights=distance, vect__countvectorizer__max_features=100, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 2/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=4, knnr__weights=distance, vect__countvectorizer__max_features=100, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 3/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=4, knnr__weights=distance, vect__countvectorizer__max_features=100, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 1/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=5, knnr__weights=uniform, vect__countvectorizer__max_features=None, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 2/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=5, knnr__weights=uniform, vect__countvectorizer__max_features=None, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 3/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=5, knnr__weights=uniform, vect__countvectorizer__max_features=None, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 1/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=5, knnr__weights=uniform, vect__countvectorizer__max_features=50, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 2/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=5, knnr__weights=uniform, vect__countvectorizer__max_features=50, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 3/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=5, knnr__weights=uniform, vect__countvectorizer__max_features=50, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 1/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=5, knnr__weights=uniform, vect__countvectorizer__max_features=100, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 2/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=5, knnr__weights=uniform, vect__countvectorizer__max_features=100, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 3/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=5, knnr__weights=uniform, vect__countvectorizer__max_features=100, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 1/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=5, knnr__weights=distance, vect__countvectorizer__max_features=None, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 2/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=5, knnr__weights=distance, vect__countvectorizer__max_features=None, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 3/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=5, knnr__weights=distance, vect__countvectorizer__max_features=None, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 1/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=5, knnr__weights=distance, vect__countvectorizer__max_features=50, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 2/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=5, knnr__weights=distance, vect__countvectorizer__max_features=50, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 3/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=5, knnr__weights=distance, vect__countvectorizer__max_features=50, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 1/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=5, knnr__weights=distance, vect__countvectorizer__max_features=100, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 2/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=5, knnr__weights=distance, vect__countvectorizer__max_features=100, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
[CV 3/3] END knnr__leaf_size=20, knnr__metric=euclidean, knnr__n_neighbors=5, knnr__weights=distance, vect__countvectorizer__max_features=100, vect__countvectorizer__ngram_range=(1, 1); total time=   0.0s
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
~\anaconda3\envs\deepai\lib\site-packages\sklearn\utils\__init__.py in _get_column_indices(X, key)
    373         try:
--> 374             all_columns = X.columns
    375         except AttributeError:

AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_9996/3908756850.py in <module>
      1 from time import time
      2 t0 = time()
----> 3 grid_search.fit(X_train[['clean_text','sklearn_cosine_similarity']].values, y_train.values)
      4 print("done in %0.3fs" % (time() - t0))

~\anaconda3\envs\deepai\lib\site-packages\sklearn\utils\validation.py in inner_f(*args, **kwargs)
     61             extra_args = len(args) - len(all_args)
     62             if extra_args <= 0:
---> 63                 return f(*args, **kwargs)
     64 
     65             # extra_args > 0

~\anaconda3\envs\deepai\lib\site-packages\sklearn\model_selection\_search.py in fit(self, X, y, groups, **fit_params)
    878             refit_start_time = time.time()
    879             if y is not None:
--> 880                 self.best_estimator_.fit(X, y, **fit_params)
    881             else:
    882                 self.best_estimator_.fit(X, **fit_params)

~\anaconda3\envs\deepai\lib\site-packages\sklearn\pipeline.py in fit(self, X, y, **fit_params)
    339         """
    340         fit_params_steps = self._check_fit_params(**fit_params)
--> 341         Xt = self._fit(X, y, **fit_params_steps)
    342         with _print_elapsed_time('Pipeline',
    343                                  self._log_message(len(self.steps) - 1)):

~\anaconda3\envs\deepai\lib\site-packages\sklearn\pipeline.py in _fit(self, X, y, **fit_params_steps)
    301                 cloned_transformer = clone(transformer)
    302             # Fit or load from cache the current transformer
--> 303             X, fitted_transformer = fit_transform_one_cached(
    304                 cloned_transformer, X, y, None,
    305                 message_clsname='Pipeline',

~\anaconda3\envs\deepai\lib\site-packages\joblib\memory.py in __call__(self, *args, **kwargs)
    347 
    348     def __call__(self, *args, **kwargs):
--> 349         return self.func(*args, **kwargs)
    350 
    351     def call_and_shelve(self, *args, **kwargs):

~\anaconda3\envs\deepai\lib\site-packages\sklearn\pipeline.py in _fit_transform_one(transformer, X, y, weight, message_clsname, message, **fit_params)
    752     with _print_elapsed_time(message_clsname, message):
    753         if hasattr(transformer, 'fit_transform'):
--> 754             res = transformer.fit_transform(X, y, **fit_params)
    755         else:
    756             res = transformer.fit(X, y, **fit_params).transform(X)

~\anaconda3\envs\deepai\lib\site-packages\sklearn\compose\_column_transformer.py in fit_transform(self, X, y)
    504         self._validate_transformers()
    505         self._validate_column_callables(X)
--> 506         self._validate_remainder(X)
    507 
    508         result = self._fit_transform(X, y, _fit_transform_one)

~\anaconda3\envs\deepai\lib\site-packages\sklearn\compose\_column_transformer.py in _validate_remainder(self, X)
    330         cols = []
    331         for columns in self._columns:
--> 332             cols.extend(_get_column_indices(X, columns))
    333 
    334         remaining_idx = sorted(set(range(self._n_features)) - set(cols))

~\anaconda3\envs\deepai\lib\site-packages\sklearn\utils\__init__.py in _get_column_indices(X, key)
    374             all_columns = X.columns
    375         except AttributeError:
--> 376             raise ValueError("Specifying the columns using strings is only "
    377                              "supported for pandas DataFrames")
    378         if isinstance(key, str):

ValueError: Specifying the columns using strings is only supported for pandas DataFrames

如果需要更多详细信息，请告诉我。

请不要删除此问题，因为我没有找到任何类似的问题。 如果有重复的问题，请参考链接。

Answer 1

对我来说，您的代码中的问题与您通过 make_column_transformer 定义的 ColumnTransformer 有关。它引用transformers 元组的参数columns。根据docs of ColumnTransformer（或等价于docs of make_column_transformer）：

columns : str、str 的类数组、int、int 的类数组、bool、slice 或 callable 的类数组

索引第二个轴上的数据。整数被解释为位置列，而 字符串可以按名称引用 DataFrame 列。 当转换器期望 X 是一维数组（向量）时，应使用标量字符串或整数，否则将向转换器传递一个二维数组。可调用对象传递输入数据 X 并可以返回上述任何内容。要按名称或 dtype 选择多个列，可以使用 make_column_selector..

确实，一方面，当在 numpy 数组上拟合 GridSearchCV 实例时（您在 X_train 和 y_train 上调用 .values()），您应该更好地根据位置（即通过整数）引用列上面的文档。换句话说，您违反了这样一个事实，即您可以通过名称引用列 - 即使用字符串或字符串列表 - 只有在直接拟合 DataFrame 时（strings can ONLY 按名称引用 DataFrame 列）。

另一方面，您应该注意CountVectorizer() 要求其输入是一维而不是二维的事实。在这种情况下，文档说明您应该通过字符串 ('clean_text') 引用 DataFrame 中的列，而不是像您在 categorical_features (['clean_text']) 中所做的那样的字符串数组（例如字符串列表） .

因此，如果您想继续在 ColumnTransformer 定义中按名称引用列，我建议将 categorical_features 定义为 categorical_features = 'clean_text' 并直接在 grid_search.fit() 中拟合数据框。

我还建议处理类似问题的以下帖子：

• Sklearn custom transformers with pipeline: all the input array dimensions for the concatenation axis must match exactly

Answer 2

您遇到的问题是您没有传递数据框来适应，而只是传递了一个 numpy 数组，请参阅 X_train[['clean_text','sklearn_cosine_similarity']].values 其中 values 只是提取底层的 numpy 数组。尝试删除.values