混合数据类型的转换器

Question

我遇到了麻烦一次将不同的转换器应用于具有不同类型（文本与数字）的列，并将这些转换器连接成一个以供以后使用。

我尝试按照Column Transformer with Mixed Types 文档中的步骤进行操作，该文档解释了如何对分类数据和数值数据混合执行此操作，但它似乎不适用于文本数据。

TL;DR

您如何创建一个可存储的转换器，它遵循文本和数字数据的不同管道？

数据下载和准备

# imports
import numpy as np

from sklearn.compose import ColumnTransformer
from sklearn.datasets import fetch_openml
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import FeatureUnion, Pipeline
from sklearn.preprocessing import StandardScaler

np.random.seed(0)

# download Titanic data
X, y = fetch_openml("titanic", version=1, as_frame=True, return_X_y=True)

# data preparation
numeric_features = ['age', 'fare']
text_features = ['name', 'cabin', 'home.dest']
X.fillna({text_col: '' for text_col in text_features}, inplace=True)

# train test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

转换数值特征：ok

按照上面链接中的步骤，可以为数字特征创建一个转换器，如下所示：

# handling missing data and normalization
numeric_transformer = Pipeline(steps=[('imputer', SimpleImputer(strategy='median')),
                                      ('scaler', StandardScaler())])

num_preprocessor = ColumnTransformer(transformers=[('num', numeric_transformer, numeric_features)])

# this works
num_preprocessor.fit(X_train)
train_feature_set = num_preprocessor.transform(X_train)
test_feature_set = num_preprocessor.transform(X_test)

# verify shape = (number of data points, number of numerical features (2) )
train_feature_set.shape  # (1047, 2)
test_feature_set.shape  # (262, 2)

转换文本特征：ok

为了处理文本特征，我使用 TF-IDF 对每个文本列进行矢量化（而不是连接所有文本列，并且只应用一次 TF-IDF）：

# Tfidf of max 30 features
text_transformer = TfidfVectorizer(use_idf=True,
                                   max_features=30)
# apply separately to each column
text_transformer_list = [(x + '_vectorizer', text_transformer, x) for x in text_features]
text_preprocessor = ColumnTransformer(transformers=text_transformer_list)

# this works
text_preprocessor.fit(X_train)
train_feature_set = text_preprocessor.transform(X_train)
test_feature_set = text_preprocessor.transform(X_test)

# verify shape = (number of data points, number of text features (3) times max_features(30) )
train_feature_set.shape  # (1047, 90)
test_feature_set.shape  # (262, 90)

你如何同时做到这两点？

我尝试了各种策略将上述两个程序保存在一个转换器中，但由于不同的错误，它们都失败了。

尝试 1：遵循记录在案的策略

一旦文本数据替换分类数据，遵循文档 (Column Transformer with Mixed Types) 将不起作用：

# documented strategy
sum_preprocessor = ColumnTransformer(transformers=[('num', numeric_transformer, numeric_features),
                                                   ('text', text_transformer, text_features)])
# fails
sum_preprocessor.fit(X_train)

返回以下错误信息：

ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 1047 and the array at index 1 has size 3

尝试 2：FeatureUnion 在转换器列表中

# create a list of numerical transformer, like those for text
numerical_transformer_list = [(x + '_scaler', numeric_transformer, x) for x in numeric_features]

# fails
column_trans = FeatureUnion([text_transformer_list, numerical_transformer_list])

返回以下错误信息：

TypeError: All estimators should implement fit and transform. '('cabin_vectorizer', TfidfVectorizer(max_features=30), 'cabin')' (type <class 'tuple'>) doesn't

尝试 3：ColumnTransformer 在转换器列表中

# create a list of all transformers, text and numerical
sum_transformer_list = text_transformer_list + numerical_transformer_list

# works
sum_preprocessor = ColumnTransformer(transformers=sum_transformer_list)

# fails
sum_preprocessor.fit(X_train)

返回以下错误信息：

ValueError: Expected 2D array, got 1D array instead:
array=[54. nan nan ... 20. nan nan].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

我的问题

如何创建一个可以fit 和transform 数据混合文本和数字类型的对象？

Answer 1

简答：

all_transformers = text_transformer_list + [('num', numeric_transformer, numeric_features)]

all_preprocessor = ColumnTransformer(transformers=all_transformers)

all_preprocessor.fit(X_train)
train_all = all_preprocessor.transform(X_train)
test_all = all_preprocessor.transform(X_test)

print(train_all.shape, test_all.shape)
# prints (1047, 92) (262, 92)

这里的困难在于（大多数？）文本转换器需要一维输入，但（大多数？）数字转换器需要二维输入。 ColumnTransformer 通过允许您指定单个列或列列表来处理该问题：在第一种情况下，一维数组被传递给转换器，而在第二种情况下，一个二维数组被传递。

那么，解释一下三个尝试中的错误：

尝试 1：TF-IDF 正在接收二维数组，并将 列 视为文档而不是单个条目，因此仅产生三个输出。当它尝试将其连接到 1047 行数字输出时，它会失败。

尝试 2：FeatureUnion 与 ColumnTransformer 的输入格式不同：在这种情况下，您不应使用三元组 (name, transformer, columns)。无论如何，FeatureUnion 不适合你在这里做的事情。

尝试 3：这次您尝试将 1d 数据发送到数值转换器，但这些数据需要 2d 数据。