由于 NaN，LabelBinarizer 的行为不一致

Question

我正在尝试将带有 DataFrame 文本的列转换为一个热编码矩阵。这在一段时间内工作得很好，但由于我不知道的原因而停止工作。消息说：“TypeError：'>' 在 'str' 和 'float' 的实例之间不支持”对我来说这似乎是无稽之谈，因为我只使用 tekst 数据。当我用一个小数据集重复实验时，LabelBinarizer 工作得很好并产生了所需的输出。

我注意到 X_train 数据帧的大小为 4.6 GB。我的机器只有 8 GB。我应该注意一些内存限制吗？所有数字都比较小，我应该转换成 int32 和 float32 吗？

我能够重现以下错误。但我不确定这是否提供了足够的信息。

from sklearn.preprocessing import LabelBinarizer

lb=LabelBinarizer()

s=['a','b','c','b','a']

df=pd.DataFrame (s)

df = pd.Series (s)

dd = X_train['state']

type(dd)
Out[9]: pandas.core.series.Series

type(df)
Out[10]: pandas.core.series.Series

lb.fit(dd)
Traceback (most recent call last):

  File "<ipython-input-11-5ec245111e31>", line 1, in <module>
    lb.fit(dd)

  File "C:\packages\Anaconda3\lib\site-packages\sklearn\preprocessing\label.py", line 296, in fit
    self.y_type_ = type_of_target(y)

  File "C:\packages\Anaconda3\lib\site-packages\sklearn\utils\multiclass.py", line 275, in type_of_target
    if (len(np.unique(y)) > 2) or (y.ndim >= 2 and len(y[0]) > 1):

  File "C:\packages\Anaconda3\lib\site-packages\numpy\lib\arraysetops.py", line 214, in unique
    ar.sort()

TypeError: '>' not supported between instances of 'str' and 'float'


lb.fit(df)
Out[12]: LabelBinarizer(neg_label=0, pos_label=1, sparse_output=False)

df.value_counts()
Out[13]: 
a    2
b    2
c    1
dtype: int64

dd.value_counts()
Out[14]: 
MI    228601
CA      5020
TX      2420
FL      2237
IL      1310
SC      1304
OH       967
NY       673
MN       632
GA       535
NV       484
UT       477
PA       466
NJ       395
VA       385
NC       353
MD       349
AZ       329
ME       261
OK       248
AL       215
TN       207
WA       192
MA       182
IA       159
WI       159
OR       153
MO       151
CO       147
KY       146
IN       106
AR        82
LA        81
AK        79
UK        77
NB        77
MS        64
CT        60
DC        58
ON        51
DE        50
KS        37
RI        35
SD        33
ID        33
MT        28
NM        21
BC        17
WY        12
HI        10
NH         9
VT         7
VI         6
WV         6
PR         5
QC         5
QL         3
ND         2
BL         2
Name: state, dtype: int64

len(df)
Out[15]: 5

len(dd)
Out[16]: 250306

Answer 1

也许它的输入数据可能包含缺失值。

from sklearn.preprocessing import LabelBinarizer
import numpy as np
import pandas as pd

lb = LabelBinarizer()

s = ['a','b','c','b','a', np.nan]
df = pd.DataFrame(s, columns=["state"])

df_binarized = lb.fit_transform(df['state'])
df_binarized

Traceback (most recent call last):
  File "/home/kuroyanagi/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-45-f16e01b4e1be>", line 4, in <module>
    df_binarized = lb.fit_transform(df['state'])
  File "/home/kuroyanagi/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages/sklearn/base.py", line 494, in fit_transform
    return self.fit(X, **fit_params).transform(X)
  File "/home/kuroyanagi/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages/sklearn/preprocessing/label.py", line 296, in fit
    self.y_type_ = type_of_target(y)
  File "/home/kuroyanagi/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages/sklearn/utils/multiclass.py", line 275, in type_of_target
    if (len(np.unique(y)) > 2) or (y.ndim >= 2 and len(y[0]) > 1):
  File "/home/kuroyanagi/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages/numpy/lib/arraysetops.py", line 210, in unique
    return _unique1d(ar, return_index, return_inverse, return_counts)
  File "/home/kuroyanagi/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages/numpy/lib/arraysetops.py", line 277, in _unique1d
ar.sort()
TypeError: '<' not supported between instances of 'float' and 'str'

如果没有缺失值，则工作如下。

from sklearn.preprocessing import LabelBinarizer
import numpy as np
import pandas as pd

s = ['a','b','c','b','a']
df = pd.DataFrame(s, columns=["state"])

df_binarized = lb.fit_transform(df['state'])
df_binarized

Out[46]:
array([[1, 0, 0],
       [0, 1, 0],
       [0, 0, 1],
       [0, 1, 0],
       [1, 0, 0]])