【问题标题】：what mean error :AttributeError: lower not found in classification project?什么意思错误：AttributeError：在分类项目中找不到下限？
【发布时间】：2021-10-27 22:29:09
【问题描述】：

我有一个使用 jupyter notebook 编写的 python 并处理不平衡数据集中的分类主题项目，为此我使用了 SMOTE 但是当我尝试拆分数据集并创建一个使用机器学习模型的管道系统崩溃并显示以下错误：

--------------------------------------------------------------------------- AttributeError                            Traceback (most recent call last) <ipython-input-17-7ae8518f1892> in <module>
     15                 ('clf',MultinomialNB()), # model classifier
     16             ])
---> 17 nb.fit(x_train,y_train)

f:\AIenv\lib\site-packages\sklearn\pipeline.py in fit(self, X, y,
**fit_params)
    328         """
    329         fit_params_steps = self._check_fit_params(**fit_params)
--> 330         Xt = self._fit(X, y, **fit_params_steps)
    331         with _print_elapsed_time('Pipeline',
    332                                  self._log_message(len(self.steps) - 1)):

f:\AIenv\lib\site-packages\sklearn\pipeline.py in _fit(self, X, y,
**fit_params_steps)
    294                 message_clsname='Pipeline',
    295                 message=self._log_message(step_idx),
--> 296                 **fit_params_steps[name])
    297             # Replace the transformer of the step with the fitted
    298             # transformer. This is necessary when loading the transformer

f:\AIenv\lib\site-packages\joblib\memory.py in __call__(self, *args,
**kwargs)
    353 
    354     def __call__(self, *args, **kwargs):
--> 355         return self.func(*args, **kwargs)
    356 
    357     def call_and_shelve(self, *args, **kwargs):

f:\AIenv\lib\site-packages\sklearn\pipeline.py in
_fit_transform_one(transformer, X, y, weight, message_clsname, message, **fit_params)
    738     with _print_elapsed_time(message_clsname, message):
    739         if hasattr(transformer, 'fit_transform'):
--> 740             res = transformer.fit_transform(X, y, **fit_params)
    741         else:
    742             res = transformer.fit(X, y, **fit_params).transform(X)

f:\AIenv\lib\site-packages\sklearn\feature_extraction\text.py in fit_transform(self, raw_documents, y)    1197     1198         vocabulary, X = self._count_vocab(raw_documents,
-> 1199                                           self.fixed_vocabulary_)    1200     1201         if self.binary:

f:\AIenv\lib\site-packages\sklearn\feature_extraction\text.py in
_count_vocab(self, raw_documents, fixed_vocab)    1108         for doc in raw_documents:    1109             feature_counter = {}
-> 1110             for feature in analyze(doc):    1111                 try:    1112                     feature_idx = vocabulary[feature]

f:\AIenv\lib\site-packages\sklearn\feature_extraction\text.py in
_analyze(doc, analyzer, tokenizer, ngrams, preprocessor, decoder, stop_words)
    102     else:
    103         if preprocessor is not None:
--> 104             doc = preprocessor(doc)
    105         if tokenizer is not None:
    106             doc = tokenizer(doc)

f:\AIenv\lib\site-packages\sklearn\feature_extraction\text.py in
_preprocess(doc, accent_function, lower)
     67     """
     68     if lower:
---> 69         doc = doc.lower()
     70     if accent_function is not None:
     71         doc = accent_function(doc)

f:\AIenv\lib\site-packages\scipy\sparse\base.py in __getattr__(self, attr)
    685             return self.getnnz()
    686         else:
--> 687             raise AttributeError(attr + " not found")
    688 
    689     def transpose(self, axes=None, copy=False):

AttributeError: lower not found

代码：

import pandas as pd
import numpy as np
from imblearn.over_sampling import SMOTE# for inbalance dataset
from sklearn.feature_extraction.text import CountVectorizer,TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB

df = pd.read_csv("data/emotion_dataset_raw.csv")
df["clean_text"] = df["Text"].apply(clean_text)
vectorizer =TfidfVectorizer(ngram_range=(1,2))
vect_df =vectorizer.fit_transform(df["clean_text"])
oversample = SMOTE(random_state = 42)
x_smote,y_smote = oversample.fit_resample(vect_df, df["Emotion"])
print("shape x before SMOTE: {}".format(vect_df.shape))
print("shape x after SMOTE: {}".format(x_smote.shape))
print("balance of targets feild %")
y_smote.value_counts(normalize = True)*100

# the result of the code above :
#shape x before SMOTE: (34792, 209330)
#shape x after SMOTE: (88360, 209330)

x_train,x_test,y_train,y_test = train_test_split(x_smote,y_smote,test_size = 0.2,random_state =42)

#Naiive Bayes Classifier
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfTransformer
  
nb = Pipeline([
                ('vect',CountVectorizer(ngram_range=(1,2))),
                ('tfidf',TfidfTransformer()),
                ('clf',MultinomialNB()), # model classifier 
            ])
nb.fit(x_train,y_train)

我的代码中的错误在哪里以及它的含义？？？

【问题讨论】：

标签： python scikit-learn classification tfidfvectorizer smote

【解决方案1】：

我相信TfidfTransformer 足以生成文本嵌入。您可以删除CountVectorizer 并再次运行代码。我应该工作！

pipe = Pipeline(
[
    ('tfidf', TfidfVectorizer()),
    ('sampler', RandomOverSampler(sampling_strategy='not majority', random_state=7)),
    ('model', XGBClassifier())
]
)

pipe.fit(data['features'], data['labels'])

【讨论】：