Abstract - Undersampling is a popular method in dealing with class-imbalance problems, which uses only a subset of the majority class and thus is very efficient. The main deficiency is that many majority class examples are ignored. We propose two algorithms to overcome this deficiency. EasyEnsemblesamples several subsets from the majority class, trains a learner using each of them, and combines the outputs of those learners. BalanceCascadetrains the learners sequentially, where in each step, the majority class examples that are correctly classified by the current trained learners are removed from further consideration. Experimental results show that both methods have higher Area Under the ROC Curve, F-measure, and G-mean values than many existing class-imbalance learning methods. Moreover, they have approximately the same training time as that of undersampling when the same number of weak classifiers is used, which is significantly faster than other methods.
Index Terms - Class-imbalance learning, data mining, ensemble learning, machine learning, undersampling
In many real-world problems, the data sets are typically imbalanced, i.e., some classes have much more instances than others. The level of imbalance (ratio of size of the majority class to minority class) can be as huge as . It is noteworthy that class imbalance is emerging as an important issue in designing classifiers.
Imbalance has a serious impact on the performance of classifiers. Learning algorithm that do not consider class imbalance tend to be overwhelmed by the majority class and ignore the minority class. For example, in a problem with imbalance level of , a learning algorithm that minimizes error rate could decide to classify all examples as the majority class in order to achieve a low error rate of
. However, all minority class examples will be wrong classified in this case. In problems where the imbalance level is huge, class imbalance must be carefully handled to build a good classifier.
Class imbalance is also closely related to cost-sensitive learning, another important issue in machine learning. Misclassifying a minority class instance is usually more serious than misclassifying a majority class one. For example, approving a fraudulent credit card application is more costly than declining a credible one. Breiman et al. pointed out that training set size, class priors, cost of errors in different classes, and placement of decision boundaries are all closely connected. In fact, many existing methods for dealing with class imbalance rely on connections among these four components. Sampling methods handle class imbalance by varying the minority and majority class sizes in the training set. Cost-sensitive learning deals with class imbalance by incurring different costs for the two classes and is considered as an important class of methods to handle class imbalance. More details about class-imbalance learning methods are presented in Section II.
In this paper, we examine only binary classification problems by ensembling classifiers built from multiple under sampled training sets. Undersampling is an efficient method for class-imbalance learning. This method uses a subset of the majority class to train the classifier. Since many majority class examples are ignored, the training set becomes more balanced and the training process becomes faster. However, the main drawback of undersampling is that potentially useful information contained in these ignored examples is neglected. The intuition of our proposed methods is then to wisely explore these ignored data while keeping the fast training speed of understanding.
We propose two way to use these data. One straightforward way is to sample several subsets independently from (the majority class), use these subsets to train classifiers separately, and combine the trained classifiers. Another method is to use trained classifiers to guide the sampling process for subsequent classifiers. After we have trained
classifiers, examples correctly classified by them will be removed from
. Experiments on 16 UCI data sets show that both methods have higher Area Under the receiver operating characteristics (ROC) Curve (AUC), F-measure, and G-mean values than many existing class-imbalance learning methods.
III. EasyEnsemble AND BalanceCascade
As was shown by Drummond and Holte, undersampling is an efficient strategy to deal with class imbalance. However, the drawback of undersampling is that it throws away many potentially useful data. In this section, we propose two strategies to explore the majority class examples ignored by undersampling: EasyEnsemble and BalanceCascade.
A. EasyEnsemble
Given the minority training set and the majority training set
, the undersampling method randomly samples a subset
from
, where
. Usually, we choose
and therefore have
for highly imbalanced problems.
EasyEnsemble is probably the most straightforward way to further exploit the majority class examples ignored by undersampling, i.e., examples in . In this method, we independently sample several subsets
from
. For each subset
(
), a classifier
is trained using
and all of
. All generated classifiers are combined for the final decision. AdaBoost is used to train the classifier
. The pseudocode for EasyEnsemble is shown in Algorithm 1.
Algorithm 1 The EasyEnsemble algorithm
1: {Input: A set of minority class examples , a set of majority class examples
,
, the number of subsets
to sample from
, and
, the number of iterations to train an AdaBoost ensemble
}
2:
3: repeat
4:
5: Randomly sample a subset from
,
.
6: Learn using
and
.
is an AdaBoost ensemble with
weak classifiers
and corresponding weights
. The ensemble's threshold is
, i.e.,
(1)
7: until
8: Output: An ensemble
(2)
The idea behind EasyEnsemble is simple. Similar to the Balanced Random Forests, EasyEnsemble generates balanced subproblems. The output of the
th subproblem is AdaBoost classifier
, an ensemble with
weak classifiers
. An alternative view of
is to treat it as a feature that is extracted by the ensemble learning method and can only take binary values.
, in this viewpoint, is simply a linear classifier built on these features. Features extracted from different subsets
thus contain information of different aspects of the original majority training set
. Finally, instead of counting votes from
, we collect all the features
and form an ensemble classifier from them.
The output of EasyEnsemble is a single ensemble, but it looks like an “ensemble of ensembles”. It is known that boosting mainly reduces bias, while bagging mainly reduces variance. Several works, combine different ensemble strategies to achieve stronger generalization. MultiBoosting, combines boosting with bagging/wagging by using boosted ensembles as base learners. Stochastic Gradient Boosting and Cocktail Ensemble also combine different ensemble strategies. It is evident that EasyEnsemble has benefited from the combination of boosting and a bagging-like strategy with balanced class distribution.
Both EasyEnsemble and Balanced Random Forests try to use balanced boostrap samples; however, the former uses the samples to generate boosted ensembles, while the latter uses the samples to train decision trees randomly. Costing also uses multiple samples of the original training set. Costing was initially proposed as a cost-sensitive learning method, while EasyEnsemble is proposed to deal with class imbalance directly. Moreover, the working style of EasyEnsemble is quite different from costing. For example, the costing method samples the examples with probability in proportion to their costs (rejection sampling). Since this is a probability-based sampling method, no positive example will definitely appear in all the samples (in fact, the probability of a positive example appearing in all the samples is small). While in EasyEnsemble, all the positive examples will definitely appear in all the samples. When the size of minority class is very small, it is important to utilize every minority class example.
B. BalanceCascade
EasyEnsemble is an unsupervised strategy to explore since it uses independent random sampling with replacement. Our second algorithm, BalanceCascade, explores
in a supervised manner. The idea is as follows. After
is trained, if an example
is correctly classified to be in the majority class by
, it is reasonable to conjecture that
is somewhat redundant in
, given that we already have
. Thus, we can remove some correctly classified majority class examples from
. As in EasyEnsemble, we use AdaBoost in this method. The pseudocode of BalanceCascade is described in Algorithm 2.
Algorithm 2 The BalanceCascade algorithm
1: {Input: A set of minority class examples , a set of majority class examples
,
, the number of subsets
to sample from
, and
, the number of iterations to train an AdaBoost ensemble
}
2: ,
,
is the false positive rate (the error rate of misclassifying a majority class example to the minority class) that
should achieve.
3: repeat
4:
5: Randomly sample a subset from
,
.
6: Learn using
and
.
is an AdaBoost ensemble with
weak classifiers
and corresponding weights
.
The ensemble's threshold is i.e.,
(3)
7: Adjust such that
's false positive rate is
.
8: Remove from all examples that are correctly classified by
.
9: until
10: Output: A single ensemble
(4)
This method is called BalanceCascade since it is somewhat similar to the cascade classifier. The majority training set is shrunk after every
is trained, and every node
is dealing with a balanced subproblem (
). However, the final classifier is different. A cascade classifier is the conjunction of all
, i.e.,
predicts positive if and only if all
(
) predicts positive. Viola and Jones used the cascade classifier mainly to achieve fast testing speed. While in BalanceCascade, sequential dependence between classifiers is mainly exploited for reducing the redundant information in the majority class. This sampling strategy leads to a restricted sample space for the following undersampling process to explore as much useful information as possible.
BalanceCascade is similar to EasyEnsemble in their structures. The main difference between them is the lines 7 and 8 of Algorithm 2. Line removes the true majority class examples from , and line 7 specifies how many majority class examples can be removed. At the beginning of the
th iteration,
has been shrunk
times, and therefore, its current size is
. Thus, after
is trained and
is shrunk again, the size of
is smaller than
. We can stop the training process at this time.