重采样数据 - 使用来自 imblearn 的 SMOTE 和 3D numpy 数组

Question

我想重新采样我的数据集。这包括具有 3 个类别标签的分类转换数据。每类样本量为：

• A 类计数：6945
• B 类计数：650
• C 类计数：9066
• 样本总数：16661

没有标签的数据形状是 (16661, 1000, 256)。这意味着 (1000,256) 的 16661 个样本。我想要的是将数据上采样到多数类的样本数，即A类->（6945）

但是，调用时：

from imblearn.over_sampling import SMOTE
print(categorical_vector.shape)
sm = SMOTE(random_state=2)
X_train_res, y_labels_res = sm.fit_sample(categorical_vector, labels.ravel())

它一直说 ValueError: Found array with dim 3. Estimator expected

我怎样才能以估算器可以拟合并且也有意义的方式展平数据？此外，在获得 X_train_res 后如何展开（具有 3D 维度）？

Answer 1

我正在考虑一个虚拟的3d 数组并自己假设一个2d 数组大小，

arr = np.random.rand(160, 10, 25)
orig_shape = arr.shape
print(orig_shape)

输出：(160, 10, 25)

arr = np.reshape(arr, (arr.shape[0], arr.shape[1]))
print(arr.shape)

输出：(4000, 10)

arr = np.reshape(arr, orig_shape))
print(arr.shape)

输出：(160, 10, 25)

Answer 2

from imblearn.over_sampling 
import RandomOverSampler 
import numpy as np 
oversample = RandomOverSampler(sampling_strategy='minority')

X 可以是时间步长的 3D 数据，例如 X[sample,time,feature]，而 y 可以是每个样本的二进制值。例如：(1,1),(2,1),(3,1) -> 1

X = np.array([[[1,1],[2,1],[3,1]],
             [[2,1],[3,1],[4,1]],
             [[5,1],[6,1],[7,1]],
             [[8,1],[9,1],[10,1]],
             [[11,1],[12,1],[13,1]]
             ])

y = np.array([1,0,1,1,0])

无法使用 3D X 值训练 OVERSAMPLER，因为如果使用 2D，您将获得 2D 数据。

Xo,yo = oversample.fit_resample(X[:,:,0], y)
Xo:
[[ 1  2  3]
 [ 2  3  4]
 [ 5  6  7]
 [ 8  9 10]
 [11 12 13]
 [ 2  3  4]]

yo:
[1 0 1 1 0 0]

但如果你使用 2D 数据 (sample,time,0) 来拟合模型，它会返回索引，创建 3D 过采样数据就足够了

oversample.fit_resample(X[:,:,0], y)
Xo = X[oversample.sample_indices_]
yo = y[oversample.sample_indices_]

Xo:
[[[ 1  1][ 2  1][ 3  1]]
 [[ 2  1][ 3  1][ 4  1]]
 [[ 5  1][ 6  1][ 7  1]]
 [[ 8  1][ 9  1][10  1]]
 [[11  1][12  1][13  1]]
 [[ 2  1][ 3  1][ 4  1]]]
yo:
[1 0 1 1 0 0]

Answer 3

我将为 2-dim 数组创建每个点，然后将其重塑为 3 dim 数组。我已经提供了我的脚本。如果有任何混淆，请发表评论；请回复。

x_train, y_train = zip(*train_dataset)
x_test, y_test = zip(*test_dataset)

dim_1 = np.array(x_train).shape[0]
dim_2 = np.array(x_train).shape[1]
dim_3 = np.array(x_train).shape[2]

new_dim = dim_1 * dim_2

new_x_train = np.array(x_train).reshape(new_dim, dim_3)


new_y_train = []
for i in range(len(y_train)):
    # print(y_train[i])
    new_y_train.extend([y_train[i]]*dim_2)

new_y_train = np.array(new_y_train)

# transform the dataset
oversample = SMOTE()
X_Train, Y_Train = oversample.fit_sample(new_x_train, new_y_train)
# summarize the new class distribution
counter = Counter(Y_Train)
print('The number of samples in TRAIN: ', counter)



x_train_SMOTE = X_Train.reshape(int(X_Train.shape[0]/dim_2), dim_2, dim_3)

y_train_SMOTE = []
for i in range(int(X_Train.shape[0]/dim_2)):
    # print(i)
    value_list = list(Y_Train.reshape(int(X_Train.shape[0]/dim_2), dim_2)[i])
    # print(list(set(value_list)))
    y_train_SMOTE.extend(list(set(value_list)))
    ## Check: if there is any different value in a list 
    if len(set(value_list)) != 1:
        print('\n\n********* STOP: THERE IS SOMETHING WRONG IN TRAIN ******\n\n')
    


dim_1 = np.array(x_test).shape[0]
dim_2 = np.array(x_test).shape[1]
dim_3 = np.array(x_test).shape[2]

new_dim = dim_1 * dim_2

new_x_test = np.array(x_test).reshape(new_dim, dim_3)


new_y_test = []
for i in range(len(y_test)):
    # print(y_train[i])
    new_y_test.extend([y_test[i]]*dim_2)

new_y_test = np.array(new_y_test)

# transform the dataset
oversample = SMOTE()
X_Test, Y_Test = oversample.fit_sample(new_x_test, new_y_test)
# summarize the new class distribution
counter = Counter(Y_Test)
print('The number of samples in TEST: ', counter)



x_test_SMOTE = X_Test.reshape(int(X_Test.shape[0]/dim_2), dim_2, dim_3)

y_test_SMOTE = []
for i in range(int(X_Test.shape[0]/dim_2)):
    # print(i)
    value_list = list(Y_Test.reshape(int(X_Test.shape[0]/dim_2), dim_2)[i])
    # print(list(set(value_list)))
    y_test_SMOTE.extend(list(set(value_list)))
    ## Check: if there is any different value in a list 
    if len(set(value_list)) != 1:
        print('\n\n********* STOP: THERE IS SOMETHING WRONG IN TEST ******\n\n')