在 Python 中使用副本进行分层 K 折拆分答案

【问题标题】：Stratified K-fold splitting with replicas in Python在 Python 中使用副本进行分层 K 折拆分
【发布时间】：2019-12-25 02:22:23
【问题描述】：

我有一个数据集，其中包含多个测试系统，每个系统都重复了测试。

所有系统的副本总数是恒定的（平衡的），但每个副本可能包含不同数量的总观测值。

我想使用交叉验证将数据拆分为训练集和测试集，这样：

测试和训练集中的每个系统
训练集将包含每个系统的除 1 个副本之外的所有副本，而测试集包含每个系统的剩余副本
测试集中每个系统的观察百分比与训练集中每个系统的观察百分比相匹配

我希望使用 sci-kit learn 的 StratifiedKFold 功能，但它似乎无法满足我的需求。

例如，使用此示例标记数据：

labels=np.concatenate([['Sys1']*35,['Sys2']*33,['Sys3']*36])
reps=np.concatenate([
    np.concatenate([
        ['Rep_0']*10,['Rep_1']*10,['Rep_2']*5,['Rep_3']*10]),
    np.concatenate([
        ['Rep_0']*8,['Rep_1']*10,['Rep_2']*10,['Rep_3']*5]),
    np.concatenate([
        ['Rep_0']*10,['Rep_1']*7,['Rep_2']*9,['Rep_3']*10])
])
frames=np.concatenate([
    np.concatenate([
        np.arange(10),np.arange(10),np.arange(5),np.arange(10)]),
    np.concatenate([
        np.arange(8),np.arange(10),np.arange(10),np.arange(5)]),
    np.concatenate([
        np.arange(10),np.arange(7),np.arange(9),np.arange(10)])
])
sampleKeys=np.array(map(lambda x,y: '.'.join([x,y]),
               labels,
               reps))

我尝试拆分标签：

cvSplitter=skl.model_selection.StratifiedKFold(n_splits=4)
iSplit=0
for train_indices, test_indices in cvSplitter.split(labels,labels):
    print '--- split %g ---'%iSplit
    print 'TRAIN:'
    for sample in np.array([np.unique(sampleKeys[train_indices],return_counts=True)[0],
           np.unique(sampleKeys[train_indices],return_counts=True)[1]]).T:
        print sample

    print 'TEST:'
    for sample in np.array([np.unique(sampleKeys[test_indices],return_counts=True)[0],
           np.unique(sampleKeys[test_indices],return_counts=True)[1]]).T:
        print sample

    iSplit=iSplit+1

但是，虽然结果包含来自每个系统的训练与测试中相同百分比的观察结果，但训练集包含某些或所有系统的所有副本，而我希望训练集包含除一个副本之外的所有副本，而测试集包含缺少副本。

--- split 0 ---
TRAIN:
['Sys1.Rep_0' '1']
['Sys1.Rep_1' '10']
['Sys1.Rep_2' '5']
['Sys1.Rep_3' '10']
['Sys2.Rep_1' '9']
['Sys2.Rep_2' '10']
['Sys2.Rep_3' '5']
['Sys3.Rep_0' '1']
['Sys3.Rep_1' '7']
['Sys3.Rep_2' '9']
['Sys3.Rep_3' '10']
TEST:
['Sys1.Rep_0' '9']
['Sys2.Rep_0' '8']
['Sys2.Rep_1' '1']
['Sys3.Rep_0' '9']
--- split 1 ---
TRAIN:
['Sys1.Rep_0' '9']
['Sys1.Rep_1' '2']
['Sys1.Rep_2' '5']
['Sys1.Rep_3' '10']
['Sys2.Rep_0' '8']
['Sys2.Rep_1' '2']
['Sys2.Rep_2' '10']
['Sys2.Rep_3' '5']
['Sys3.Rep_0' '9']
['Sys3.Rep_2' '8']
['Sys3.Rep_3' '10']
TEST:
['Sys1.Rep_0' '1']
['Sys1.Rep_1' '8']
['Sys2.Rep_1' '8']
['Sys3.Rep_0' '1']
['Sys3.Rep_1' '7']
['Sys3.Rep_2' '1']
--- split 2 ---
TRAIN:
['Sys1.Rep_0' '10']
['Sys1.Rep_1' '8']
['Sys1.Rep_3' '8']
['Sys2.Rep_0' '8']
['Sys2.Rep_1' '9']
['Sys2.Rep_2' '3']
['Sys2.Rep_3' '5']
['Sys3.Rep_0' '10']
['Sys3.Rep_1' '7']
['Sys3.Rep_2' '1']
['Sys3.Rep_3' '9']
TEST:
['Sys1.Rep_1' '2']
['Sys1.Rep_2' '5']
['Sys1.Rep_3' '2']
['Sys2.Rep_1' '1']
['Sys2.Rep_2' '7']
['Sys3.Rep_2' '8']
['Sys3.Rep_3' '1']
--- split 3 ---
TRAIN:
['Sys1.Rep_0' '10']
['Sys1.Rep_1' '10']
['Sys1.Rep_2' '5']
['Sys1.Rep_3' '2']
['Sys2.Rep_0' '8']
['Sys2.Rep_1' '10']
['Sys2.Rep_2' '7']
['Sys3.Rep_0' '10']
['Sys3.Rep_1' '7']
['Sys3.Rep_2' '9']
['Sys3.Rep_3' '1']
TEST:
['Sys1.Rep_3' '8']
['Sys2.Rep_2' '3']
['Sys2.Rep_3' '5']
['Sys3.Rep_3' '9']
1

如果我改为拆分“reps”，我最终会发现一些系统被排除在测试和/或训练数据之外。

【问题讨论】：

标签： python cross-validation

【解决方案1】：

到目前为止，我已经设法生成一个函数将拆分在代表上，确保培训数据包含所有除了每个系统的一个代表，并且测试数据包含训练集中未包含的REB（函数到目前为止均吐了所需的指数）......我有点困惑，以便如何确保数据来自每个系统的样本数量。

使用原始测试示例

labels=np.concatenate([['Sys1']*35,['Sys2']*33,['Sys3']*36])
reps=np.concatenate([
    np.concatenate([
        ['Rep_0']*10,['Rep_1']*10,['Rep_2']*5,['Rep_3']*10]),
    np.concatenate([
        ['Rep_0']*8,['Rep_1']*10,['Rep_2']*10,['Rep_3']*5]),
    np.concatenate([
        ['Rep_0']*10,['Rep_1']*7,['Rep_2']*9,['Rep_3']*10])
])
frames=np.concatenate([
    np.concatenate([
        np.arange(10),np.arange(10),np.arange(5),np.arange(10)]),
    np.concatenate([
        np.arange(8),np.arange(10),np.arange(10),np.arange(5)]),
    np.concatenate([
        np.arange(10),np.arange(7),np.arange(9),np.arange(10)])
])
sampleKeys=np.array(map(lambda x,y: '.'.join([x,y]),
               labels,
               reps))

和分裂功能：

def stratifiedReplicaSplit(sysLabels,repLabels,nSplits):
    sampleLabels=map(lambda x,y: '.'.join([str(x),str(y)]),
                     sysLabels,repLabels)
    nSysLabels=len(np.unique(sysLabels))
    nRepLabels=len(np.unique(repLabels))

    outInds=[]

    sysTypes=np.sort(np.unique(sysLabels))
    repTypes=np.sort(np.unique(repLabels))
    #generate a list of all possible combinations for having a single
    #rep from each system.
    #we will shamelessly hack np.meshgrid to achieve this.
    #we then sample nSplits entries from this list (without replacement)
    tempRepSets=[np.arange(nRepLabels)]*nSysLabels
    comboSetGrids=np.array([np.array(tempGrid.flat) for tempGrid in np.meshgrid(*tempRepSets)])
    #we can now generate a unique split by sampling from the total set of all
    #possible combo sets (without replacement)
    comboInds=np.random.choice(comboSetGrids.shape[1],nSplits,replace=False)
    combosArray=comboSetGrids[:,comboInds].T
    print 'sysTypes',
    print sysTypes
    print 'repTypes',
    print repTypes
    print combosArray
    #print 'sample labels'
    #print sampleLabels
    for iSplit in np.arange(nSplits):
        comboSet=combosArray[iSplit].flatten()
        print comboSet
        outInds.append(
            [np.argwhere(
                np.product(
                    [map(lambda x: x!='%s.%s'%(
                                sysTypes[iEntry],
                                repTypes[entry]),
                           sampleLabels) \
                             for iEntry,entry in enumerate(comboSet)],
                    axis=0)).flatten(),
            np.argwhere(
                np.sum(
                    [map(lambda x: x=='%s.%s'%(
                                sysTypes[iEntry],
                                repTypes[entry]),
                           sampleLabels) \
                             for iEntry,entry in enumerate(comboSet)],
                    axis=0)>0).flatten()])
    return outInds

然后运行：

cvSplitter=stratifiedReplicaSplit
iSplit=0
for train_indices, test_indices in cvSplitter(labels,reps,4):
    print '--- split %g ---'%iSplit
    print 'TRAIN:'
    for sample in np.array([np.unique(sampleKeys[train_indices],return_counts=True)[0],
           np.unique(sampleKeys[train_indices],return_counts=True)[1]]).T:
        print sample

    print 'TEST:'
    for sample in np.array([np.unique(sampleKeys[test_indices],return_counts=True)[0],
           np.unique(sampleKeys[test_indices],return_counts=True)[1]]).T:
        print sample

    iSplit=iSplit+1

并获得 systypes ['sys1''sys2'''sys3']

repTypes ['Rep_0' 'Rep_1' 'Rep_2' 'Rep_3']
[[2 0 1]
 [0 1 2]
 [3 0 1]
 [0 2 3]]
[2 0 1]
[0 1 2]
[3 0 1]
[0 2 3]
--- split 0 ---
TRAIN:
['Sys1.Rep_0' '10']
['Sys1.Rep_1' '10']
['Sys1.Rep_3' '10']
['Sys2.Rep_1' '10']
['Sys2.Rep_2' '10']
['Sys2.Rep_3' '5']
['Sys3.Rep_0' '10']
['Sys3.Rep_2' '9']
['Sys3.Rep_3' '10']
TEST:
['Sys1.Rep_2' '5']
['Sys2.Rep_0' '8']
['Sys3.Rep_1' '7']
--- split 1 ---
TRAIN:
['Sys1.Rep_1' '10']
['Sys1.Rep_2' '5']
['Sys1.Rep_3' '10']
['Sys2.Rep_0' '8']
['Sys2.Rep_2' '10']
['Sys2.Rep_3' '5']
['Sys3.Rep_0' '10']
['Sys3.Rep_1' '7']
['Sys3.Rep_3' '10']
TEST:
['Sys1.Rep_0' '10']
['Sys2.Rep_1' '10']
['Sys3.Rep_2' '9']
--- split 2 ---
TRAIN:
['Sys1.Rep_0' '10']
['Sys1.Rep_1' '10']
['Sys1.Rep_2' '5']
['Sys2.Rep_1' '10']
['Sys2.Rep_2' '10']
['Sys2.Rep_3' '5']
['Sys3.Rep_0' '10']
['Sys3.Rep_2' '9']
['Sys3.Rep_3' '10']
TEST:
['Sys1.Rep_3' '10']
['Sys2.Rep_0' '8']
['Sys3.Rep_1' '7']
--- split 3 ---
TRAIN:
['Sys1.Rep_1' '10']
['Sys1.Rep_2' '5']
['Sys1.Rep_3' '10']
['Sys2.Rep_0' '8']
['Sys2.Rep_1' '10']
['Sys2.Rep_3' '5']
['Sys3.Rep_0' '10']
['Sys3.Rep_1' '7']
['Sys3.Rep_2' '9']
TEST:
['Sys1.Rep_0' '10']
['Sys2.Rep_2' '10']
['Sys3.Rep_3' '10']

......所以，实质上，我已经再现了'regetonegroupout'分裂的变体（更具体地，留下一个目标组对的一个组合）。这没关系，但是测试数据中每个系统的相对样本量与训练数据中该系统的相对样本量不成比例......我需要以某种方式分层/平衡。例如。一个系统在训练集中的样本百分比应该与它在测试集中的样本百分比相匹配......

【讨论】：