使用逻辑将 Pandas 数据采样成不同的比率

Question

我有一个如下所示的数据框，我想使用 order_id.Each 对应该将每个客户数据分成三个桶的数据进行采样，分别为 train(70%)、validation(15%) 和 test(15%)客户应该出现在所有三个存储桶中。每个客户的 order_id 计数和项目可能不同

数据框：

Customer  Orderid   item_name
   A        1        orange
   A        1        apple
   A        1        banana
   A        2        apple
   A        2        carrot
   A        3        orange
   A        4        grape
   A        4        watermelon
   A        4        banana
   B        1        pineapple
   B        2        banana
   B        3        papaya
   B        3        Lime

采样后的所有三个数据集（train、validation 和 test）应该包含相同数量的客户，并且来自验证和 test 的项目应该是 train 的子集。

预期结果：

  train: should contain all customers and all item_names (70% of complete data)
train:
     customer  item
         A     orange
         A     apple
         A     banana
         A     carrot
         A     grape
         A     watermelon
         B     pinepple 
         B     banana
         B     papaya
         B     Lime
  validation : should contain all customers and item_names can be subset of train(15% of complete data)
        customer  item
         A     orange
         A     apple
         A     banana
         B     pinepple 
         B     banana
         B     papaya
         B     Lime
  test : should contain all customers and item_names can be subset of train(15% of complete data)
       Customer  item
         A     carrot
         A     grape
         A     watermelon
         B     papaya
         B     Lime

Answer 1

正如@Parth 在 cmets 中提到的，首先您需要有一个符合这种分层拆分条件的数据集。然后，您可以使用“Customer”和“item_name”的组合创建一个新列，以提供“train_test_split”方法的“stratify”参数，这是 sklearn 的一部分。

下面，你可以找到一个例子。

import pandas as pd
from sklearn.model_selection import train_test_split

#Create sample data
data = {
    "Customer":["A", "A", "A", "A","A","A","A","A","A", "B", "B", "B","B", "B", "B", "B","B","B"],
    "Orderid":[1, 1, 1, 2, 2, 2, 2, 3, 2, 1, 2, 1, 1, 1, 1, 2, 2, 2],
    "item_name":[
        "orange",
        "apple",
        "orange",
        "apple",
        "orange",
        "apple",
        "orange",
        "apple",
        "orange",
        "apple",
        "orange",
        "apple",
        "orange",
        "apple",
        "orange",
        "apple",
        "orange",
        "apple"
       ]
}
# Convert data to dataframe
df = pd.DataFrame(data)
# Create a new column with combination of "Customer" and "item_name" to feed the "stratify" parameter
# train_test_split method which is a part of "sklearn.model_selection"
df["CustAndItem"] = df["Customer"]+"_"+df["item_name"]

# First split the "train" and "test" set. In this example I have split %40 of the data as "test"
# and %60 of data as "train"
X_train, X_test, y_train, y_test = train_test_split(df.index,
                                                    df["CustAndItem"],
                                                    test_size=0.4,
                                                    stratify=df["CustAndItem"])

# Get actual data after split operation
df_train = df.loc[X_train].copy(True)
df_test = df.loc[X_test].copy(True)

# Now split "test" set to "validation" and "test" sets. In this example I have split them equally 
# (test_size = 0.5) which will contain %20 of the main set.
X_validate, X_test, y_validate, y_test = train_test_split(df_test.index,
                                                          df_test["CustAndItem"],
                                                          test_size= 0.5,
                                                          stratify=df_test["CustAndItem"])
# Get actual data after split
df_validate = df_test.loc[X_validate]
df_test = df_test.loc[X_test]

# Print results
print(df_train)
print(df_validate)
print(df_test)