使用配置文件值循环过滤数据帧答案

【问题标题】：Filtering dataframe in a loop with use of config file values使用配置文件值循环过滤数据帧
【发布时间】：2022-07-21 17:59:18
【问题描述】：

我有以下玩具数据集

data = {"Subject":["1","2","3","3","4","5","5"],
    "date": ["2020-05-01 16:54:25","2020-05-03 10:31:18","2020-05-08 10:10:40","2020-05-08 10:10:42","2020-05-06 09:30:40","2020-05-07 12:46:30","2020-05-07 12:55:10"],
    "Accept": ["True","False","True","True","False","True","True"],
    "Amount" : [150,30,32,32,300,100,50],
    "accept_1": ["True","False","True","True","False","True","True"],
    "amount_1" : [20,30,32,32,150,100,30],
    "Transaction":["True","True","False","False","True","True","False"],
    "Label":["True","True","True","False","True","True","True"]}
     data = pd.DataFrame(data)

还有一个小玩具配置文件

config = [{"colname": "Accept","KeepValue":"True","RemoveTrues":"True"},
    {"colname":"Transaction","KeepValue":"False","RemoveTrues":"False"}]

我想遍历数据集并应用这些过滤器。在我应用了第一个过滤器后，我想对过滤后的数据应用以下过滤器等等。

我运行以下代码，它似乎第一次对数据应用过滤器，然后对原始数据应用第二个过滤器，而不是过滤后的数据。

for i in range(len(config)):
    filtering = config[i]
    if filtering["RemoveTrues"] == "True":
        col = filtering["colname"]
        test  = data[data[col] == filtering["KeepValue"]]
        print(test)
    else:
        col = filtering["colname"]
        test = data[(data[col]== filtering["KeepValue"]) | data["Label"]]
        print(test)

如何对数据应用第一个过滤器，然后对过滤后的数据应用第二个过滤器，依此类推？我需要使用循环，因为我必须从配置文件中获取过滤器。

【问题讨论】：

标签： python pandas dataframe if-statement

【解决方案1】：

我建议将 True/False 字符串更改为布尔值。您可以为 df 分配一个新值，该值将在循环期间持续存在（不要创建额外的 test 变量）。

df = pd.DataFrame(data)

config = [{"colname": "Accept","KeepValue":"True","RemoveTrues":"True"},
    {"colname":"Transaction","KeepValue":"False","RemoveTrues":"False"}]

for conf in config:
    if conf["RemoveTrues"] == "True":
        df = df[df[conf['colname']] == conf['KeepValue']]
        print(df)
    else:
        df = df[(df[conf['colname']]== conf["KeepValue"]) | df["Label"]]
        print(df)

【讨论】：

【解决方案2】：

据我所知，您希望在每次发生过滤时都保存过滤，并且根据我在代码中看到的每个循环您尝试过滤的内容，但使用数据框的原始引用，它将执行此操作每次在原始数据帧上的过滤器，您必须将其更改为新的引用，称为“test”，并将其保存到相同的引用“test”，以便在下一个循环中使用

test = data.copy() # copy the original dataframe so we can refreance for it each time in loop
for i in range(len(config)):
    filtering = config[i]
    if filtering["RemoveTrues"] == "True":
        col = filtering["colname"]
        test  = test[test[col] == filtering["KeepValue"]] # change it to the new reference, and save it to the same reference so it can be used in next loop
        print(test)
    else:
        col = filtering["colname"]
        test = test[(test[col]== filtering["KeepValue"]) | test["Label"]] # change it to the new reference, and save it to the same reference so it can be used in next loop
        print(test)

【讨论】：