如何清理数据框字典的旧数据？答案

【问题标题】：How to clean up older data for a dictionary of dataframes?如何清理数据框字典的旧数据？
【发布时间】：2020-06-16 14:33:31
【问题描述】：

这是我的代码的 sn-p。我需要清理一天的旧数据。我们如何为数据帧字典做到这一点？

    master_train_dict = {}
    for id in list_of_id:
        temp_df = df.loc[df["id"] == id].copy(deep=False)
        temp_df.drop('id', axis=1, inplace=True)
        temp_df.reset_index(drop=True, inplace=True)
        alert_list = list(temp_df["title"])
        train_embedding = get_embeddings(alert_list, model)
        temp_df["train_embedding"] = train_embedding
        master_train_dict[parent_id] = 
        temp_df[["title","train_embedding","@timestamp"]]
        #master_train_dict[parent_id] = temp_df
    global master_dict
master_dict = master_train_dict    
print(master_dict)
#clean up function
if len(master_dict)>0:
    d = datetime.today() - timedelta(hours=1, minutes= 0)
    master_dict=master_dict[id]['@timestamp']>d.strftime("%Y-%m-%d %H:%M:%S")
    print(master_dict)

【问题讨论】：

parent_id 是从哪里得到的？不应该是master_train_dict[id] = ...中的id吗？

标签： python list dataframe dictionary

【解决方案1】：

考虑使用已定义的方法并使用groupby 构建子集数据帧的列表或字典。然后通过字典推导调用函数。

def build_df(sub):
   sub_df.drop('id', axis=1, inplace=True)
   sub_df.reset_index(drop=True, inplace=True)

   alert_list = list(sub_df["title"])
   train_embedding = get_embeddings(alert_list, model)

   sub_df["train_embedding"] = train_embedding
   sub_df = sub_df.reindex(["title","train_embedding","@timestamp"], axis="columns")

   return sub_df

master_train_dict = {i:build_df(g) for i, g in df.groupby(["id"])}


def clean_df(df):        
    d = datetime.today() - timedelta(hours=1, minutes= 0)
    df = df[df['@timestamp'] > d.strftime("%Y-%m-%d %H:%M:%S")]   

    return df 

clean_master_train_dict = {k:clean_df(v) for k, v in master_train_dict.items()}

【讨论】：