熊猫创建数据框仅保留来自另一个数据框的列并附加

Question

鉴于，我有两个数据框。第一个是具有 22 列的数据框 (df1)。这包含所有变量的值。第二个（good_models）是一个数据框，其中包含我想从df1 中提取的感兴趣的列。对于good_models 中的每一行，我需要使用（var1-var10 和target）中的值，并仅保留df1 中的这些列。重命名列（var1-var10 和 target），然后附加到名为 model_data_long 的新数据框

我可以使用 for 循环执行此任务，但是它非常慢，我希望有更有效的方法来执行此任务。

可能我可以参考这个，但不确定如何应用它。 Stack Overflow

import numpy as np
import pandas as pd
df1=pd.DataFrame(np.random.randint(0,100,size=(100,20+1)),columns=list(range(0,20+1)))
df1['target']=np.random.randint(2,size=100)
### This needs to be the columns in model_data_long
labels=['var1','var2','var3','var4','var5','var6','var7','var8','var9','var10','target']
### Contains the columns I want to exctract from df1 and append to model_data_long    
good_models=pd.DataFrame.from_records([(0,1,2,3,4,5,6,7,8,9,'target'),
                                     (9,8,7,6,5,4,3,2,1,0,'target'),
                                     (20,19,18,17,16,15,14,13,12,11,'target')],columns=labels)
### works but is slow
model_data_long=pd.DataFrame()
for i in range(0,len(good_models)):
    ### Extracting the values for a record from good_models
    t_list=good_models[good_models.index==i].values.tolist()[0]
    ### Keeping only the columns from t_list from the df1 frame.
    temp_data=pd.DataFrame(data=df1.filter(items=t_list,axis=1))
    ### renaming the columns in temp_data
    temp_data.columns=[labels]
    ### It is imparative that I have an index variable in the model_data_long dataframe.
    ### Setting the model_index variable, critical.
    temp_data['model_index']=i
    ### Finally, append to a long running dataframe.
    model_data_long=model_data_long.append([temp_data],ignore_index=True)

Answer 1

您可以使用非常快速的 numpy 解决方案，感谢divakar 的回答：

#convert df1 to numpy array
a = df1.values
#convert first 10 columns to numpy array
b = good_models.iloc[:, :10].values
#reshape in numpy, add all columns names without last
df = pd.DataFrame(a[:, b].swapaxes(0,1).reshape(-1,b.shape[1]), columns=labels[:-1])

#add new columns - by repating with tile and repeat
df['target'] = np.tile(df1['target'].values, len(good_models))
df['model_index'] = np.repeat(good_models.index, len(df1))

时间安排：

In [251]: %timeit (jez())
100 loops, best of 3: 2.47 ms per loop

In [252]: %timeit (orig())
1 loop, best of 3: 703 ms per loop

设置：

import numpy as np
import pandas as pd
np.random.seed(452)
df1=pd.DataFrame(np.random.randint(0,100,size=(100,20+1)),columns=list(range(0,20+1)))
df1['target']=np.random.randint(2,size=100)
### This needs to be the columns in model_data_long
labels=['var1','var2','var3','var4','var5','var6','var7','var8','var9','var10','target']
### Contains the columns I want to exctract from df1 and append to model_data_long    
good_models=pd.DataFrame.from_records([(0,1,2,3,4,5,6,7,8,9,'target'),
                                     (9,8,7,6,5,4,3,2,1,0,'target'),
                                     (20,19,18,17,16,15,14,13,12,11,'target')],columns=labels)
### works but is slow

#100 times repeat rows for 300 rows                                         
good_models = pd.concat([good_models]*100).reset_index(drop=True)

功能：

def orig():

    model_data_long=pd.DataFrame()
    for i in range(0,len(good_models)):
        ### Extracting the values for a record from good_models
        t_list=good_models[good_models.index==i].values.tolist()[0]
        ### Keeping only the columns from t_list from the df1 frame.
        temp_data=pd.DataFrame(data=df1.filter(items=t_list,axis=1))
        ### renaming the columns in temp_data
        temp_data.columns=[labels]
        ### It is imparative that I have an index variable in the model_data_long dataframe.
        ### Setting the model_index variable, critical.
        temp_data['model_index']=i
        ### Finally, append to a long running dataframe.
        model_data_long=model_data_long.append([temp_data],ignore_index=True)

    return model_data_long


def jez():
    #convert df1 to numpy array
    a = df1.values
    #convert first 10 columns to numpy array
    b = good_models.iloc[:, :10].values
    #reshape in numpy, add all columns names without last
    df = pd.DataFrame(a[:, b].swapaxes(0,1).reshape(-1,b.shape[1]), columns=labels[:-1])

    #add new columns - by repating with tile and repeat
    df['target'] = np.tile(df1['target'].values, len(good_models))
    df['model_index'] = np.repeat(good_models.index, len(df1))
    return df

Answer 2

我想出了 2 种方法来做到这一点，而无需使用多处理包。 我使用了简单的熊猫技巧，看看它是否适合你。稍后我会分享另一个

pd.concat([pd.DataFrame(df1[i].values, columns=labels) for i in good_models.values.tolist()],ignore_index=True)