如何并行化列绑定的函数？

Question

我有一个函数，它对每个 DataFrame 列执行一些操作并从中提取较短的序列（在原始代码中进行了一些耗时的计算） 然后在继续下一列之前将其添加到字典中。

最后它从字典中创建一个数据框并操作它的索引。

如何并行化处理每一列的循环？

这是一个不太复杂的可重现代码示例。

import pandas as pd
raw_df = pd.DataFrame({"A":[ 1.1 ]*100000, 
                       "B":[ 2.2 ]*100000,
                       "C":[ 3.3 ]*100000})

def preprocess_columns(raw_df, ):
 
    df = {}
    width = 137 
 
    for name in raw_df.columns:
        '''
        Note: the operations in this loop do not have a deep sense and are just for illustration of the function preprocess_columns. In the original code there are ~ 50 lines of list comprehensions etc.
        '''


        # 3. do some column operations. (actually theres more than just this operation)
        seriesF =  raw_df[[name]].dropna()
        afterDropping_indices = seriesF.index.copy(deep=True) 
        list_ = list(raw_df[name])[width:]  
        df[name]=pd.Series(list_.copy(), index=afterDropping_indices[width:]) 
    
    
    # create df from dict and reindex
    df=pd.concat(df,axis=1) 
    df=df.reindex(df.index[::-1])
    return df 
 
raw_df = preprocess_columns(raw_df )

Answer 1

也许你可以使用这个： https://github.com/xieqihui/pandas-multiprocess

pip install pandas-multiprocess

from pandas_multiprocess import multi_process


args = {'width': 137}
result = multi_process(func=func, data=df, num_process=8, **args)