使用 apply 方法提高 pandas 的性能

Question

我正在使用 pandas 进行高性能计算，下面的函数给出 1 个循环，最好的 5 个：每个循环 7.24 秒 50,000 行。

我必须将其扩展到 100 万行。

如何向量化函数并应用于所有行。这样可以提高整体性能？

def weightedFlowAmt(startDate,endDate,tradeDate,tradeAmt):
  startInDays = datetime.strptime(startDate, "%Y-%m-%d")
  endInDays = datetime.strptime(endDate, "%Y-%m-%d")
  tradeInDays = datetime.strptime(tradeDate, "%Y-%m-%d")
  differenceTradeAndEnd=abs((endInDays - tradeInDays).days)
  differenceStartAndEnd=abs((endInDays - startInDays).days)
  weighted_FlowAmt = (tradeAmt * differenceTradeAndEnd)/differenceStartAndEnd

mutatedCashFlow['flow'] = mutatedCashFlow.apply(lambda row:
        weightedFlowAmt(row['startDate'], row['EndDate'], row['tradeDate'],
                        row['tradeAmount']),
    axis=1)

Answer 1

我认为您可以删除 apply 并使用矢量化函数：

mutatedCashFlow['startDate'] = pd.to_datetime(mutatedCashFlow['startDate'])
mutatedCashFlow['EndDate'] = pd.to_datetime(mutatedCashFlow['EndDate'])
mutatedCashFlow['tradeDate'] = pd.to_datetime(mutatedCashFlow['tradeDate'])

diffTradeAndEnd=((mutatedCashFlow['EndDate']-mutatedCashFlow['tradeDate']).dt.days).abs()
diffStartAndEnd=((mutatedCashFlow['EndDate']-mutatedCashFlow['startDate']).dt.days).abs()

mutatedCashFlow['flow'] = (mutatedCashFlow['tradeAmount']*diffTradeAndEnd)/diffStartAndEnd

替代方案：

mutatedCashFlow['startDate'] = pd.to_datetime(mutatedCashFlow['startDate'])
mutatedCashFlow['EndDate'] = pd.to_datetime(mutatedCashFlow['EndDate'])
mutatedCashFlow['tradeDate'] = pd.to_datetime(mutatedCashFlow['tradeDate'])

diffTradeAndEnd=mutatedCashFlow['EndDate'].sub(mutatedCashFlow['tradeDate']).dt.days.abs()
diffStartAndEnd=mutatedCashFlow['EndDate'].sub(mutatedCashFlow['startDate']).dt.days.abs()

mutatedCashFlow['flow'] = mutatedCashFlow['tradeAmount'].mul(diffTradeAndEnd)
                                                        .div(diffStartAndEnd)
print (mutatedCashFlow)