用于比较的高效 Pandas 行迭代

Question

我有一个基于在线游戏 EVE 的市场数据的大型数据框。 我正在尝试根据商品的买入或卖出订单的价格来确定最有利可图的交易。 我发现循环所有可能性需要相当长的时间，并希望获得一些关于如何使我的代码更高效的建议。

数据 = https://market.fuzzwork.co.uk/orderbooks/latest.csv.gz

设置：

import pandas as pd
df = pd.read_csv('latest.csv', sep='\t', names=["orderID","typeID","issued","buy","volume","volumeEntered","minVolume","price","stationID","range","duration","region","orderSet"])

遍历所有可能性

buy_order = df[(df.typeID == 34) & (df.buy == True)].copy()
sell_order = df[(df.typeID == 34) & (df.buy == False)].copy()

profitable_trade = []

for i in buy_order.index:
    for j in sell_order.index:

        if buy_order.loc[i,'price'] > sell_order.loc[j, 'price']:
            profitable_trade.append(buy_order.loc[i, ['typeID', 'orderID', 'price', 'volume', 'stationID', 'range']].tolist() + sell_order.loc[j, ['orderID', 'price', 'volume', 'stationID', 'range']].tolist())

这需要相当长的时间（锐龙 2600x 上 33 秒，M1 Pro 上 12 秒）

缩短迭代时间

buy_order = df[(df.typeID == 34) & (df.buy == True)].copy()
sell_order = df[(df.typeID == 34) & (df.buy == False)].copy()

buy_order.sort_values(by='price', ascending=False, inplace=True, ignore_index=True)
sell_order.sort_values(by='price', ascending=True, inplace=True, ignore_index=True)

for i in buy_order.index:
    if buy_order.loc[i, 'price'] > sell_order.price.min():
        for j in sell_order.index:

            if buy_order.loc[i,'price'] > sell_order.loc[j, 'price']:
                profitable_trade2.append(buy_order.loc[i, ['typeID', 'orderID', 'price', 'volume', 'stationID', 'range']].tolist() + sell_order.loc[j, ['orderID', 'price', 'volume', 'stationID', 'range']].tolist())
            else:
                break
    else:
        break

这节省了大约 25%-30% 的时间（2600x 为 23 秒，M1 Pro 为 9 秒）

时间已记录在 Jupyter Notebook 中

欢迎任何提示！

Answer 1

选项 1 - 遍历所有可能性（您的）：

start = time.time()
    
buy_order = df[(df.typeID == 34) & (df.buy == True)].copy()
sell_order = df[(df.typeID == 34) & (df.buy == False)].copy()
    
profitable_trade = []
    
for i in buy_order.index:
    for j in sell_order.index:
    
        if buy_order.loc[i,'price'] > sell_order.loc[j, 'price']:
            profitable_trade.append(buy_order.loc[i, ['typeID', 'orderID', 'price', 'volume', 'stationID', 'range']].tolist() + sell_order.loc[j, ['orderID', 'price', 'volume', 'stationID', 'range']].tolist())
    
stop = time.time()
print(f"Time: {stop - start} seconds")

时间：33.145344734191895 秒

选项 2 - 缩短迭代（您的）：

start = time.time()
    
buy_order = df[(df.typeID == 34) & (df.buy == True)].copy()
sell_order = df[(df.typeID == 34) & (df.buy == False)].copy()
    
buy_order.sort_values(by='price', ascending=False, inplace=True, ignore_index=True)
sell_order.sort_values(by='price', ascending=True, inplace=True, ignore_index=True)
    
profitable_trade2 = []
    
for i in buy_order.index:
    if buy_order.loc[i, 'price'] > sell_order.price.min():
        for j in sell_order.index:
    
            if buy_order.loc[i,'price'] > sell_order.loc[j, 'price']:
                    profitable_trade2.append(buy_order.loc[i, ['typeID', 'orderID', 'price', 'volume', 'stationID', 'range']].tolist() + sell_order.loc[j, ['orderID', 'price', 'volume', 'stationID', 'range']].tolist())
            else:
                break
    else:
        break
    
stop = time.time()
print(f"Time: {stop - start} seconds")

时间：26.736826419830322 秒

选项 3 - Pandas 优化：

您可以通过应用以下优化来获得一些加速：

• 直接迭代数据框项（iterrows 而不是 index + loc）
• 卖单的单一过滤操作

start = time.time()
    
buy_order = df[(df.typeID == 34) & (df.buy == True)]
sell_order = df[(df.typeID == 34) & (df.buy == False)]
    
profitable_trade = []
    
for _, buy in buy_order.iterrows():
    filtered_sell_orders = sell_order[sell_order["price"] < buy["price"]]
    for _, sell in filtered_sell_orders.iterrows():
        profitable_trade.append(buy[['typeID', 'orderID', 'price', 'volume', 'stationID', 'range']].tolist() + sell[['orderID', 'price', 'volume', 'stationID', 'range']].tolist())
    
stop = time.time()
print(f"Time: {stop - start} seconds")

时间：19.43745183944702 秒

请注意，几乎所有时间都花在tolist()-操作上（以下选项仅用于显示这种影响，它不会返回目标列表）：

start = time.time()

buy_order = df[(df.typeID == 34) & (df.buy == True)]
sell_order = df[(df.typeID == 34) & (df.buy == False)]
    
profitable_trade = []
    
for _, buy in buy_order.iterrows():
    filtered_sell_orders = sell_order[sell_order["price"] < buy["price"]]
    for _, sell in filtered_sell_orders.iterrows():
        # removed 'tolist'-operations
        profitable_trade.append(1)
    
stop = time.time()
print(f"Time: {stop - start} seconds")

时间：2.072049617767334 秒

选项 4 - 替换 tolist-operations 并将结果存储在数据框中：

您可以通过以下方式加速您的代码

• 将过滤后的值存储在包含原始数据帧行的中间列表中
• 将中间列表转换为数据帧并将它们连接起来
• 生成的数据帧产生与列表profitable_trade相同的信息

start = time.time()

buy_orders = df[(df.typeID == 34) & (df.buy == True)]
sell_orders = df[(df.typeID == 34) & (df.buy == False)]

# store buy and cell rows in intermediate lists
buys = []
sells = []

for _, buy in buy_orders.iterrows():
    # apply filtering operation once
    filtered_sell_orders = sell_orders[sell_orders.price < buy.price]
    sell_rows = list(filtered_sell_orders.iterrows())

    # store buy and sell row items
    buys.extend([buy] * len(sell_rows))
    sells.extend([sell for _, sell in sell_rows])

# convert intermediate lists to dataframes
buys = pd.DataFrame(buys)
sells = pd.DataFrame(sells)

# rename columns for buys / cells dataframes for unique column names
buys = buys.rename(columns={column: f"{column}_buy"  for column in buys.columns})
sells = sells.rename(columns={column: f"{column}_sell"  for column in sells.columns})

# reset indices and concatenate buys / cells along the column axis
buys.reset_index(drop=True, inplace=True)
sells.reset_index(drop=True, inplace=True)
profitable_trade = pd.concat([buys, sells], axis=1)

stop = time.time()
print(f"Time: {stop - start} seconds")

时间：3.661072015762329 秒