避免循环熊猫系列；动态更改参数的子集答案

【问题标题】：Avoiding to loop a pandas series; Subsetting on dynamically changed arguments避免循环熊猫系列；动态更改参数的子集
【发布时间】：2016-03-09 15:19:01
【问题描述】：

我知道最好避免循环/迭代熊猫系列。但是，对于我的用例，我找不到合适的解决方案。

我有一个整数系列，如 (100,101,103,102,99,100,102) 等，我想在值向上移动时找到条目/索引，例如3. 对于上面的迷你系列，这将是索引值 (2,6)。

系统信息：Python 版本：3.5.0 |Anaconda 2.5.0（64 位）| （默认，2015 年 10 月 19 日，21:57:25）[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]，Pandas 版本：'0.17.1'

这是我的代码，包括一个示例系列

import pandas as pd


def getMoves(t: pd.Series, triggervar = 3):
    # t is a series of integer values
    # iterate a given series and collect all movements that are equal or greater than 3
    triggerindexes= []

    currentreference = t[0]
    trigger = t[0] + triggervar

    for item in t.iteritems():
        print(item)
        if item[1] >= trigger:
            triggerindexes.append(item[0])
            currentreference = item[1]
            trigger = item[1] + triggervar
            print('--- New move triggered :' + str(item))
        elif item[1] < currentreference:
            currentreference = item[1]
            trigger = item[1] + triggervar
            print('--- Set trigger, as current value lower than refvalue, new refvalue: ' + str(item[1]))

    return triggerindexes



#create a sample time series
ts = pd.read_json('{"1457427600000":100,"1457427615000":100,"1457427630000":101,"1457427645000":100,"1457427660000":100,"1457427675000":101,"1457427690000":101,"1457427705000":101,"1457427720000":100,"1457427735000":99,"1457427750000":98,"1457427765000":97,"1457427780000":96,"1457427795000":97,"1457427810000":98,"1457427825000":97,"1457427840000":98,"1457427855000":97,"1457427870000":98,"1457427885000":98,"1457427900000":97,"1457427915000":97,"1457427930000":96,"1457427945000":97,"1457427960000":96,"1457427975000":97,"1457427990000":96,"1457428005000":96,"1457428020000":95,"1457428035000":96,"1457428050000":95,"1457428065000":94,"1457428080000":94,"1457428095000":95,"1457428110000":96,"1457428125000":97,"1457428140000":97,"1457428155000":97,"1457428170000":97,"1457428185000":97,"1457428200000":98,"1457428215000":98,"1457428230000":99,"1457428245000":99,"1457428260000":99,"1457428275000":98,"1457428290000":97,"1457428305000":96,"1457428320000":96,"1457428335000":96,"1457428350000":96,"1457428365000":97,"1457428380000":97,"1457428395000":98,"1457428410000":99,"1457428425000":100,"1457428440000":101,"1457428455000":102,"1457428470000":101,"1457428485000":101,"1457428500000":102,"1457428515000":102,"1457428530000":103,"1457428545000":103,"1457428560000":102,"1457428575000":102,"1457428590000":101,"1457428605000":102,"1457428620000":102,"1457428635000":103,"1457428650000":102,"1457428665000":103,"1457428680000":103,"1457428695000":104,"1457428710000":104,"1457428725000":104,"1457428740000":105,"1457428755000":104,"1457428770000":103,"1457428785000":104,"1457428800000":104,"1457428815000":104,"1457428830000":104,"1457428845000":103,"1457428860000":102,"1457428875000":101,"1457428890000":102,"1457428905000":102,"1457428920000":101,"1457428935000":101,"1457428950000":102,"1457428965000":103,"1457428980000":102,"1457428995000":101,"1457429010000":102,"1457429025000":102,"1457429040000":102,"1457429055000":102,"1457429070000":101,"1457429085000":101,"1457429100000":102,"1457429115000":101,"1457429130000":101,"1457429145000":102,"1457429160000":102,"1457429175000":101,"1457429190000":100,"1457429205000":100,"1457429220000":101,"1457429235000":100,"1457429250000":101,"1457429265000":100,"1457429280000":99,"1457429295000":100,"1457429310000":99,"1457429325000":100,"1457429340000":100,"1457429355000":99,"1457429370000":98,"1457429385000":98,"1457429400000":97,"1457429415000":97,"1457429430000":96,"1457429445000":95,"1457429460000":96,"1457429475000":97,"1457429490000":98,"1457429505000":99,"1457429520000":98,"1457429535000":97,"1457429550000":98,"1457429565000":97,"1457429580000":96,"1457429595000":95,"1457429610000":94,"1457429625000":93,"1457429640000":94,"1457429655000":95,"1457429670000":94,"1457429685000":95,"1457429700000":94,"1457429715000":94,"1457429730000":95,"1457429745000":95,"1457429760000":94,"1457429775000":93,"1457429790000":94,"1457429805000":93,"1457429820000":92,"1457429835000":91,"1457429850000":92,"1457429865000":93,"1457429880000":93,"1457429895000":92,"1457429910000":92,"1457429925000":91,"1457429940000":90,"1457429955000":89,"1457429970000":90,"1457429985000":89,"1457430000000":88,"1457430015000":88,"1457430030000":89,"1457430045000":89,"1457430060000":90,"1457430075000":90,"1457430090000":90,"1457430105000":90,"1457430120000":91,"1457430135000":90,"1457430150000":91,"1457430165000":91,"1457430180000":91,"1457430195000":92,"1457430210000":92,"1457430225000":91,"1457430240000":91,"1457430255000":92,"1457430270000":93,"1457430285000":92,"1457430300000":93,"1457430315000":94,"1457430330000":93,"1457430345000":94,"1457430360000":95,"1457430375000":94,"1457430390000":95,"1457430405000":94,"1457430420000":94,"1457430435000":93,"1457430450000":93,"1457430465000":94,"1457430480000":94,"1457430495000":93,"1457430510000":94,"1457430525000":95,"1457430540000":94,"1457430555000":93,"1457430570000":93,"1457430585000":92,"1457430600000":91,"1457430615000":90,"1457430630000":90,"1457430645000":89,"1457430660000":88,"1457430675000":87,"1457430690000":86,"1457430705000":85,"1457430720000":85,"1457430735000":85,"1457430750000":85,"1457430765000":86,"1457430780000":85,"1457430795000":85,"1457430810000":84,"1457430825000":83,"1457430840000":84,"1457430855000":85,"1457430870000":85,"1457430885000":84,"1457430900000":84,"1457430915000":83,"1457430930000":83,"1457430945000":84,"1457430960000":83,"1457430975000":84,"1457430990000":85,"1457431005000":86,"1457431020000":87,"1457431035000":86,"1457431050000":86,"1457431065000":85,"1457431080000":84,"1457431095000":84,"1457431110000":84,"1457431125000":83,"1457431140000":82,"1457431155000":83,"1457431170000":82,"1457431185000":81}', typ='series', orient='records')

#get up-moves by 3; resulting in 9 hits
results=getMoves(ts)

嗯，它有效，但由于我的现实世界问题有很多记录（> 25 亿），这是一种非常缓慢的方法。我通过 apply 应用函数尝试了一些事情，但由于参考值是动态设置的，我找不到可行的解决方案。我也尝试研究管道功能，但它让我很困惑。

因此，如果有人有一些提示可以避免详尽的循环，我将非常高兴和感激。这是我在这里的第一个问题，因为我还是个新手，我希望我能很好地提出我的问题。

【问题讨论】：

标签： python pandas vectorization series

【解决方案1】：

这是一个更新，因为我找到了一个好的解决方案。

我通过使用 numba 库显着提高了速度。很抱歉将其发布为答案，但我找不到我的编辑按钮：(

增强是

%timeit getMoves_numba(series) 100 个循环，3 个循环中的最佳：每个循环 9.72 毫秒

与纯 Python 相比

%timeit getMoves(series) 1 个循环，3 次取胜：每个循环 1.87 秒

我创建的样本系列有 500 万条记录。函数 getMoves 现在返回一个布尔索引数组以供进一步使用...对于我的目的，10 毫秒就足够了。只有一个声明是 getMoves_numba = nb.autojit(getMoves)... 很好:)

这是我的新代码：

import numpy as np
import numba as nb




def getMoves(v:np.array,points:int = 3, up:bool=True):
    J=len(v)
    refvalue=np.int
    bound=np.int
    result= np.zeros((J), dtype=bool)
    if up == True:
        refvalue=v[0]
        bound=refvalue+points
        for j in range(J):
            if v[j] >= bound:
                result[j]=True
                refvalue = v[j]
                bound=refvalue+points
            elif v[j] < refvalue:
                refvalue = v[j]
                bound=refvalue+points                
    else:
        refvalue=v[0]
        bound=refvalue-points
        for j in range(J):
            if v[j] <= bound:
                result[j]=True
                refvalue = v[j]
                bound=refvalue-points
            elif v[j] > refvalue:
                refvalue = v[j]
                bound=refvalue-points
    return result

np.random.seed(257931)
N=5000000
series=np.random.random_integers(-1,1,size=N)
series[0]=100
series=series.cumsum()
series = series + (abs(series.min())+1)

getMoves_numba = nb.autojit(getMoves)

这是在纯 numpy 数组上完成的，所以如果在处理 pandas 系列时使用，我必须通过 pandas.series.values 传递它。

改进是好的，但如果您对风格有提示或意见，我仍然感谢一些反馈。

问候

【讨论】：