【问题标题】:python for loop parallel processing - appending data to listpython for loop parallel processing - appending data to list
【发布时间】:2022-12-01 21:09:14
【问题描述】:

I am have below step in my code which is taking around 45 to 50 mins to run (there are other steps which barely take few seconds)

So I am trying to optimize the execution/run time for this step it is essentially a for loop inside a function

def getSwitchStatus(dashboard: meraki.DashboardAPI,switches): 
    statuses = [] 
    #Establish the timestamp for midnight yesterday to enable collecting of yesterdays data 
    yesterday_midnight = datetime.combine(datetime.today(), time.min) - timedelta(days = 1) 
    for dic in switches:
        statuses.append(dashboard.switch.getDeviceSwitchPortsStatuses(dic['serial'],t0=yesterday_midnight)) 
    return statuses 

Here is what I have tried to do so far

def switchsts():
    print("Inside switchsts")
    for dic in switches:
        statuses.append(dashboard.switch.getDeviceSwitchPortsStatuses(dic['serial'],t0=yesterday_midnight)) 


def getSwitchStatus(dashboard: meraki.DashboardAPI,switches): 
    print("Testing if switches is accessible")
    print("Switches type",type(switches))
    print("Switches",switches[0])

    p = Process(target=switchsts,args=())
    p.start()
    p.join()
    return statuses
    print(statuses)

Unfortunately this is throwing an error here:

    for dic in switches:

NameError: name 'switches' is not defined

Which is strange because I am able to print 'Switches' when the code reaches inside the getswitchstatus function but somehow the function that I am trying to parallelize doesnt seem to read it.

Inside switchsts Process Process-1:

Traceback (most recent call last): File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2288.0_x64__qbz5n2kfra8p0\lib\multiprocessing\process.py", line 314, in _bootstrap self.run() File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2288.0_x64__qbz5n2kfra8p0\lib\multiprocessing\process.py", line 108, in run self._target(*self._args, **self._kwargs) File "C:\Sample_project\venv\ciscomeraki_file_parallelprocessing.py", line 83, in switchsts for dic in switches: NameError: name 'switches' is not defined

P.S.: I am new to parellel processing so I am guessing I am making some rookie mistake

*Edit1Adding code for 'switches'

   def getSwitches(dashboard: meraki.DashboardAPI,orgID, network_id=False): 
if network_id is False or network_id is None: 
    devices = dashboard.organizations.getOrganizationDevices( 
        orgID, 
        total_pages='all', 
        productTypes='switch' 
    ) 
    return devices 
else: 
    devices = dashboard.organizations.getOrganizationDevices( 
        orgID, 
        total_pages='all', 
        productTypes='switch', 
        networkIds=network_id 
    ) 
    return devices

【问题讨论】:

    标签: python python-3.x for-loop parallel-processing


    【解决方案1】:

    It's saying that your switchsts() fucntion have not switches in it.

    so try this if it works:

    def switchsts(switches):
        print("Inside switchsts")
        for dic in switches:
            statuses.append(dashboard.switch.getDeviceSwitchPortsStatuses(dic['serial'],t0=yesterday_midnight)) 
    
    def getSwitchStatus(dashboard: meraki.DashboardAPI,switches): 
        print("Testing if switches is accessible")
        print("Switches type",type(switches))
        print("Switches",switches[0])
    
        p = Process(target=switchsts, args=(switches,))
        p.start()
        p.join()
        return statuses
        print(statuses)
    

    【讨论】:

    • tried that def switchsts(switches): TypeError: switchsts() takes 1 positional argument but 175 were given
    • the 'switches' is essentially a list with 175 items
    • define switches globally. Or can you share code code also here?
    • have added the code for switches in original code , its gets called as switches = getSwitches(dashboard,orgID)
    • p = Process(target=switchsts, args=(switches,))
    猜你喜欢
    • 2022-12-26
    • 2019-05-28
    • 1970-01-01
    • 2021-08-17
    • 2023-02-26
    • 2017-04-26
    • 2019-09-24
    • 1970-01-01
    • 2018-10-01
    相关资源
    最近更新 更多