Python Pool.map() - 在本地工作，在服务器上失败答案

【问题标题】：Python Pool.map() - locally works, on server failsPython Pool.map() - 在本地工作，在服务器上失败
【发布时间】：2016-10-04 14:35:35
【问题描述】：

我在 SO 上研究了许多 pool.map ，但似乎仍然找不到任何暗示我的问题的东西。

我在每个 .py 文件中都有if __name__ == '__main__'。我在每个包含import multiprocessing 的.py 中都有freeze_support()，我仍然对正在发生的事情感到茫然。我已经在我的代码中移动了 freeze_support()，但结果同样不成功。

脚本 A 调用脚本 B，脚本 B 调用脚本 C（多处理发生的地方）。在本地，这个场景完美运行，但是当我将它加载到 Windows Server 2008 机器上时，奇怪的事情开始发生。

在服务器上，我可以看到第一个迭代打印到解释器，但它随后跳回脚本 B 并继续处理。脚本 C 的列表中还有 51 项。

脚本 B 代码：

if not arcpy.Exists(MergedDataFC):
    ScriptC.intersect_main(input1, input2) 

if not arcpy.Exists(MergedDataSHP):
    shpList = arcpy.ListFields(*.shp) # output of multiprocess
    # Merge all shapefiles into single shapefile
    # Being executed before the multiprocess finishes all 52 items

脚本 C 代码：

import multiprocessing as mp

def intersect_main(input1,input2):      
try: 
    mp.freeze_support()
    # Create a list of states for input1 polygons  
    log.log("Creating Polygon State list...")  
    fldList = arcpy.ListFields(input1) 
    flds = [fld.name for fld in fldList]
    idList = []  
    with arcpy.da.SearchCursor(input1, flds) as cursor:  
        for row in cursor:  
            idSTATE = row[flds.index("STATE")]
            idList.append(idSTATE)  

    idList = set(idList)
    log.log("There are " + str(len(idList)) + " States (polygons) to process.")  

    log.log("Sending to pool")  
    # declare number of cores to use, use 1 less than the max  
    cpuNum = mp.cpu_count() -1

    # Create the pool object  
    pool = mp.Pool(processes=cpuNum)  

    # Fire off list to worker function.  
    # res is a list that is created with what ever the worker function is returning
    log.log ("Entering intersectWork")  
    res = pool.map((intersectWork(input1, input2, idSTATE)),idList)
    pool.close()  
    pool.join()  

    # If an error has occurred report it  
    if False in res:  
        log.log ("A worker failed!")  
        log.log (strftime('[%H:%M:%S]', localtime()))
        raise Exception
    else:
        log.log("Finished multiprocessing!")
        log.log (strftime('[%H:%M:%S]', localtime()))  
except Exception, e:
    tb = sys.exc_info()[2]   
    # Geoprocessor threw an error 
    log.log("An error occurred on line " + str(tb.tb_lineno)) 
    log.log (str(e))

def intersectWork(input1,input2, idSTATE):  
try:  
    if idSTATE == None:
        query = "STATE IS NULL"
        idSTATE = 'pr'
    else:
        query = "STATE = '" + idSTATE + "'"

    DEMOlayer = arcpy.MakeFeatureLayer_management(input1,"input1_" + idSTATE)

    log.log (query)
    arcpy.SelectLayerByAttribute_management(DEMOlayer,"NEW_SELECTION",query)  

    # Do the Intersect  
    outFC = r'C:/EclipseWorkspace' + '/INTER_' + idSTATE.upper() + '.shp'
    strIntersect = str(DEMOlayer) + ";" + str(input2)
    arcpy.Intersect_analysis(strIntersect, outFC, "ALL", "", "LINE")
    return True  
except:  
    # Some error occurred so return False 
    log.log(arcpy.GetMessage(2))
    return False

if __name__ == '__main__':
    intersect_main(input1, input2)

编辑

服务器上的所有数据都存储在本地，不跨网络处理。

【问题讨论】：

标签： python windows server multiprocessing arcpy

【解决方案1】：

问题是数据的完整路径没有从以前的模块正确传递到服务器上的 pool.map() 中。我必须在 import 语句下添加所有文件路径。看起来不是很优雅，但它正在工作。

【讨论】：