Python 程序在第 400 次循环后被杀死

Question

我的 python 程序在大约 400 次循环后被杀死，它没有完成我的列表。该程序用于重新网格化来自 ESRI ascii 的降雨数据并导出为 netCDF。

我的系统是ubuntu 18.xx，python是3.7版本。我本来是在jupyter lab上跑的，我想可能是jupyter lab的约束。然后，我作为脚本运行，但它是一样的。

# Libraries
import numpy as np
import pandas as pd
import xarray as xr
import xesmf as xe
import os

# Funtion to read ESRI ASCII
def read_grd(filename):
    with open(filename) as infile:
        ncols = int(infile.readline().split()[1])
        nrows = int(infile.readline().split()[1])
        xllcorner = float(infile.readline().split()[1])
        yllcorner = float(infile.readline().split()[1])
        cellsize = float(infile.readline().split()[1])
        nodata_value = int(infile.readline().split()[1])
    lon = xllcorner + cellsize * np.arange(ncols)
    lat = yllcorner + cellsize * np.arange(nrows)
    value = np.loadtxt(filename, skiprows=6)

    return lon, lat, value


### Static data setting ===============================

# Input variables (date)
dir_rainfall = "./HI_1994_1999/"
dir_out = "./HI_1994_1999_regridded/"
arr = sorted(os.listdir(dir_rainfall)) #len(arr) = 50404

# Read the spatial data that we're going to regridde
ds_sm = xr.open_dataset('./Spatial_Metadata.nc',autoclose=True)

ds_out = xr.open_dataset('./regrid_frame.nc')
ds_out.rename({'XLONG_M': 'lon', 'XLAT_M': 'lat'}, inplace=True)
ds_out['lat'] = ds_out['lat'].sel(Time=0, drop=True)
ds_out['lon'] = ds_out['lon'].sel(Time=0, drop=True)

### LOOP ==============================================
i = 1690  # It kept stopped every ~400 iteration, so I added this to restart from the last killed.
arr = arr[i:len(arr)]

for var in arr:

    # get dateTime
    dateTime = var.replace('hawaii_',"")
    dateTime = dateTime.replace("z","")
    print("Regridding now " + str(i) + " : " + dateTime)

    # Read rainfall ASCII and calculate rain rate
    asc = read_grd(dir_rainfall + var)
    precip_rate = np.array(asc[2] / (60.0*60.0))
    precip_rate = precip_rate.astype('float')
    precip_rate[precip_rate == -9999.] = np.nan
    x = np.repeat([asc[0]], 1, axis=0).transpose()
    y = np.repeat([asc[1]], 1, axis=0)

    # Format rain rate to netCDF
    ds = xr.Dataset({'RAINRATE': (['lat', 'lon'],  precip_rate)},
                    coords={'lon': (['lon'], asc[0]),
                            'lat': (['lat'], asc[1])})

    # Regrid
    regridder = xe.Regridder(ds, ds_out, 'bilinear', reuse_weights=True) # create regriding frame

    dr = ds.RAINRATE
    dr_out = regridder(dr)

    # change the name of coordinates for the regridded netCDF
    dr_out.coords['west_east'] = ds_sm.x.values
    dr_out.coords['south_north'] = ds_sm.y.values
    dr_out = dr_out.rename({'west_east': 'x', 'south_north': 'y'})

    # add attributes to the regridded netCDF
    dr_out.attrs['esri_pe_string'] = ds_sm.crs.attrs['esri_pe_string']
    dr_out.attrs['units'] = 'mm/s'


    # Export the YYYYMMDDHHMMPRECIP_FORCING.nc
    dr_out.to_netcdf(dir_out + dateTime + '00.PRECIP_FORCING.nc')

    i = i + 1

前约 400 个 ESRI ASCII 已成功转换。然后它就卡住了，而我在 jupyter lab 中运行它；如果我从脚本中下雨，python xxx.py，在运行约 400 次后，它会从我的终端返回“Killed”。

=== [edit 7/11/2019] 添加内存使用信息 ===

根据@zmike，可能是内存问题，所以我打印了内存，这是内存问题-内存一直累积。

我将下面的代码添加到我的代码中。

    # Print out total memory
    process = psutil.Process(os.getpid())
    print(process.memory_info().rss)

Answer 1

经过搜索和测试，我正在回答自己的问题，希望对有相同问题的人有所帮助：

python 程序由于内存不足而被杀死。减少内存的方法很少，包括

• 删除变量并使用gc.collect() (How can I release memory after creating matplotlib figures)
• 优化了代码（https://www.codementor.io/satwikkansal/python-practices-for-efficient-code-performance-memory-and-usability-aze6oiq65；https://dzone.com/articles/python-memory-issues-tips-and-tricks；https://www.reddit.com/r/Python/comments/4fcnfy/how_can_i_manage_memory_in_python_running_out_of/）

我可以从上述步骤中释放一些内存，它最多可以循环约 500 次（当我在服务器而不是个人计算机上运行时，最多可以循环约 2000 次）。内存问题仍然存在，但已经好转。