错误：使用 17%-21% 时超出内存限制，根据 psutils

Question

我正在使用以下要求部署具有一些密集计算的云函数：

requirements.txt 

google-cloud-storage
google-cloud-datastore 
numpy==1.16.2
pandas==0.24.2
scikit-image==0.16.1
psutil
memory-profiler==0.55.0
scikit-learn==0.20.3
opencv-python==4.0.0.21

我为部署设置了以下参数：

[--memory: "2147483648", --runtime: "python37", --timeout: "540", --trigger-http: "True", --verbosity: "debug"]

随着函数迭代处理帧，使用量增加，但当达到 18% - 21% 时，它会停止并显示：

“错误：超出内存限制。函数调用被中断。

使用 psutils 跟踪代码，在调用函数的开头我有这个输出（来自函数的日志）：

"svmem(总数=2147483648, 可用=1882365952, 百分比=12.3, 使用=152969216，免费=1882365952，活动=221151232，非活动=43954176， 缓冲区=0，缓存=112148480，共享=24240128，平板=0)"

根据我的理解，这应该意味着一开始只使用了 12.3%。 这是有道理的，因为代码包本身（包含一些二进制文件）加上原始视频块一起使用 100MB，我假设上述要求的所有安装都可能使用额外的 160MB。

大约 15 次迭代后，这是 psutil 的踪迹：

svmem(总计=2147483648, 可用=1684045824, 百分比=21.6, 使用=351272960，免费=1684045824，活动=419463168，非活动=43962368， 缓冲区=0，缓存=112164864，共享=24240128，slab=0）

然后，函数被中止。

这是代码停止的函数：

    def capture_to_array(self, capture):
        """
        Function to convert OpenCV video capture to a list of
        numpy arrays for faster processing and analysis
        """

        # List of numpy arrays
        frame_list = []
        frame_list_hd = []
        i = 0
        pixels = 0
        # Iterate through each frame in the video
        while capture.isOpened():

            # Read the frame from the capture
            ret_frame, frame = capture.read()

            # If read successful, then append the retrieved numpy array to a python list
            if ret_frame:
                i += 1
                # Count the number of pixels
                height = frame.shape[1]
                width = frame.shape[0]
                pixels += height * width

                # Add the frame to the list if it belong to the random sampling list
                if i in self.random_sampler:
                    # Change color space to have only luminance
                    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)[:, :, 2]
                    # Resize the frame 
                    if frame.shape[0] != 1920:
                        frame_hd = cv2.resize(frame, (1920, 1080), interpolation=cv2.INTER_LINEAR)
                    else:
                        frame_hd = frame

                    frame_list_hd.append(frame_hd)

                    frame = cv2.resize(frame, (480, 270), interpolation=cv2.INTER_LINEAR)
                    frame_list.append(frame)
                    print('Frame size: {}, HD frame size: {}'.format(sys.getsizeof(frame), sys.getsizeof(frame_hd)), i)
                    print('Frame list size: {}, HD size: {}'.format(sys.getsizeof(frame_list), sys.getsizeof(frame_list_hd)), i)
                    print(psutil.virtual_memory())
            # Break the loop when frames cannot be taken from original
            else:
                break

        # Clean up memory
        capture.release()

        return np.array(frame_list), np.array(frame_list_hd), pixels, height, width

Answer 1

好的。解决了。 在此函数之后，在以下函数中调用创建的帧列表：

def compute(self, frame_list, frame_list_hd, path, dimensions, pixels):
        """
        Function to compare lists of numpy arrays extracting their corresponding metrics.
        It basically takes the global original list of frames and the input frame_list
        of numpy arrrays to extract the metrics defined in the constructor.
        frame_pos establishes the index of the frames to be compared.
        It is optimized by means of the ThreadPoolExecutor of Python's concurrent package
        for better parallel performance.
        """

        # Dictionary of metrics
        rendition_metrics = {}
        # Position of the frame
        frame_pos = 0
        # List of frames to be processed
        frames_to_process = []

        # Iterate frame by frame and fill a list with their values
        # to be passed to the ThreadPoolExecutor. Stop when maximum
        # number of frames has been reached.

        frames_to_process = range(len(frame_list)-1)
        print('computing')
        # Execute computations in parallel using as many processors as possible
        # future_list is a dictionary storing all computed values from each thread
        with ThreadPoolExecutor(max_workers=3) as executor:
            # Compare the original asset against its renditions
            future_list = {executor.submit(self.compare_renditions_instant,
                                           i,
                                           frame_list,
                                           frame_list_hd,
                                           dimensions,
                                           pixels,
                                           path): i for i in frames_to_process}

        # Once all frames in frame_list have been iterated, we can retrieve their values
        for future in future_list:
            # Values are retrieved in a dict, as a result of the executor's process
            result_rendition_metrics, frame_pos = future.result()
            # The computed values at a given frame

            rendition_metrics[frame_pos] = result_rendition_metrics

        # Return the metrics for the currently processed rendition
        return rendition_metrics

问题是，由于没有参数调用 ThreadPoolExecutor()，它使用了默认的工作程序数量（可用 CPU 数量的 5 倍，即 2）。这使许多帧对于内存来说太大了，因此使我的系统饱和。 假设每个线程都在输出自己的 psutil 数据，我就会被自己的踪迹误导。