如何改进我的 python openCV 视频流？

Question

我一直在做一个项目，我使用树莓派将实时视频源发送到我的服务器。这有点工作，但不是我想要的。 问题主要是速度。现在我可以以大约 3.5 FPS 的速度发送一个 640x480 的视频流，以大约 0.5 FPS 的速度发送一个 1920x1080 的视频流，这太糟糕了。由于我不是专业人士，我认为应该有一种方法可以改进我的代码。

发件人（树莓派）：

def send_stream():
    connection = True
    while connection:
        ret,frame = cap.read()
        if ret:
            # You might want to enable this while testing.
            # cv2.imshow('camera', frame)
            b_frame = pickle.dumps(frame)
            b_size = len(b_frame)
            try:
                s.sendall(struct.pack("<L", b_size) + b_frame)
            except socket.error:
                print("Socket Error!")
                connection = False

        else:
            print("Received no frame from camera, exiting.")
            exit()

接收方（服务器）：

    def recv_stream(self):
        payload_size = struct.calcsize("<L")
        data = b''
        while True:
            try:
                start_time = datetime.datetime.now()
                # keep receiving data until it gets the size of the msg.
                while len(data) < payload_size:
                    data += self.connection.recv(4096)
                # Get the frame size and remove it from the data.
                frame_size = struct.unpack("<L", data[:payload_size])[0]
                data = data[payload_size:]
                # Keep receiving data until the frame size is reached.
                while len(data) < frame_size:
                    data += self.connection.recv(32768)
                # Cut the frame to the beginning of the next frame.
                frame_data = data[:frame_size]
                data = data[frame_size:]

                frame = pickle.loads(frame_data)
                frame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)

                end_time = datetime.datetime.now()
                fps = 1/(end_time-start_time).total_seconds()
                print("Fps: ",round(fps,2))

                self.detect_motion(frame,fps)

                self.current_frame = frame

            except (socket.error,socket.timeout) as e:
                # The timeout got reached or the client disconnected. Clean up the mess.
                print("Cleaning up: ",e)
                try:
                    self.connection.close()
                except socket.error:
                    pass
                self.is_connected = False
                break

Answer 1

一个潜在的原因可能是读取帧时的 I/O 延迟。由于cv2.VideoCapture().read() 是一个阻塞操作，所以主程序会停止，直到从摄像头设备读取一帧并返回。提高性能的一种方法是生成另一个线程以并行 处理抓取帧，而不是依赖单个线程以顺序 顺序抓取帧。我们可以通过创建一个新线程来提高性能，该线程只轮询新帧，而主线程处理/绘制最近的帧。

您当前的方法（顺序）：

线程1：抓取框架->进程框架->绘图

建议的方法（并行）：

线程 1：抓取框架

from threading import Thread
import time

def get_frames():
    while True:
        ret, frame = cap.read()
        time.sleep(.01)

thread_frames = Thread(target=self.get_frames, args=())
thread_frames.daemon = True
thread_frames.start()

线程 2：进程框架-> Plot

def process_frames():
    while True:
        # Grab most recent frame
        # Process/plot frame
        ...

通过拥有单独的线程，您的程序将是并行的，因为总会有一个准备好处理的帧，而不是在处理可以完成之前等待一个帧被读入。

注意：此方法会在减少 I/O 延迟的基础上提高性能。这并不是真正的 FPS 增加，因为它延迟显着减少（一帧始终可用于处理；我们不需要轮询相机设备并等待 I/O去完成）。

Answer 2

在互联网上搜索了很长时间后，我找到了一个快速解决方案，可以将 fps 翻倍（这仍然太低：1.1 fps @1080p）。我所做的是我停止使用 pickle 并改用 base64。显然腌制图像只需要一段时间。无论如何，这是我的新代码：

发件人（树莓派）：

def send_stream():
global connected
connection = True
while connection:
    if last_frame is not None:

        # You might want to uncomment these lines while testing.
        # cv2.imshow('camera', frame)
        # cv2.waitKey(1)
        frame = last_frame

        # The old pickling method.
        #b_frame = pickle.dumps(frame)

        encoded, buffer = cv2.imencode('.jpg', frame)
        b_frame = base64.b64encode(buffer)

        b_size = len(b_frame)
        print("Frame size = ",b_size)
        try:
            s.sendall(struct.pack("<L", b_size) + b_frame)
        except socket.error:
            print("Socket Error!")
            connection = False
            connected = False
            s.close()
            return "Socket Error"
    else:
        return "Received no frame from camera"

接收方（服务器）：

    def recv_stream(self):
    payload_size = struct.calcsize("<L")
    data = b''
    while True:
        try:
            start_time = datetime.datetime.now()
            # keep receiving data until it gets the size of the msg.
            while len(data) < payload_size:
                data += self.connection.recv(4096)
            # Get the frame size and remove it from the data.
            frame_size = struct.unpack("<L", data[:payload_size])[0]
            data = data[payload_size:]
            # Keep receiving data until the frame size is reached.
            while len(data) < frame_size:
                data += self.connection.recv(131072)
            # Cut the frame to the beginning of the next frame.
            frame_data = data[:frame_size]
            data = data[frame_size:]

            # using the old pickling method.
            # frame = pickle.loads(frame_data)

            # Converting the image to be sent.
            img = base64.b64decode(frame_data)
            npimg = np.fromstring(img, dtype=np.uint8)
            frame = cv2.imdecode(npimg, 1)

            frame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)

            end_time = datetime.datetime.now()
            fps = 1/(end_time-start_time).total_seconds()
            print("Fps: ",round(fps,2))
            self.detect_motion(frame,fps)

            self.current_frame = frame

        except (socket.error,socket.timeout) as e:
            # The timeout got reached or the client disconnected. Clean up the mess.
            print("Cleaning up: ",e)
            try:
                self.connection.close()
            except socket.error:
                pass
            self.is_connected = False
            break

我还增加了数据包大小，这在测试时从本地机器发送到本地机器时提高了 fps，但是在使用树莓派时这并没有改变任何东西。

你可以在我的github上看到完整的代码：https://github.com/Ruud14/SecurityCamera