Opencv RTSP相机缓冲滞后

Question

我很难理解为什么我无法从我的 IP 摄像头获得“实时”提要。

似乎有一个缓冲区，如果没有被读取，它会导致帧建立 - 而且由于我的代码的每次迭代都需要一些时间，所以会出现积压，最终对实际发生的事情来说几乎是缓慢的。

我发现下面的代码会触发一个线程在循环中读取相机以尝试避免这种情况。但现在我得到了大约 5 帧的“实时”提要，然后它停止并在另外几帧中显示相同的图像。

##camera class - this stops the RTSP feed getting caught in the buffer 

class Camera:

    def __init__(self, rtsp_link):

        #init last ready and last frame
        self.last_frame = None
        self.last_ready = None
        self.lock = Lock()

        #set capture decive
        capture = cv2.VideoCapture(rtsp_link,apiPreference=cv2.CAP_FFMPEG)

        #set thread to clear buffer
        thread = threading.Thread(target=self.rtsp_cam_buffer, args=(capture,), name="rtsp_read_thread")
        thread.daemon = True
        thread.start()

        #delay start of next step to avoid errors
        time.sleep(2)

    def rtsp_cam_buffer(self, capture):
        #loop forever 
        while True:
            with self.lock:           
                capture.grab()
                self.last_ready, self.last_frame = capture.retrieve()


    def getFrame(self):        
        #get last frame
        if (self.last_ready is not None) and (self.last_frame is not None):
            return self.last_frame.copy())
        else:
            return None

在这种情况下正确的做法是什么？有没有办法解决这个问题？

或

我应该使用 gstreamer 或 ffmpeg 之类的东西来获取相机源吗？如果是这样，哪个更好，为什么？有什么建议或页面可以给我一些让它工作的python示例吗？我找不到对我有意义的大量内容。

谢谢

Answer 1

在线搜索多个资源后，使用线程从缓冲区中删除帧的建议出现了很多。尽管它似乎工作了一段时间，但由于某种我无法解决的原因，它导致我出现重复帧显示的问题。

然后我尝试从支持 gstreamer 的源代码构建 opencv，但即使正确编译，它似乎仍然不喜欢与 gstreamer 正确连接。

最终，我认为最好的办法是返回线程方法，但又无法使其正常工作。所以我尝试了多处理。

我编写了以下类来处理相机连接：

import cv2
import time
import multiprocessing as mp

class Camera():
    
    def __init__(self,rtsp_url):        
        #load pipe for data transmittion to the process
        self.parent_conn, child_conn = mp.Pipe()
        #load process
        self.p = mp.Process(target=self.update, args=(child_conn,rtsp_url))        
        #start process
        self.p.daemon = True
        self.p.start()
        
    def end(self):
        #send closure request to process
        
        self.parent_conn.send(2)
        
    def update(self,conn,rtsp_url):
        #load cam into seperate process
        
        print("Cam Loading...")
        cap = cv2.VideoCapture(rtsp_url,cv2.CAP_FFMPEG)   
        print("Cam Loaded...")
        run = True
        
        while run:
            
            #grab frames from the buffer
            cap.grab()
            
            #recieve input data
            rec_dat = conn.recv()
            
            
            if rec_dat == 1:
                #if frame requested
                ret,frame = cap.read()
                conn.send(frame)
                
            elif rec_dat ==2:
                #if close requested
                cap.release()
                run = False
                
        print("Camera Connection Closed")        
        conn.close()
    
    def get_frame(self,resize=None):
        ###used to grab frames from the cam connection process
        
        ##[resize] param : % of size reduction or increase i.e 0.65 for 35% reduction  or 1.5 for a 50% increase
             
        #send request
        self.parent_conn.send(1)
        frame = self.parent_conn.recv()
        
        #reset request 
        self.parent_conn.send(0)
        
        #resize if needed
        if resize == None:            
            return frame
        else:
            return self.rescale_frame(frame,resize)
        
    def rescale_frame(self,frame, percent=65):
        
        return cv2.resize(frame,None,fx=percent,fy=percent) 

显示框架可以如下完成

cam = Camera("rtsp://admin:[somepassword]@192.168.0.40/h264Preview_01_main")

print(f"Camera is alive?: {cam.p.is_alive()}")

while(1):
    frame = cam.get_frame(0.65)
    
    cv2.imshow("Feed",frame)
    
    key = cv2.waitKey(1)

    if key == 13: #13 is the Enter Key
        break

cv2.destroyAllWindows()     

cam.end()

这个解决方案解决了我所有的缓冲滞后和重复帧问题。 #

希望它能帮助遇到同样情况的其他人。

Answer 2

到目前为止，Lewis 的解决方案有助于减少延迟，但在我的情况下仍然存在一些延迟，我发现了这个 gist，它更快一点：

import os
import sys
import time
import threading
import numpy as np
import cv2 as cv

# also acts (partly) like a cv.VideoCapture
class FreshestFrame(threading.Thread):
    def __init__(self, capture, name='FreshestFrame'):
        self.capture = capture
        assert self.capture.isOpened()

        # this lets the read() method block until there's a new frame
        self.cond = threading.Condition()

        # this allows us to stop the thread gracefully
        self.running = False

        # keeping the newest frame around
        self.frame = None

        # passing a sequence number allows read() to NOT block
        # if the currently available one is exactly the one you ask for
        self.latestnum = 0

        # this is just for demo purposes        
        self.callback = None
        
        super().__init__(name=name)
        self.start()

    def start(self):
        self.running = True
        super().start()

    def release(self, timeout=None):
        self.running = False
        self.join(timeout=timeout)
        self.capture.release()

    def run(self):
        counter = 0
        while self.running:
            # block for fresh frame
            (rv, img) = self.capture.read()
            assert rv
            counter += 1

            # publish the frame
            with self.cond: # lock the condition for this operation
                self.frame = img if rv else None
                self.latestnum = counter
                self.cond.notify_all()

            if self.callback:
                self.callback(img)

    def read(self, wait=True, seqnumber=None, timeout=None):
        # with no arguments (wait=True), it always blocks for a fresh frame
        # with wait=False it returns the current frame immediately (polling)
        # with a seqnumber, it blocks until that frame is available (or no wait at all)
        # with timeout argument, may return an earlier frame;
        #   may even be (0,None) if nothing received yet

        with self.cond:
            if wait:
                if seqnumber is None:
                    seqnumber = self.latestnum+1
                if seqnumber < 1:
                    seqnumber = 1
                
                rv = self.cond.wait_for(lambda: self.latestnum >= seqnumber, timeout=timeout)
                if not rv:
                    return (self.latestnum, self.frame)

            return (self.latestnum, self.frame)

然后你像这样使用它：

# open some camera
    cap = cv.VideoCapture('rtsp://URL')
    cap.set(cv.CAP_PROP_FPS, 30)

    # wrap it
    fresh = FreshestFrame(cap)

用fresh应对开着的相机