如何在 Windows 上从 ffmpeg 获取实时视频帧和时间戳到 python

Question

搜索 OpenCV 的替代方案不会提供 live 相机流的时间戳（在 Windows 上），这是我的计算机视觉算法所必需的，我发现 ffmpeg 和这个优秀文章https://zulko.github.io/blog/2013/09/27/read-and-write-video-frames-in-python-using-ffmpeg/ 该解决方案使用 ffmpeg，访问其标准输出 (stdout) 流。我也将它扩展为读取标准错误 (stderr) 流。

在 Windows 上处理 python 代码，而我从 ffmpeg 标准输出接收到视频帧，但标准错误在提供第一帧的 showinfo 视频过滤器详细信息（时间戳）后冻结。

我记得在某处的 ffmpeg 论坛上看到，像 showinfo 这样的视频过滤器在重定向时会被绕过。这就是为什么以下代码无法按预期工作的原因吗？

预期：它应该将视频帧写入磁盘并打印时间戳详细信息。
实际：写入视频文件但不获取时间戳（showinfo）详细信息。

这是我尝试过的代码：

import subprocess as sp
import numpy
import cv2

command = [ 'ffmpeg', 
            '-i', 'e:\sample.wmv',
            '-pix_fmt', 'rgb24',
            '-vcodec', 'rawvideo',
            '-vf', 'showinfo', # video filter - showinfo will provide frame timestamps
            '-an','-sn', #-an, -sn disables audio and sub-title processing respectively
            '-f', 'image2pipe', '-'] # we need to output to a pipe

pipe = sp.Popen(command, stdout = sp.PIPE, stderr = sp.PIPE) # TODO someone on ffmpeg forum said video filters (e.g. showinfo) are bypassed when stdout is redirected to pipes??? 

for i in range(10):
    raw_image = pipe.stdout.read(1280*720*3)
    img_info = pipe.stderr.read(244) # 244 characters is the current output of showinfo video filter
    print "showinfo output", img_info
    image1 =  numpy.fromstring(raw_image, dtype='uint8')
    image2 = image1.reshape((720,1280,3))  

    # write video frame to file just to verify
    videoFrameName = 'Video_Frame{0}.png'.format(i)
    cv2.imwrite(videoFrameName,image2)

    # throw away the data in the pipe's buffer.
    pipe.stdout.flush()
    pipe.stderr.flush()

那么如何仍然将帧时间戳从 ffmpeg 获取到 python 代码中，以便可以在我的计算机视觉算法中使用...

Answer 1

重定向 stderr 在 python 中有效。
所以不是这个pipe = sp.Popen(command, stdout = sp.PIPE, stderr = sp.PIPE)
这样做pipe = sp.Popen(command, stdout = sp.PIPE, stderr = sp.STDOUT)

我们可以通过添加异步调用来读取 ffmpeg 的标准流（stdout 和 stderr）来避免重定向。这将避免视频帧和时间戳的任何混合，从而避免容易出错的分离。 因此修改原始代码以使用threading 模块将如下所示：

# Python script to read video frames and timestamps using ffmpeg
import subprocess as sp
import threading

import matplotlib.pyplot as plt
import numpy
import cv2

ffmpeg_command = [ 'ffmpeg',
                   '-nostats', # do not print extra statistics
                    #'-debug_ts', # -debug_ts could provide timestamps avoiding showinfo filter (-vcodec copy). Need to check by providing expected fps TODO
                    '-r', '30', # output 30 frames per second
                    '-i', 'e:\sample.wmv',
                    '-an','-sn', #-an, -sn disables audio and sub-title processing respectively
                    '-pix_fmt', 'rgb24',
                    '-vcodec', 'rawvideo', 
                    #'-vcodec', 'copy', # very fast!, direct copy - Note: No Filters, No Decode/Encode, no quality loss
                    #'-vframes', '20', # process n video frames only. For Debugging
                    '-vf', 'showinfo', # showinfo videofilter provides frame timestamps as pts_time
                    '-f', 'image2pipe', 'pipe:1' ] # outputs to stdout pipe. can also use '-' which is redirected to pipe


# seperate method to read images on stdout asynchronously
def AppendProcStdout(proc, nbytes, AppendList):
    while proc.poll() is None: # continue while the process is alive
        AppendList.append(proc.stdout.read(nbytes)) # read image bytes at a time

# seperate method to read image info. on stderr asynchronously
def AppendProcStderr(proc, AppendList):
    while proc.poll() is None: # continue while the process is alive
        try: AppendList.append(proc.stderr.next()) # read stderr until empty
        except StopIteration: continue # ignore stderr empty exception and continue


if __name__ == '__main__':
    # run ffmpeg command
    pipe = sp.Popen(ffmpeg_command, stdout=sp.PIPE, stderr=sp.PIPE) 

    # 2 threads to talk with ffmpeg stdout and stderr pipes
    framesList = [];
    frameDetailsList = []
    appendFramesThread = threading.Thread(group=None, target=AppendProcStdout, name='FramesThread', args=(pipe, 1280*720*3, framesList), kwargs=None, verbose=None) # assuming rgb video frame with size 1280*720 
    appendInfoThread = threading.Thread(group=None, target=AppendProcStderr, name='InfoThread', args=(pipe, frameDetailsList), kwargs=None, verbose=None) 

    # start threads to capture ffmpeg frames and info.
    appendFramesThread.start()
    appendInfoThread.start()

    # wait for few seconds and close - simulating cancel
    import time; time.sleep(2) 
    pipe.terminate() 

    # check if threads finished and close
    appendFramesThread.join() 
    appendInfoThread.join() 

    # save an image per 30 frames to disk 
    savedList = []
    for cnt,raw_image in enumerate(framesList):
        if (cnt%30 != 0): continue
        image1 =  numpy.fromstring(raw_image, dtype='uint8')
        image2 = image1.reshape((720,1280,3))  # assuming rgb image with size 1280 X 720
        # write video frame to file just to verify
        videoFrameName = 'video_frame{0}.png'.format(cnt)
        cv2.imwrite(videoFrameName,image2)
        savedList.append('{} {}'.format(videoFrameName, image2.shape))

    print '### Results ###'
    print 'Images captured: ({}) \nImages saved to disk:{}\n'.format(len(framesList), savedList) # framesList contains all the video frames got from the ffmpeg
    print 'Images info captured: \n', ''.join(frameDetailsList) # this contains all the timestamp details got from the ffmpeg showinfo videofilter and some initial noise text which can be easily removed while parsing

Answer 2

你可以使用MoviePy:

import moviepy.editor as mpy

vid = mpy.VideoFileClip('e:\\sample.wmv')
for timestamp, raw_img in vid.iter_frames(with_times=True):
    # do stuff

Answer 3

您可以尝试指定缓冲区大小，以确保整个帧都适合它：

bufsize = w*h*3 + 100 
pipe = sp.Popen(command, bufsize=bufsize, stdout = sp.PIPE, stderr = sp.PIPE)

通过此设置，您通常可以在 pipe.stdout 上阅读框架信息，在 pipe.stderr 上阅读其信息