在 Mac OS X 10.10.2 上使用 waitKey 的 opencv imshow 太慢（使用 c++）

Question

我在 Mac OS X 10.10.2 上使用 opencv c++ 来处理视频帧并显示它们。带waitKey的imshow显示视频的性能极慢。

我有以下代码可以正确显示高清 (1920x1080) 灰度帧，只是它运行速度太慢了大约 10 倍（即每秒 2 到 3 帧而不是每秒 30 帧）。

          cv::Mat framebuf[TEST_COUNT];

    //--- Code here to allocate and fill the frame buffer with about 4 seconds of video. This part works correctly.

        //--- This loop runs too slow by factor of approximately 10x
    for (int f = 0; f < TEST_COUNT; f++)
    {
        cv::imshow(windowName, framebuf[f]);
        cv::waitKey(33);
    }

谁能建议如何从 opencv imshow() 获得实时或接近实时的性能？我看到很多帖子说他们正在实时显示视频，甚至比实时更快，所以我不确定我做错了什么。任何帮助将不胜感激。

Answer 1

OpenCV 4 已解决此问题，请更新到新版本。

还有一件事，处理视频并在两个线程中显示视频。

#include <stdio.h>
#include <iostream>

#include <opencv2/opencv.hpp>
#include <dispatch/dispatch.h>

using namespace cv;
using namespace std;


bool newFrame = false;
Mat back_frame;

int opencvmain(int argc, char** argv ) {

    // open camear
    cv::VideoCapture cap;
    cap.open(0);
    if (!cap.isOpened()) {
        std::cout << "Couldn't open camera 0." << std::endl;
        return EXIT_FAILURE;
    }

    // define frame images
    cv::Mat frame;

    // frame loop
    for (;;) {

        // get video frame
        cap >> frame;
        if (frame.empty()) {
            break;
        }

        // render
        back_frame = frame.clone();
        newFrame = true;        
    }
    return 0;
}

int main(int argc, char** argv ) {


    namedWindow("video", WINDOW_AUTOSIZE );

    dispatch_queue_t opencvq = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0);
    dispatch_async(opencvq, ^{
        opencvmain(argc, argv);
    });


    while(true) {
        usleep(3*1000);
        if(newFrame) {                
            imshow("video", back_frame);
            auto key = cv::waitKey(1);
            if (key == ' ') {
                break;
            }
            newFrame = false;                
        }
    }

    return 0;
}

Answer 2

您可以创建自己的窗口来显示图像。将 MyWindow.m MyWindow.h 文件添加到项目中。

MyWindow.h

#ifndef MY_WINDOW_H
#define MY_WINDOW_H

#ifdef __cplusplus
extern "C" {
#endif
    void* createNSWindow(int x, int y, int w, int h);
    void renderNSWindow(void* inwindow, void* data, int w, int h, int c);
    void processNSEvent();
#ifdef __cplusplus
}
#endif

#endif

用法，在main.cpp中，别忘了waitKey

#include "MyWindow.h"

// need create a cv window and do nothing
cv::namedWindow("xxx", 1);

// create window
void* w = createNSWindow(0, 0, 0, 0);

// frame image
cv::Mat frameImage;

// render loop
renderNSWindow(w, frameImage.data, frameImage.cols, frameImage.rows, frameImage.channels());

// need waitKey to display window
processNSEvent();

在 MyWindow.m 中实现 delete import "MyWindow.h"

#import <Cocoa/Cocoa.h>

@interface MyWindow : NSWindow
@property(nonatomic, strong) NSImageView *imgv;
@end

@implementation MyWindow
@end


static NSImage* _createNSImage(void* data, int w, int h, int c);

void* createNSWindow(int x, int y, int w, int h) {

    NSRect screenFrame = [[NSScreen mainScreen] frame];
    NSRect frame = NSMakeRect(x, y, w, h);
    if (w == 0 || h == 0) {
        frame = screenFrame;
    }

    MyWindow* window  = [[MyWindow alloc] initWithContentRect:frame
                                                    styleMask:NSWindowStyleMaskBorderless
                                                      backing:NSBackingStoreBuffered
                                                        defer:NO] ;


    //_initApp(window);

    [window makeKeyAndOrderFront:NSApp];
    window.titleVisibility = TRUE;
    window.styleMask = NSWindowStyleMaskResizable | NSWindowStyleMaskTitled |NSWindowStyleMaskFullSizeContentView;

    window.imgv = [[NSImageView alloc] initWithFrame:NSMakeRect(0, 0, frame.size.width, frame.size.height)];
    [window.contentView addSubview:window.imgv];


    return (void*)CFBridgingRetain(window);
}


static NSImage* _createNSImage(void* data, int w, int h, int c) {

    size_t bufferLength = w * h * c;

    CGDataProviderRef provider = CGDataProviderCreateWithData(NULL, data, bufferLength, NULL);
    size_t bitsPerComponent = 8;
    size_t bitsPerPixel = c * bitsPerComponent;
    size_t bytesPerRow = c * w;
    CGColorSpaceRef colorSpaceRef = CGColorSpaceCreateDeviceRGB();
    CGBitmapInfo bitmapInfo = kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedLast;
    if (c < 4) {
        bitmapInfo = kCGBitmapByteOrderDefault | kCGImageAlphaNone;
        unsigned char* buf = data;
        for(int i = 0; i < w*h; i++) {
            unsigned char temp = buf[i*c];
            buf[i*c] = buf[i*c+c-1];
            buf[i*c+c-1] = temp;
        }
    }
    CGColorRenderingIntent renderingIntent = kCGRenderingIntentDefault;

    CGImageRef iref = CGImageCreate(w,
                                    h,
                                    bitsPerComponent,
                                    bitsPerPixel,
                                    bytesPerRow,
                                    colorSpaceRef,
                                    bitmapInfo,
                                    provider,   // data provider
                                    NULL,       // decode
                                    YES,        // should interpolate
                                    renderingIntent);

    NSImage* image = [[NSImage alloc] initWithCGImage:iref size:NSMakeSize(w, h)];
    return image;
}

void renderNSWindow(void* inwindow, void* data, int w, int h, int c) {
    MyWindow* window = (__bridge MyWindow*)inwindow;

    window.imgv.image = _createNSImage(data, w, h, c);

}

void processNSEvent() {
    for (;;)
    {
        NSEvent* event = [NSApp nextEventMatchingMask:NSEventMaskAny
                                            untilDate:[NSDate distantPast]
                                               inMode:NSDefaultRunLoopMode
                                              dequeue:YES];
        if (event == nil)
            break;

        [NSApp sendEvent:event];
    }
}

其他的，waitKey现在需要20ms左右，你可以在后台线程做OpenCV，在主线程显示窗口。也使用 processNSEvent 而不是 waitKey 只需要大约 10 毫秒。

完整源代码：

#include <iostream>
#include "opencv2/core/core.hpp"
#include "opencv2/highgui/highgui.hpp"
#include <dispatch/dispatch.h>
#include "MyWindow.h"


using namespace std;
using namespace cv;


int opencvfunc(int argc, const char *argv[]);
bool newFrame = false;
cv::Mat back_frame;

int main(int argc, const char * argv[]) {
    cv::namedWindow("render", 1);

    void* w = createNSWindow(0, 0, 0, 0);

    dispatch_queue_t opencvq = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_LOW, 0);
    dispatch_async(opencvq, ^{
        opencvfunc(argc, argv);
    });


    while(true) {
        usleep(3*1000);
        if(newFrame) {
            std::chrono::system_clock::time_point starttime = std::chrono::system_clock::now();
            renderNSWindow(w, back_frame.data, back_frame.cols, back_frame.rows, back_frame.channels());
            newFrame = false;
            //auto key = cv::waitKey(1);
            //if (key == 'q') {
            //    break;
            //}
            processNSEvent();
            std::chrono::system_clock::time_point endtime = std::chrono::system_clock::now();
            std::cout << "imshow:" << std::chrono::duration_cast<std::chrono::duration<double>>(endtime-starttime).count()*1000 << std::endl;
        }
    }

    return 0;
}

int opencvfunc(int argc, const char *argv[]) {

    cv::VideoCapture cap;
    cap.open(0);
    if (!cap.isOpened()) {
        std::cout << "Couldn't open camera 0." << endl;
        return EXIT_FAILURE;
    }

    Mat frame, unmodified_frame;

    for (;;) {

        cap >> frame; // get a new frame from camera
        if (frame.empty()) { // stop if we're at the end of the video
            break;
        }


        //unmodified_frame = frame.clone();

        // ...

        back_frame = frame.clone();
        newFrame = true;
    }

    return EXIT_SUCCESS;
}

Answer 3

我可能错了，但对我来说，问题不在于您的代码，而在于您的操作系统/配置。我写了一个小测试：

import cv2
import numpy as np
from random import randrange
img = np.zeros((1920, 1080), dtype = np.uint8)
counter = 0
while counter < 1000:
    cv2.line(img, (randrange(0, 1920), randrange(0, 1080)), (randrange(0, 1920), randrange(0, 1080)), (randrange(0, 255)))
    cv2.imshow('test', img)
    temp = cv2.waitKey(1)
    counter += 1
    print counter

在我的机器上（Core 2 duo 2,6Ghz x64、8gb ram、ssd）完成此测试大约需要 30 秒。运行它，如果你得到的时间比你的笔记本电脑/opencv 配置/等肯定有问题。我在 Mac OS X 上使用过 OpenCV 2.4.x（我认为是 10.9）并且运行良好。重新安装 OpenCV 是我想到的最明显的解决方案。当您删除 OpenCV 时，使用 brew 再次安装它 - brew install opencv --with-tbb --with-python --with-ffpmeg（或类似的东西 - 使用 brew options opencv 检查）应该没问题。第一个选项告诉 brew 用 tbb 构建 opencv（线程构建块 - 多线程库，有时可以显着提高速度），第二个是安装 python 包装器，最后一个是安装 ffmpeg（处理编解码器等）。

Answer 4

您必须减少对功能等待键的输入。尝试使用 2-5 范围内的较小数字。这也取决于您同时运行的其他进程，尝试关闭其他进程，看看是否有所改善