将每个视频的帧作为 QImage

Question

我正在启动一个项目，该项目包括从 RTSP 服务器接收视频并使用 QT c++ 和 GSTreamer 在面板中显示它。我想接收每一帧作为 QImage 对象来运行我需要的一些功能。我知道我们可以使用 QTMultimedia 链接 Gstreamer 并查看视频，但在这种特殊情况下，我想要 QImages。我找到了一种方法（或者我认为我做到了）通过调用函数“gst_app_sink_pull_sample”从视频中获取样本。我们从中得到一个 GTSample。但是，我没有找到任何方法将其转换为例如易于转换为 QImage 的 JPG 原始数据。

我还找到了一种从 GSTSample 访问数据的方法：“How to get video stream frame-by-frame from Gstreamer pipeline? (without OpenCV)” 但是，再一次，我不知道如何将这些数据转换为 QImage。

 /* Initialize GStreamer */
        gst_init (NULL, NULL);

        /* Create the elements */
        GstElement * source = gst_element_factory_make ("videotestsrc", "source"); // HERE I'M USING THE GSTREAMER SOURCE TEST
        GstElement * sink = gst_element_factory_make ("appsink", "sink");
        GstAppSink *appsink = GST_APP_SINK(sink);

        /* Create the empty pipeline */
         GstElement * pipeline = gst_pipeline_new ("test-pipeline");

        if (!pipeline || !source || !sink) {
            g_printerr ("Not all elements could be created.\n");
            return ;
        }

        /* Build the pipeline */
        gst_bin_add_many (GST_BIN (pipeline), source, sink, NULL);
        if (gst_element_link (source, sink) != TRUE) {
            g_printerr ("Elements could not be linked.\n");
            gst_object_unref (pipeline);
            return ;
        }

        /* Modify the source's properties */
        g_object_set (source, "pattern", 0, NULL);


        /* Start playing */
        ret = gst_element_set_state (pipeline, GST_STATE_PLAYING);
        if (ret == GST_STATE_CHANGE_FAILURE) {
            g_printerr ("Unable to set the pipeline to the playing state.\n");
            gst_object_unref (pipeline);
            return;
        }


        // My question starts HERE
        GstSample *sample = nullptr;

        do{

            sample = gst_app_sink_pull_sample(appsink); // Get the frame
            if (!sample) {
                printf("sample is NULL\n");
            }else{
                
                // Get the raw data (or trying to...)
                GstBuffer * buffer = gst_sample_get_buffer(sample);
                GstMapInfo info;
                gst_buffer_map(buffer, &info, GST_MAP_READ);

                // Dumb way to check if the frame is being received properly
                QImage imageAux((const unsigned char*)info.data, 100, 100, QImage::Format_RGB16);
                imageAux.save("out.jpg"); //returns garbage

            }


        }while(sample);

Answer 1

据此topic

cv::Mat frame(cv::Size(width, height), CV_8UC4, (char *)map.data, cv::Mat::AUTOSTEP);

实际图像数据从map.data指针开始。并且在 OpenCV 方面具有 CV_8UC4 格式。 用下一个构造函数QImage(uchar *data, int width, int height, int bytesPerLine, QImage::Format format, ImageCleanupFunction cleanupFunction = nullptr, void *cleanupInfo = nullptr)构造QImage

您应该了解什么是 CV_8UC4 格式。并在QImage::Format 列表中找到合适的一个。并在该“映射”中找到其他参数以推送到构造函数中。在好的情况下，当您在QImage::Format 中找到合适的格式时，将map.data 推入uchar *data。在错误的情况下，您应该尝试找到另一种方法将 CV_8UC4 转换为适合 Qt 图像的东西。

此外，上述主题的作者可能会误认为 CV_8UC4。然后是map.data指针下的图像格式是什么的问题。但是，一旦您获得正确的格式，获取QImage 的步骤无论如何都是合适的。

Answer 2

来自 cv::imwrite 的结果

结果不好。上图是使用 cv::imwrite("some.jpg", frame); 将 Mat frame() 保存为图像的结果。它可能会返回这样的图像：videotestsrc image

有代码：

GstBuffer * buffer = gst_sample_get_buffer(sample);
GstMapInfo map;
GstCaps *caps = gst_sample_get_caps(sample);
GstStructure *structure = gst_caps_get_structure(caps, 0);
const int width = g_value_get_int(gst_structure_get_value(structure, "width"));
const int height = g_value_get_int(gst_structure_get_value(structure, "height"));

gst_buffer_map(buffer, &map, GST_MAP_READ);

cv::Mat frame(cv::Size(width, height),  CV_8UC3, (char *)map.data, cv::Mat::AUTO_STEP);

cv::imwrite("some.jpg", frame);

QImage imgIn= QImage((uchar*) frame.data, frame.cols, frame.rows, frame.step, QImage::Format_RGB888);
imgIn.save("out.jpg");

QImage 非常相似，但现在我更专注于尝试了解 v::imwrite("some.jpg", frame); 发生了什么。我确实尝试了几种帧格式，例如 CV_8UC4、CV_8UC1 等...

Answer 3

知道了！我找到了这个例子：Qt+GStreamer: How to take a snapshot while playing live video stream 它使用了 QT-Gstreamer 库，不幸的是，它在我的系统中不起作用。但是经过一些修改以便仅使用 Gstreamer，我可以得到一个 QImage。我错过了框架的格式。所以，我需要把它转换成已知格式(gst_caps_new_simple(...) )，最后再转换成QImage。

/* Initialize GStreamer */
gst_init (NULL, NULL);

/* Create the elements */
source = gst_element_factory_make ("videotestsrc", "source");
//sink = gst_element_factory_make ("autovideosink", "sink");
sink = gst_element_factory_make ("appsink", "sink");
GstAppSink *appsink = GST_APP_SINK(sink);

/* Create the empty pipeline */
pipeline = gst_pipeline_new ("test-pipeline");

if (!pipeline || !source || !sink) {
    g_printerr ("Not all elements could be created.\n");
    return ;
}

/* Build the pipeline */
gst_bin_add_many (GST_BIN (pipeline), source, sink, NULL);
if (gst_element_link (source, sink) != TRUE) {
    g_printerr ("Elements could not be linked.\n");
    gst_object_unref (pipeline);
    return ;
}

/* Modify the source's properties */
g_object_set (source, "pattern", 0, NULL);


// Start the ThreadVideoProcessor

/* Start playing */
ret = gst_element_set_state (pipeline, GST_STATE_PLAYING);
if (ret == GST_STATE_CHANGE_FAILURE) {
    g_printerr ("Unable to set the pipeline to the playing state.\n");
    gst_object_unref (pipeline);
    return;
}

GstSample *sample = nullptr;

do{

    sample = gst_app_sink_pull_sample(appsink);
    if (!sample) {
        printf("sample is NULL\n");
    }else{

        GstMapInfo map;
        GstCaps *caps = gst_sample_get_caps(sample);
        GError *err = NULL;

        GstStructure *structure = gst_caps_get_structure(caps, 0);
        const int width = g_value_get_int(gst_structure_get_value(structure, "width"));
        const int height = g_value_get_int(gst_structure_get_value(structure, "height"));

        GstCaps * capsTo = gst_caps_new_simple("video/x-raw",
                                               "format", G_TYPE_STRING, "RGB",
                                               "width", G_TYPE_INT, width,
                                               "height", G_TYPE_INT, height,
                                               NULL);


         GstSample *convertedSample = gst_video_convert_sample(sample,capsTo,GST_SECOND,&err);


        if (convertedSample == nullptr) {
            //qWarning() << "gst_video_convert_sample Failed:" << err->message;
        }
        else {
            //qDebug() << "Converted sample caps:" << convertedSample->caps()->toString();

            GstBuffer * buffer = gst_sample_get_buffer(convertedSample);
            gst_buffer_map(buffer, &map, GST_MAP_READ);

            QImage snapShot = QImage((const uchar *)map.data,
                              width,
                              height,
                              QImage::Format_RGB888);

            //qDebug() << "Saving snap to" << "out.jpg";
            snapShot.save("out.jpg");

    }

   }

执行后：