cudaErrorCudartUnloading（错误 29）由于“驱动程序关闭”

Question

我最近从github repo 安装了opencv，并在CUDA 支持下进行了编译。我计划将公司框架从 CPU 版本更新到 GPU 版本，但我遇到了一个反复出现的问题：

========= Program hit cudaErrorCudartUnloading (error 29) due to "driver shutting down" on CUDA API call to cudaStreamSynchronize. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2eeda3]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcudart.so.7.5 (cudaStreamSynchronize + 0x15e) [0x421ce]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0x29a909]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 (_ZN2cv4cuda6Stream4ImplD1Ev + 0x6e) [0x29c0ee]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0x29c12a]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0x29a886]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0x29d08e]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__cxa_finalize + 0x8f) [0x37eaf]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0xbb033]
=========
========= Program hit cudaErrorCudartUnloading (error 29) due to "driver shutting down" on CUDA API call to cudaSetDevice. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2eeda3]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcudart.so.7.5 (cudaSetDevice + 0x180) [0x2fa30]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0x29d0fd]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__cxa_finalize + 0x8f) [0x37eaf]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0xbb033]
=========
========= Program hit cudaErrorCudartUnloading (error 29) due to "driver shutting down" on CUDA API call to cudaFree. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2eeda3]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcudart.so.7.5 (cudaFree + 0x186) [0x3c666]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0x29d0cd]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__cxa_finalize + 0x8f) [0x37eaf]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0xbb033]
=========
========= Program hit cudaErrorCudartUnloading (error 29) due to "driver shutting down" on CUDA API call to cudaSetDevice. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2eeda3]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcudart.so.7.5 (cudaSetDevice + 0x180) [0x2fa30]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0x29d0fd]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__cxa_finalize + 0x8f) [0x37eaf]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0xbb033]
=========
========= LEAK SUMMARY: 0 bytes leaked in 0 allocations
========= ERROR SUMMARY: 4 errors

这是什么，或者我如何追踪它？ 从终端运行应用程序时，我没有收到任何错误、异常或分段错误。

编辑：

#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/cudaimgproc.hpp>
#include <opencv2/core/cuda.hpp>
#include <opencv2/cudaobjdetect.hpp>
#include <opencv2/cudafeatures2d.hpp>    
#include <iostream>
#include <stdio.h>
#include <chrono>

int detectAndDisplay(
                     cv::Mat frame, 
                     cv::Ptr<cv::cuda::CascadeClassifier> & cascade_gpu
                    )
{
    cv::cuda::GpuMat image_src, image_grey;
    std::vector<cv::Rect> faces;
    image_src.upload(frame);
    cv::cuda::cvtColor(image_src, image_grey, CV_BGR2GRAY);
    cv::cuda::equalizeHist(image_grey, image_grey);
    cascade_gpu->setFindLargestObject(true);
    cascade_gpu->setScaleFactor(1.23);
    cascade_gpu->setMinNeighbors(3);
    cascade_gpu->setMinObjectSize(cv::Size(0, 0));
    cascade_gpu->setMaxObjectSize(image_src.size());
    cv::cuda::GpuMat buffer;
    cascade_gpu->detectMultiScale(image_grey, buffer);
    cascade_gpu->convert(buffer, faces);
    return faces.size();
}

int main( int argc, const char* argv[] )
{
    cv::String face_cascade_name = "/usr/share/opencv/haarcascades/haarcascade_frontalface_alt.xml";
    cv::Mat image;
    image = cv::imread("catniss.jpeg", CV_LOAD_IMAGE_COLOR);
    cv::Ptr<cv::cuda::CascadeClassifier> cascade_gpu;
    cascade_gpu = cv::cuda::CascadeClassifier::create(face_cascade_name);
    for (int j = 0; j < 100; j++) {
        detectAndDisplay(image, cascade_gpu); 
    }
    return 0;
}

为了您的方便，一个最小的CMakeLists.txt：

cmake_minimum_required(VERSION 2.6)
project(face_detection)
add_executable(face_detection source/main.cpp)
find_package(OpenCV REQUIRED)
target_link_libraries(face_detection  ${OpenCV_LIBS})
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}-std=c++1y")

遵循典型的构建阶段，然后运行

cuda-memcheck --leak-check full ./face_detection

我有一个 Debian jessie-backports 安装的 CUDA 7.5 和各自的驱动程序，在两个 GTX 660 上运行。

Answer 1

您能否向我们展示您的代码或做一个简短的示例？没有它就很难指出你的错误。

出现此错误消息是因为您在卸载运行时后使用了一些 cudaCalls。

如果您将 GpuMat 用作全局变量或静态变量，就会发生这种情况。

您可以忽略这些警告，或确保在调用应用程序的析构函数之前完成所有 cuda 运行时调用。