检查失败：训练 SSD 期间的错误 == cudaSuccess

Question

我正在训练SSD，我有错误

I0116 13:10:31.206343  3447 net.cpp:761] Ignoring source layer drop6
I0116 13:10:31.207219  3447 net.cpp:761] Ignoring source layer drop7
I0116 13:10:31.207229  3447 net.cpp:761] Ignoring source layer fc8
I0116 13:10:31.207233  3447 net.cpp:761] Ignoring source layer prob
F0116 13:10:31.227303  3447 parallel.cpp:130] Check failed: error == cudaSuccess (10 vs. 0)  invalid device ordinal
*** Check failure stack trace: ***
    @     0x7f158382e5cd  google::LogMessage::Fail()
    @     0x7f1583830433  google::LogMessage::SendToLog()
    @     0x7f158382e15b  google::LogMessage::Flush()
    @     0x7f1583830e1e  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f158412f7bd  caffe::DevicePair::compute()
    @     0x7f15841354e0  caffe::P2PSync<>::Prepare()
    @     0x7f1584135fee  caffe::P2PSync<>::Run()
    @           0x40af10  train()
    @           0x407608  main
    @     0x7f1581fbd830  __libc_start_main
    @           0x407ed9  _start
    @              (nil)  (unknown)
Aborted (core dumped)

我的图形是 Quadro4200。 ./deviceQuery 给我

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "Quadro K4200"
  CUDA Driver Version / Runtime Version          9.0 / 8.0
  CUDA Capability Major/Minor version number:    3.0
  Total amount of global memory:                 4034 MBytes (4230479872 bytes)
  ( 7) Multiprocessors, (192) CUDA Cores/MP:     1344 CUDA Cores
  GPU Max Clock rate:                            784 MHz (0.78 GHz)
  Memory Clock rate:                             2700 Mhz
  Memory Bus Width:                              256-bit
  L2 Cache Size:                                 524288 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 4 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 9.0, CUDA Runtime Version = 8.0, NumDevs = 1, Device0 = Quadro K4200
Result = PASS

我可以成功测试 SSD 库，只是我在训练中有错误。

那张显卡不够强大，无法训练库吗？

Answer 1

我发现了错误。

如果我们在ssd中运行python examples/ssd/ssd_pascal.py这个命令，下一步训练命令如下。

gdb --args ./build/tools/caffe train --solver="models/VGGNet/VOC0712/SSD_300x300/solver.prototxt" --weights="models/VGGNet/VGG_ILSVRC_16_layers_fc_reduced.caffemodel" --gpu 0,1,2,3 2>&1 | tee jobs/VGGNet/VOC0712/SSD_300x300/VGG_VOC0712_SSD_300x300.log

这个--gpu 0,1,2,3 2>&1 给出了问题。我改为 --gpu 0 并直接从训练命令运行

./build/tools/caffe train --solver="models/VGGNet/VOC0712/SSD_300x300/solver.prototxt" --weights="models/VGGNet/VGG_ILSVRC_16_layers_fc_reduced.caffemodel" --gpu 0 | tee jobs/VGGNet/VOC0712/SSD_300x300/VGG_VOC0712_SSD_300x300.log,

然后就解决了。