CUDA 程序将在特定节点上出现段错误

Question

我们正在尝试在我们的集群上运行cardioid，它可以在一个节点上毫无问题地运行，但是，在另一个节点上，程序会出现段错误。

两个节点配置相同，环境应该相同，因为/root、/home、/opt目录是NFS共享的，环境由Spack管理（安装在共享目录下）和驱动程序的版本相同。

使用cuda-memcheck，报错“设备内核映像无效”和“资源句柄无效”，不知道是什么原因导致这些问题（只发生在一个节点上）？

cuda-memcheck的输出

========= CUDA-MEMCHECK
========= Program hit CUDA_ERROR_INVALID_IMAGE (error 200) due to "device kernel image is invalid" on CUDA API call to cuModuleLoadDataEx.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/lib/x86_64-linux-gnu/libcuda.so.1 [0x2420f1]
=========     Host Frame:../../../build_user_debug/bin/cardioid (_ZN10BetterTT0612ThisReaction15constructKernelEv + 0xa25) [0x1a69d5]
=========     Host Frame:../../../build_user_debug/bin/cardioid (_Z28reactionFactoryForBetterTT06P9object_stdiRK10ThreadTeam + 0x15ac) [0x1ac22c]
=========     Host Frame:../../../build_user_debug/bin/cardioid (_Z15reactionFactoryRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEdiRK10ThreadTeam + 0x12c) [0x19e59c]
=========     Host Frame:../../../build_user_debug/bin/cardioid (_ZN15ReactionManager6createEd12ro_array_ptrIiERK10ThreadTeam + 0x14bd) [0x199bcd]
=========     Host Frame:../../../build_user_debug/bin/cardioid (_Z18initializeSimulateRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEER8Simulate + 0x1ae3) [0xb8033]
=========     Host Frame:../../../build_user_debug/bin/cardioid (main + 0x73f) [0xa73ff]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xeb) [0x2409b]
=========     Host Frame:../../../build_user_debug/bin/cardioid (_start + 0x2a) [0xa6bda]
=========
========= Program hit CUDA_ERROR_INVALID_HANDLE (error 400) due to "invalid resource handle" on CUDA API call to cuModuleGetFunction.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/lib/x86_64-linux-gnu/libcuda.so.1 [0x241b89]
=========     Host Frame:../../../build_user_debug/bin/cardioid (_ZN10BetterTT0612ThisReaction15constructKernelEv + 0xa42) [0x1a69f2]
=========     Host Frame:../../../build_user_debug/bin/cardioid (_Z28reactionFactoryForBetterTT06P9object_stdiRK10ThreadTeam + 0x15ac) [0x1ac22c]
=========     Host Frame:../../../build_user_debug/bin/cardioid (_Z15reactionFactoryRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEdiRK10ThreadTeam + 0x12c) [0x19e59c]
=========     Host Frame:../../../build_user_debug/bin/cardioid (_ZN15ReactionManager6createEd12ro_array_ptrIiERK10ThreadTeam + 0x14bd) [0x199bcd]
=========     Host Frame:../../../build_user_debug/bin/cardioid (_Z18initializeSimulateRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEER8Simulate + 0x1ae3) [0xb8033]
=========     Host Frame:../../../build_user_debug/bin/cardioid (main + 0x73f) [0xa73ff]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xeb) [0x2409b]
=========     Host Frame:../../../build_user_debug/bin/cardioid (_start + 0x2a) [0xa6bda]
=========
========= Program hit CUDA_ERROR_INVALID_HANDLE (error 400) due to "invalid resource handle" on CUDA API call to cuLaunchKernel.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/lib/x86_64-linux-gnu/libcuda.so.1 [0x20d695]
=========     Host Frame:../../../build_user_debug/bin/cardioid (_ZN10BetterTT0612ThisReaction4calcEd14ro_mgarray_ptrIiES1_IdES3_14wo_mgarray_ptrIdE + 0x255) [0x1a79e5]
=========     Host Frame:../../../build_user_debug/bin/cardioid (_ZN15ReactionManager4calcEd14ro_mgarray_ptrIdES1_14wo_mgarray_ptrIdE + 0xc5) [0x1983e5]
=========     Host Frame:../../../build_user_debug/bin/cardioid (_Z14simulationLoopR8Simulate + 0x450) [0xa8330]
=========     Host Frame:../../../build_user_debug/bin/cardioid (main + 0x80b) [0xa74cb]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xeb) [0x2409b]
=========     Host Frame:../../../build_user_debug/bin/cardioid (_start + 0x2a) [0xa6bda]
=========
========= ERROR SUMMARY: 3 errors

Answer 1

好的，仍然不确定为什么会出现这个问题，但我设法通过重新安装驱动程序和 cuda 来解决它。