【问题标题】:Nvidia device error in tensorflow张量流中的Nvidia设备错误
【发布时间】:2017-06-17 08:07:42
【问题描述】:

为了测试我的 tensorflow 安装,我使用了 tensorflow 存储库中提供的 mnist 示例,但是当我执行 convolutional.py 脚本时,我得到了以下输出:

    I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcublas.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcufft.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcurand.so.8.0 locally
Extracting data/train-images-idx3-ubyte.gz
Extracting data/train-labels-idx1-ubyte.gz
Extracting data/t10k-images-idx3-ubyte.gz
Extracting data/t10k-labels-idx1-ubyte.gz
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: 
name: GeForce GTX 980 Ti
major: 5 minor: 2 memoryClockRate (GHz) 1.2405
pciBusID 0000:03:00.0
Total memory: 5.93GiB
Free memory: 5.83GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:590] creating context when one is currently active; existing: 0x29020c0
E tensorflow/core/common_runtime/direct_session.cc:137] Internal: failed initializing StreamExecutor for CUDA device ordinal 1: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_INVALID_DEVICE
Traceback (most recent call last):
  File "convolutional.py", line 339, in <module>
    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 44, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "convolutional.py", line 284, in main
    with tf.Session() as sess:
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1187, in __init__
    super(Session, self).__init__(target, graph, config=config)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 552, in __init__
    self._session = tf_session.TF_NewDeprecatedSession(opts, status)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/errors_impl.py", line 469, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InternalError: Failed to create session.

我的第一个想法是,也许我在安装 cuda 时遇到了问题,但我使用为 nvidia 提供的一个示例进行了测试。在这种情况下,我使用了这个例子:

NVIDIA_CUDA-8.0_Samples/6_Advanced/c++11_cuda

输出是这样的:

GPU Device 0: "GeForce GTX 980 Ti" with compute capability 5.2

Read 3223503 byte corpus from ./warandpeace.txt
counted 107310 instances of 'x', 'y', 'z', or 'w' in "./warandpeace.txt"

那么我的结论是 cuda 安装正确。但我不知道这里发生了什么。如果有人可以帮助我,我将不胜感激。

有关更多信息,这是我的 gpu 配置:

Tue Jan 31 19:42:10 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.57                 Driver Version: 367.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 560 Ti  Off  | 0000:01:00.0     N/A |                  N/A |
| 25%   45C    P0    N/A /  N/A |    463MiB /   958MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 980 Ti  Off  | 0000:03:00.0     Off |                  N/A |
|  0%   31C    P8    13W / 280W |      1MiB /  6077MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0                  Not Supported                                         |
+-----------------------------------------------------------------------------+

编辑:

两张nvidia卡有相同的物理id是正常的吗?

sudo lshw -C "display"
  *-display               
       description: VGA compatible controller
       product: GM200 [GeForce GTX 980 Ti]
       vendor: NVIDIA Corporation
       physical id: 0
       bus info: pci@0000:03:00.0
       version: a1
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress vga_controller bus_master cap_list rom
       configuration: driver=nvidia latency=0
       resources: irq:50 memory:f9000000-f9ffffff memory:b0000000-bfffffff memory:c0000000-c1ffffff ioport:d000(size=128) memory:fa000000-fa07ffff
  *-display
       description: VGA compatible controller
       product: GF114 [GeForce GTX 560 Ti]
       vendor: NVIDIA Corporation
       physical id: 0
       bus info: pci@0000:01:00.0
       version: a1
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress vga_controller bus_master cap_list rom
       configuration: driver=nvidia latency=0
       resources: irq:45 memory:f6000000-f7ffffff memory:c8000000-cfffffff memory:d0000000-d3ffffff ioport:e000(size=128) memory:f8000000-f807ffff

【问题讨论】:

  • 回答您的编辑,他们没有相同的物理 id
  • 但是,在这两种情况下,物理 id:0。那么他们是否具有相同的物理ID?
  • pci 总线不同。您的编辑完全无关紧要
  • 是的,总线不同,但物理 id 没有不同,如果我没记错,物理 id 用于选择您要使用的 gpu。
  • 你错了。 cuda 中的设备枚举不依赖于 pci id。

标签: cuda tensorflow


【解决方案1】:

您显示的输出中的重点是:

I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: 
name: GeForce GTX 980 Ti
major: 5 minor: 2 memoryClockRate (GHz) 1.2405
pciBusID 0000:03:00.0
Total memory: 5.93GiB
Free memory: 5.83GiB

即您想要的计算设备被枚举为设备 0 和

E tensorflow/core/common_runtime/direct_session.cc:137] Internal: failed initializing StreamExecutor for CUDA device ordinal 1: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_INVALID_DEVICE

即产生错误的计算设备被枚举为设备 1。设备 1 是您的显示 GPU,它不能用于 Tensorflow 中的计算。如果您使用nvidia-smi 将该设备标记为禁止计算,或者使用CUDA_VISIBLE_DEVICES 环境变量仅使您的计算设备对CUDA 可见,则错误可能会消失。

【讨论】:

  • 使用 cuda 可见设备的解决方案正在运行,但在我的情况下使用 de nvidia-smi 的解决方案不起作用。我正在使用这个命令: sudo nvidia-smi -i $GPU_ID --compute-mode=PROHIBITED 但 tensorflow 它仍在尝试使用显示 gpu。但另一个解决方案运行良好。感谢您的帮助。
【解决方案2】:

当我尝试运行属于the image recognition tutorialclassify_image.py 脚本时遇到了类似的错误。由于我已经有一个正在运行的 Python 会话 (elpy),我在其中运行了一些 TensorFlow 代码,GPU 被分配在那里,因此对于我试图从 shell 运行的脚本来说是不可用的。

退出现有的 Python 会话解决了错误。

【讨论】:

    猜你喜欢
    • 2018-06-21
    • 2018-07-21
    • 2017-08-04
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2018-07-06
    • 2017-02-11
    相关资源
    最近更新 更多