【问题标题】:CUDA_ERROR_INVALID DEVICE with keras=2.0.5 and tensorflow-gpu=1.2.1具有 keras=2.0.5 和 tensorflow-gpu=1.2.1 的 CUDA_ERROR_INVALID DEVICE
【发布时间】:2018-12-13 14:38:00
【问题描述】:

我正在使用这些特定规格:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130                Driver Version: 384.130                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:05:00.0 Off |                    0 |
| N/A   62C    P0   101W / 149W |  10912MiB / 11439MiB |    100%   E. Process |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           Off  | 00000000:06:00.0 Off |                    0 |
| N/A   39C    P0    72W / 149W |  10919MiB / 11439MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   2  Tesla K80           Off  | 00000000:84:00.0 Off |                    0 |
| N/A   50C    P0    57W / 149W |  10919MiB / 11439MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   3  Tesla K80           Off  | 00000000:85:00.0 Off |                    0 |
| N/A   42C    P0    69W / 149W |  10919MiB / 11439MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+

使用 Python 3.6、CUDA 8:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Tue_Jan_10_13:22:03_CST_2017
Cuda compilation tools, release 8.0, V8.0.61

还有 CUDNN 5.1.10:

#define CUDNN_MAJOR      5
#define CUDNN_MINOR      1
#define CUDNN_PATCHLEVEL 10
--
#define CUDNN_VERSION    (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)

我想使用带有 tensorflow 后端的 GPU #1 运行 keras。鉴于我的 CUDA/CUDNN 版本,我知道我必须安装 tensorflow-gpu 1.2 和 keras 2.0.5(有关兼容性,请参阅 here 和 here)。

首先,我创建一个这样的虚拟环境:

conda create -n keras
source activate keras
conda install keras=2.0.5 tensorflow-gpu=1.2

然后,如果我使用以下脚本测试整个事情:

import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]="1"
import keras
model = keras.models.Sequential()
model.add(keras.layers.Dense(1,input_dim=1))
model.compile(loss="mse",optimizer="adam")
import numpy as np
model.fit(np.arange(12).reshape(-1,1),np.arange(12))

我收到以下错误:

Epoch 1/10
2018-12-13 15:20:42.971806: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2018-12-13 15:20:42.971827: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2018-12-13 15:20:42.971833: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2018-12-13 15:20:42.971838: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2018-12-13 15:20:42.971843: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2018-12-13 15:20:42.996052: E tensorflow/core/common_runtime/direct_session.cc:138] Internal: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_INVALID_DEVICE

我们可以在日志中看到它尝试在设备 0 上创建会话,该会话已被占用,如 nvidia-smi 命令所示。但是我指定在脚本中使用数字 1。

你知道这里会出什么问题吗?

如果这个问题不恰当,我很抱歉,但我已经为此苦苦挣扎了几天,似乎无法进一步取得进展。

【问题讨论】:

    标签: python tensorflow keras


    【解决方案1】:

    既然我解决了我的问题,我就回答我自己的问题。

    实际上有两个问题:

    1. 在安装 tensorflow-gpu=1.2 时,安装的是 6.0 版的 cudnn(我有 cudnn 5.1.10)。解决方案是像这样安装软件包:

      conda install keras=2.0.5 tensorflow-gpu=1.2 cudnn=5.1.10

    2. 第二个问题,实际上是“真正的”问题,是我的一些旧进程仍在后台运行。虽然它们没有在nvidia-smi 面板中列出,但它们仍然持有 GPU,因此我的测试无法访问它们。使用kill 命令杀死这些进程可以解决这个问题

    我希望这些见解能帮助其他像我一样挣扎的人。

    【讨论】:

      猜你喜欢
      • 2018-05-07
      • 2018-08-16
      • 2019-10-12
      • 1970-01-01
      • 2021-04-13
      • 2020-08-11
      • 2018-05-18
      • 1970-01-01
      • 2019-08-17
      相关资源
      最近更新 更多