【问题标题】:Tensorflow and PyTorch hang on initializing with CUDATensorflow 和 PyTorch 挂起使用 CUDA 进行初始化
【发布时间】:2020-04-19 04:23:16
【问题描述】:

当我尝试运行一个非常小的 Tensorflow 示例时:

import tensorflow as tf

c = tf.constant([1,2,3])

系统永远挂起(至少十分钟),没有任何迹象表明它在做什么。在此状态下,它使用 100% 的一个虚拟 CPU 内核。在 Juypter 笔记本中运行时,内核会将其输出到控制台:

2020-03-31 11:12:04.840507: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-03-31 11:12:04.840576: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-03-31 11:12:04.840589: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2020-03-31 11:12:05.521172: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-03-31 11:12:05.539193: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-31 11:12:05.539639: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1070 computeCapability: 6.1
coreClock: 1.7845GHz coreCount: 15 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 238.66GiB/s
2020-03-31 11:12:05.539841: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-03-31 11:12:05.541113: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-03-31 11:12:05.542119: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-03-31 11:12:05.542324: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-03-31 11:12:05.543632: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-03-31 11:12:05.544401: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-03-31 11:12:05.547212: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-03-31 11:12:05.547337: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-31 11:12:05.548015: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-31 11:12:05.548512: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-03-31 11:12:05.567845: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3393550000 Hz
2020-03-31 11:12:05.568364: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x564107e16440 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-03-31 11:12:05.568395: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version

我之前确实在这个系统上运行过 Tensorflow,所以我认为这可能是由系统更新引起的某种库问题。

GPU 是 Nvidia GTX 1070。Tensorflow 版本是 2.1.0,自运行时起就没有变化。运行 Arch Linux,如果这很重要的话。

我尝试从 CUDA 10.2 降级到 10.1,但问题仍然存在。

我也可以用 PyTorch 重现这个:

import torch
import transformers

t = torch.tensor([1,2,3])
t.cuda()

import transformers 可防止“CUDA:内存不足”问题 - 它一定有一些我不知道该怎么做的初始化 PyTorch 的事情。)

这有同样的问题,它冻结了一个 CPU 内核,尽管它产生的输出更少:

020-03-31 11:13:41.428483: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-03-31 11:13:41.428571: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-03-31 11:13:41.428587: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.

我很确定有关 TensorRT 的抱怨无关紧要,因为当我之前进行此工作时,它也会输出这些内容。

我该如何解决这个问题?或者至少,我还能做些什么来确定它在冻结时正在做什么?

【问题讨论】:

    标签: tensorflow pytorch archlinux


    【解决方案1】:

    我的问题是由我对 Python 进程大声消耗的虚拟内存量设置的 ulimit 引起的(在 zsh 中使用 ulimit -Sv 12000000)。我不知道为什么会导致它挂起,但如果其他人遇到类似问题,请确保您没有限制虚拟内存。

    【讨论】:

      猜你喜欢
      • 2020-07-10
      • 2021-09-03
      • 2016-02-11
      • 2017-08-22
      • 2019-01-08
      • 1970-01-01
      • 2014-10-31
      • 2016-06-18
      相关资源
      最近更新 更多