会话和并行性在 TF2.0 中如何工作？答案

【问题标题】：How do sessions and parallelism work in TF2.0?会话和并行性在 TF2.0 中如何工作？
【发布时间】：2020-01-19 23:47:48
【问题描述】：

我正在尝试在同一进程中并行运行两个 tensorflow 模型。

在 Tensorflow 1.x 中，我们可以做例如Keras Tensorflow - Exception while predicting from multiple threads

graph = tf.Graph()
with graph.as_default():
    session = tf.Session()
    with session.as_default():
        # Create and use the model ...

在 TensorFlow 2.0 中，会话已被删除。

当我尝试在同一进程中并行运行两个模型时，Tensorflow 崩溃。 TF 2.0 有没有办法解决这个问题？

错误信息：

2019-09-18 16:26:56.978352: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.981418: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.983202: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.985757: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.987541: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.990140: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.992499: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.994886: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.997778: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.997802: W tensorflow/stream_executor/stream.cc:1919] attempting to perform BLAS operation using StreamExecutor without BLAS support
2019-09-18 16:26:56.997829: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Internal: Blas GEMM launch failed : a.shape=(1, 2520), b.shape=(2520, 2520), m=1, n=2520, k=2520
     [[{{node dense/MatMul}}]]
2019-09-18 16:26:57.000709: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:57.000744: W tensorflow/stream_executor/stream.cc:1919] attempting to perform BLAS operation using StreamExecutor without BLAS support
Traceback (most recent call last):
  File 
    result = model.predict([numpy.array([inV,]), numpy.array([inS,])])
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 915, in predict
    use_multiprocessing=use_multiprocessing)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_arrays.py", line 722, in predict
    callbacks=callbacks)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_arrays.py", line 393, in model_iteration
    batch_outs = f(ins_batch)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/keras/backend.py", line 3625, in __call__
    outputs = self._graph_fn(*converted_inputs)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1081, in __call__
    return self._call_impl(args, kwargs)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1121, in _call_impl
    return self._call_flat(args, self.captured_inputs, cancellation_manager)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1224, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 511, in call
    ctx=ctx)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InternalError:  Blas GEMM launch failed : a.shape=(1, 2520), b.shape=(2520, 2520), m=1, n=2520, k=2520
     [[node dense/MatMul (defined at usr/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]] [Op:__inference_keras_scratch_graph_378]

Function call stack:
keras_scratch_graph

【问题讨论】：

你找到解决办法了吗？
我会考虑在 tensorflow 页面上提出这个问题。您是否尝试过使用 tf.compat.v1.Session？ tensorflow 的人保留了一些 TF1.0 的东西……更糟糕的是，你需要升级吗？如果升级到 2.0 会破坏您的程序。将 tensorflow 保持在 1.0 可能仍然可行。

标签： python tensorflow tensorflow2.0 tf.keras

【解决方案1】：

尝试为每个模型允许一小部分 GPU。

import tensorflow as tf
config = tf.compat.v1.ConfigProto(gpu_options = 
                     tf.compat.v1.GPUOptions(per_process_gpu_memory_fraction=0.8)
# device_count = {'GPU': 1}
)
config.gpu_options.allow_growth = True
session = tf.compat.v1.Session(config=config)
tf.compat.v1.keras.backend.set_session(session)

【讨论】：