【问题标题】:Function call stack: keras_scratch_graph Error函数调用堆栈:keras_scratch_graph 错误
【发布时间】:2019-11-25 11:55:33
【问题描述】:

我正在重新实现一个 text2speech 项目。 我在解码器部分面临 函数调用堆栈:keras_scratch_graph 错误。 网络架构来自 Deep Voice 3 论文。

我在 Google Colab 上使用 TF 2.0 的 keras。 下面是 Decoder Keras Model 的代码。

y1 = tf.ones(shape = (16, 203, 320))
def Decoder(name = "decoder"):
    # Decoder Prenet
    din = tf.concat((tf.zeros_like(y1[:, :1, -hp.mel:]), y1[:, :-1, -hp.mel:]), 1)
    keys = K.Input(shape = (180, 256), batch_size = 16, name = "keys")
    vals = K.Input(shape = (180, 256), batch_size = 16, name = "vals")
    prev_max_attentions_li = tf.ones(shape=(hp.dlayer, hp.batch_size), dtype=tf.int32)
    #prev_max_attentions_li = K.Input(tensor = prev_max_attentions_li)
    for i in range(hp.dlayer):
        dpout = K.layers.Dropout(rate = 0 if i == 0 else hp.dropout)(din)
        fc_out = K.layers.Dense(hp.char_embed, activation = 'relu')(dpout)

    print("=======================================================================================================")
    print("The FC value is ", fc_out)
    print("=======================================================================================================")

    query_pe = K.layers.Embedding(hp.Ty, hp.char_embed)(tf.tile(tf.expand_dims(tf.range(hp.Ty // hp.r), 0), [hp.batch_size, 1]))
    key_pe = K.layers.Embedding(hp.Tx, hp.char_embed)(tf.tile(tf.expand_dims(tf.range(hp.Tx), 0), [hp.batch_size, 1]))

    alignments_li, max_attentions_li = [], []
    for i in range(hp.dlayer):
        dpout = K.layers.Dropout(rate = 0)(fc_out)
        queries = K.layers.Conv1D(hp.datten_size, hp.dfilter, padding = 'causal', dilation_rate = 2**i)(dpout)
        fc_out = (queries + fc_out) * tf.math.sqrt(0.5)
        print("=======================================================================================================")
        print("The FC value is ", fc_out)
        print("=======================================================================================================")
        queries = fc_out + query_pe
        keys += key_pe

        tensor, alignments, max_attentions = Attention(name = "attention")(queries, keys, vals, prev_max_attentions_li[i])

        fc_out = (tensor + queries) * tf.math.sqrt(0.5)

        alignments_li.append(alignments)
        max_attentions_li.append(max_attentions)

    decoder_output = fc_out

    dpout = K.layers.Dropout(rate = 0)(decoder_output)
    mel_logits = K.layers.Dense(hp.mel * hp.r)(dpout)

    dpout = K.layers.Dropout(rate = 0)(fc_out)
    done_output = K.layers.Dense(2)(dpout)

    return K.Model(inputs = [keys, vals], outputs = [mel_logits, done_output, decoder_output, alignments_li, max_attentions_li], name = name)

decode = Decoder()
kin = tf.ones(shape = (16, 180, 256))
vin = tf.ones(shape = (16, 180, 256))
print(decode(kin, vin))
tf.keras.utils.plot_model(decode, to_file = "decoder.png", show_shapes = True)

当我使用一些数据进行测试时,它会显示以下错误消息。 “fc_out”会有一些问题,但我不知道如何将“fc_out”输出从第一个 for 循环传递到第二个 for 循环? 任何答案将不胜感激。

File "Decoder.py", line 60, in <module>
    decode = Decoder()
  File "Decoder.py", line 33, in Decoder
    dpout = K.layers.Dropout(rate = 0)(fc_out)
  File "/Users/ydc/dl-npm/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 596, in __call__
    base_layer_utils.create_keras_history(inputs)
  File "/Users/ydc/dl-npm/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer_utils.py", line 199, in create_keras_history
    _, created_layers = _create_keras_history_helper(tensors, set(), [])
  File "/Users/ydc/dl-npm/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer_utils.py", line 245, in _create_keras_history_helper
    layer_inputs, processed_ops, created_layers)
  File "/Users/ydc/dl-npm/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer_utils.py", line 245, in _create_keras_history_helper
    layer_inputs, processed_ops, created_layers)
  File "/Users/ydc/dl-npm/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer_utils.py", line 245, in _create_keras_history_helper
    layer_inputs, processed_ops, created_layers)
  File "/Users/ydc/dl-npm/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer_utils.py", line 243, in _create_keras_history_helper
    constants[i] = backend.function([], op_input)([])
  File "/Users/ydc/dl-npm/lib/python3.7/site-packages/tensorflow/python/keras/backend.py", line 3510, in __call__
    outputs = self._graph_fn(*converted_inputs)
  File "/Users/ydc/dl-npm/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 572, in __call__
    return self._call_flat(args)
  File "/Users/ydc/dl-npm/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 671, in _call_flat
    outputs = self._inference_function.call(ctx, args)
  File "/Users/ydc/dl-npm/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 445, in call
    ctx=ctx)
  File "/Users/ydc/dl-npm/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.FailedPreconditionError:  Error while reading resource variable _AnonymousVar19 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/_AnonymousVar19/N10tensorflow3VarE does not exist.
     [[node dense_7/BiasAdd/ReadVariableOp (defined at Decoder.py:33) ]] [Op:__inference_keras_scratch_graph_566]

Function call stack:
keras_scratch_graph

【问题讨论】:

  • 您解决了这个问题吗?我现在面临同样的问题。

标签: python tensorflow keras nlp tensorflow2.0


【解决方案1】:

我的情况是 tensorflow 示例代码在 Google colab 中运行良好,但在我的机器上运行不正常,因为我遇到了 keras_scratch_graph 错误。

然后我在开头添加这个 Python 代码,它工作正常。

gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
    try:
        # Restrict TensorFlow to only use the fourth GPU
        tf.config.experimental.set_visible_devices(gpus[0], 'GPU')

        # Currently, memory growth needs to be the same across GPUs
        for gpu in gpus:
            tf.config.experimental.set_memory_growth(gpu, True)
        logical_gpus = tf.config.experimental.list_logical_devices('GPU')
        print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
    except RuntimeError as e:
        # Memory growth must be set before GPUs have been initialized
        print(e)

默认情况下,TensorFlow 会将几乎所有 GPU 的所有 GPU 内存(以 CUDA_VISIBLE_DEVICES 为准)映射到进程可见。

在某些情况下,希望进程只分配可用内存的一个子集,或者只根据进程的需要增加内存使用量。

例如,您想同时使用一个 GPU 训练多个小型模型。 通过调用tf.config.experimental.set_memory_growth,它尝试仅分配运行时分配所需的尽可能多的 GPU 内存:它开始分配非常少的内存,随着程序运行并需要更多 GPU 内存,我们扩展了分配的 GPU 内存区域到 TensorFlow 进程。

希望对你有帮助!

【讨论】:

  • 感谢您的详尽回答。当tf.config.experimental.list_physical_devices('GPU') 产生一个空列表时,我面临同样的错误,即gpus 是False。你能想到任何原因吗?
  • @Rumo 主要是安装问题。如果您使用的是 TF 2.0+,检查您是否安装了支持 GPU 的 TF 的简单方法是使用 tf.test.is_built_with_cuda() 或 tf.test.is_built_with_gpu_support()。如果返回 False,则您必须像 documentation 一样重新安装 TensorFlow。
  • @SurajDonthi 谢谢,但我不想使用 GPU 支持。我的意思是,此错误不仅归因于 GPU 问题。就我而言,原因是我使用了 Eager 模式不支持的 tf.metrics.iou。它在切换到tf.keras.MeanIoU 后工作。我认为出现此错误的原因可能非常不同。
【解决方案2】:

我遇到了类似的错误。我减少了批量大小,错误消失了。我不知道为什么,但它对我有用。我猜测与过度堆叠有关。

【讨论】:

    【解决方案3】:

    我认为这与 gpu 有关。查看回溯:

    File "/Users/ydc/dl-npm/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 572, in __call__
        return self._call_flat(args)
    

    tf 正在调用急切执行,这意味着如果版本可用,将使用 gpu。我在测试密集网络时遇到了同样的问题:

    inputs=Input(shape=(100,)
                 )
    x=Dense(32, activation='relu')(inputs)
    x=Dense(32, activation='relu')(x)
    x=Dense(32, activation='relu')(x)
    outputs=Dense(10, activation='softmax')(x)
    model=Model(inputs=inputs, outputs=outputs)
    model.compile(optimizer='adam', loss='sparse_categorical_crossentropy',
                  metrics=['accuracy'])
    t=tf.zeros([1,100])
    model.predict(t, steps=1, batch_size=1)
    

    ...它给出了类似的回溯,也链接到急切的执行。 然后当我使用以下行禁用 gpu 时:

    tf.config.experimental.set_visible_devices([], 'GPU')
    

    ... 代码运行良好。看看这是否有助于解决问题。 顺便说一句,colab 甚至支持 gpu 吗?我什至不知道。

    【讨论】:

      【解决方案4】:

      如果你使用 Tensorflow-GPU,那么添加:

      physical_devices = tf.config.experimental.list_physical_devices('GPU')
      print("physical_devices-------------", len(physical_devices))
      tf.config.experimental.set_memory_growth(physical_devices[0], True)
      

      另外,你可以减少你的batch_size或者换一台电脑或者云服务,比如google colab,amazon cloud来运行你的代码,因为我认为这是因为内存的限制。

      【讨论】:

        【解决方案5】:

        在我的情况下,我必须更新 keras 和 tensorflow

        pip install -U tensorflow keras 
        

        【讨论】:

          【解决方案6】:

          您需要在脚本顶部添加此代码,它适用于 TensorFlow 2.2.0

          if tf.config.list_physical_devices('GPU'):
              physical_devices = tf.config.list_physical_devices('GPU')
              tf.config.experimental.set_memory_growth(physical_devices[0], enable=True)
              tf.config.experimental.set_virtual_device_configuration(physical_devices[0], [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=4000)])
          

          【讨论】:

            【解决方案7】:

            以下 TensorFlow 和 Keras 版本消除了我的问题:

            tensorflow==2.2.0
            keras=2.3.1
            

            【讨论】:

            • 我有这些版本的 keras 和 tensorflow,但仍然有这个错误
            • 你试过keras=2.3.2吗?
            猜你喜欢
            • 1970-01-01
            • 2020-10-26
            • 1970-01-01
            • 1970-01-01
            • 1970-01-01
            • 2012-01-03
            • 1970-01-01
            • 2021-08-01
            • 1970-01-01
            相关资源
            最近更新 更多