运行 resnet 模型 tensorflow 时出现 OOM 错误

Question

我在EC2 g2(NVIDIA GRID K520) 实例上运行the resenet model 并看到OOM 错误。我尝试了各种组合来删除使用GPU 的代码，前缀CUDA_VISIBLE_DEVICES='0' 并将batch_size 减少到64。我仍然无法开始训练。你能帮我吗？

W tensorflow/core/common_runtime/bfc_allocator.cc:270] **********************x***************************************************************************xx
W tensorflow/core/common_runtime/bfc_allocator.cc:271] Ran out of memory trying to allocate 196.00MiB.  See logs for memory state.
W tensorflow/core/framework/op_kernel.cc:936] Resource exhausted: OOM when allocating tensor with shape[64,16,224,224]
E tensorflow/core/client/tensor_c_api.cc:485] OOM when allocating tensor with shape[64,16,224,224]
     [[Node: unit_1_2/sub1/conv1/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](unit_1_2/residual_only_activation/leaky_relu, unit_1_2/sub1/conv1/DW/read)]]
     [[Node: train_step/update/_1561 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_10115_train_step/update", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Traceback (most recent call last):
  File "./resnet_main.py", line 203, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv))
  File "./resnet_main.py", line 197, in main
    train(hps)
  File "./resnet_main.py", line 82, in train
    feed_dict={model.lrn_rate: lrn_rate})
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 382, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 655, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 723, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 743, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.ResourceExhaustedError: OOM when allocating tensor with shape[64,16,224,224]
     [[Node: unit_1_2/sub1/conv1/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](unit_1_2/residual_only_activation/leaky_relu, unit_1_2/sub1/conv1/DW/read)]]
     [[Node: train_step/update/_1561 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_10115_train_step/update", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Caused by op u'unit_1_2/sub1/conv1/Conv2D', defined at:
  File "./resnet_main.py", line 203, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv))
  File "./resnet_main.py", line 197, in main
    train(hps)
  File "./resnet_main.py", line 64, in train
    model.build_graph()
  File "/home/ubuntu/indu/tf-benchmark/resnet/resnet_model.py", line 59, in build_graph
    self._build_model()
  File "/home/ubuntu/indu/tf-benchmark/resnet/resnet_model.py", line 94, in _build_model
    x = res_func(x, filters[1], filters[1], self._stride_arr(1), False)
  File "/home/ubuntu/indu/tf-benchmark/resnet/resnet_model.py", line 208, in _residual
    x = self._conv('conv1', x, 3, in_filter, out_filter, stride)
  File "/home/ubuntu/indu/tf-benchmark/resnet/resnet_model.py", line 279, in _conv
    return tf.nn.conv2d(x, kernel, strides, padding='SAME')
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 394, in conv2d
    data_format=data_format, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2310, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1232, in __init__
    self._traceback = _extract_stack()

Answer 1

NVIDIA GRID K520 具有 8GB 内存 (link)。我已经在具有 12GB 内存的 NVIDIA GPU 上成功训练了 ResNet 模型。正如错误提示的那样，TensorFlow 尝试将所有网络权重放在 GPU 内存中，但失败了。我相信你有几个选择：

• 仅在 CPU 上进行训练，如 cmets 中所述，假设您的 CPU 具有超过 8GB 的​​内存。不建议这样做。
• 用更少的参数训练不同的网络。自 Resnet 以来，已经发布了几个网络，例如Inception-v4, Inception-ResNet，它们的参数更少，精度相当。尝试此选项无需任何费用！
• 购买具有更多内存的 GPU。如果你有钱，最简单的选择。
• 购买另一个具有相同内存的 GPU，并在一个上训练网络的下半部分，在另一个上训练网络的上半部分。 GPU 之间的通信困难使得该选项不太理想。

我希望这对您和其他遇到类似内存问题的人有所帮助。