NN 数字识别 - 内存错误，有什么想法吗？

Question

我试图为数字制作一个训练有素的 OCR，但我不断收到内存分配错误，似乎无法弄清楚出了什么问题，在开始时工作得很好，但在 2 次运行后它突然开始吐出这些错误，试过了删除 zip 并重新下载可能损坏的文件，但没有结果，甚至无法完成首次运行。

定位错误：

Traceback (most recent call last):
  File "/home/pcname/PycharmProjects/Bot/venv/lib/python3.6/site-packages/theano/compile/function_module.py", line 607, in __call__
    outputs = self.fn()
MemoryError: Unable to allocate array with shape (60000, 800) and data type float64

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pcname/PycharmProjects/Bot/Utility/TextHandler.py", line 90, in <module>
    train_err = train_fn(training_x, training_y)
  File "/home/pcname/PycharmProjects/Bot/venv/lib/python3.6/site-packages/theano/compile/function_module.py", line 618, in __call__
    storage_map=self.fn.storage_map)
  File "/home/pcname/PycharmProjects/Bot/venv/lib/python3.6/site-packages/theano/gof/link.py", line 269, in raise_with_op
    storage_map_list.sort(key=itemgetter(3), reverse=True)
TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'

代码：

import os
import urllib.request
import gzip
import numpy as np
import matplotlib
import matplotlib.pyplot as plt

matplotlib.use("TkAgg")
import lasagne
import theano
import theano.tensor as T

def build_nn(input_var=None):
        l_in = lasagne.layers.InputLayer(shape=(None, 1, 28, 28), input_var=input_var)
        l_in_drop = lasagne.layers.DropoutLayer(l_in, p=0.2)

        # Verborgen lagen.
        l_hid1 = lasagne.layers.DenseLayer(l_in_drop, num_units=800,
                                           nonlinearity=lasagne.nonlinearities.rectify,
                                           W=lasagne.init.GlorotUniform())

        l_hid1_drop = lasagne.layers.DropoutLayer(l_hid1, p=0.5)

        l_hid2 = lasagne.layers.DenseLayer(l_hid1_drop, num_units=800,
                                           nonlinearity=lasagne.nonlinearities.rectify,
                                           W=lasagne.init.GlorotUniform())

        l_hid2_drop = lasagne.layers.DropoutLayer(l_hid2, p=0.5)

        l_out = lasagne.layers.DenseLayer(l_hid2_drop, num_units=10,
                                          nonlinearity= lasagne.nonlinearities.softmax)

        return l_out


def load_dataset(file_path="../Lib/Zips/"):
    first_download = True
    def download(filename, source='http://yann.lecun.com/exdb/mnist/'):
        print("Downloading set: %s" % filename)
        urllib.request.urlretrieve(source+filename, file_path+filename)

    def load_imageset(file):
        # Indien het bestand nog niet gevonden is downloaden
        if not os.path.exists(file_path+file):
            download(file)

        # Openen van archief training set afbeeldingen.
        with gzip.open(file_path+file, 'rb') as reader:
            data = np.frombuffer(reader.read(), np.uint8, offset=16)
            data = data.reshape(-1, 1, 28, 28)

            return data/np.float32(256)

    def load_labelset(file):
        if not os.path.exists(file_path+file):
            download(file)

        with gzip.open(file_path+file, "rb") as reader:
            data = np.frombuffer(reader.read(), np.uint8, offset=8)

        return data

    training = ("train-images-idx3-ubyte.gz", "train-labels-idx1-ubyte.gz" )
    test = ("t10k-images-idx3-ubyte.gz", "t10k-labels-idx1-ubyte.gz")
    return load_imageset(training[0]),load_labelset(training[1]), load_imageset(test[0]), load_labelset(test[1])


x_train, y_train, x_test, y_test = load_dataset()


input_var = T.tensor4('inputs')
target_var = T.ivector('targets')

network = build_nn(input_var)

prediction = lasagne.layers.get_output(network)
loss = lasagne.objectives.categorical_crossentropy(prediction, target_var)
loss = loss.mean()

params = lasagne.layers.get_all_params(network, trainable=True)
updates = lasagne.updates.nesterov_momentum(loss, params, learning_rate=0.01, momentum=0.9)

theano.config.optimizer='fast_compile'
theano.config.exception_verbosity='high'
theano.config.compute_test_value = 'warn'

train_fn = theano.function([input_var, target_var], loss, updates=updates)
num_training_steps = 10

for step in range(num_training_steps):
    train_err = train_fn(x_train, y_train)

Answer 1

您的内存错误可能是因为您试图将所有您的数据同时通过网络。

你的代码：

for step in range(num_training_steps):
    train_err = train_fn(x_train, y_train)

更好的训练方法是使用小批量。文档在这里：https://lasagne.readthedocs.io/en/latest/user/tutorial.html

如果您有足够的内存来进行小批量大小的快速试验，您可以尝试将上面的代码替换为：

train_err = train_fn(x_train[0:8], y_train[0:8])

其中 8 将模拟 8 的批处理大小。如果代码运行没有错误，您可以确信，一旦您浏览文档链接，它就会运行良好。

你需要做的是实现一个生成器函数iterate_minibatches

然后像这样训练......

batch_size = 8 #you can edit
num_epochs = 4 #you can edit
for epoch in range(num_epochs):
    for batch in iterate_minibatches(x_train, y_train, batch_size, shuffle=True):
        inputs, targets = batch
        train_fn(inputs, targets)

或者尝试一些有点 hacky 的东西（用这个替换你的代码）：

batch_size = 8
ix = 0
for step in range(num_training_steps):
    train_err = train_fn(x_train[ix*batch_size:ix*(batch_size + 1)], y_train[ix*batch_size:ix*(batch_size + 1)])

注意！！未经测试的代码，没有任何错误检查或数组边界检查......只是一些想法让你开始。

Answer 2

我同意 guidot，这可能是正常的内存不足错误。

您的输入是一个 384MB 的数组，后跟一些 dropout 和显着宽度的密集层。

也许将您的前两个 Dense 层从 800 降低到其他值，然后再试一次。也许将其减少到 600 并查看它是否运行更长时间但最终给出相同的错误会提供更多证据。