在 deeplearning4j 中翻译千层面神经网络

Question

我正在将千层面神经网络翻译成 deeplearning4j 代码。到目前为止，我已经设法使图层到位，但我不确定其他配置是否还可以。我不是神经网络方面的专家，无法在 deeplearning4j 中轻松找到等效的函数/方法。

这是烤宽面条的python代码：

    conv_net = NeuralNet(
    layers=[
        ('input', layers.InputLayer),
        ('conv1a', layers.Conv2DLayer),
        ('conv1', layers.Conv2DLayer),
        ('pool1', layers.MaxPool2DLayer),
        ('dropout1', layers.DropoutLayer),
        ('conv2a', layers.Conv2DLayer),
        ('conv2', layers.Conv2DLayer),
        ('pool2', layers.MaxPool2DLayer),
        ('dropout2', layers.DropoutLayer),
        ('conv3a', layers.Conv2DLayer),
        ('conv3', layers.Conv2DLayer),
        ('pool3', layers.MaxPool2DLayer),
        ('dropout3', layers.DropoutLayer),
        ('hidden4', layers.DenseLayer),
        ('dropout4', layers.DropoutLayer),
        ('hidden5', layers.DenseLayer),
        ('output', layers.DenseLayer),
    ],

    input_shape=(None, NUM_CHANNELS, IMAGE_SIZE, IMAGE_SIZE),
    conv1a_num_filters=16, conv1a_filter_size=(7, 7), conv1a_nonlinearity=leaky_rectify,
    conv1_num_filters=32, conv1_filter_size=(5, 5), conv1_nonlinearity=leaky_rectify, pool1_pool_size=(2, 2), dropout1_p=0.1,
    conv2a_num_filters=64, conv2a_filter_size=(5, 5), conv2a_nonlinearity=leaky_rectify,
    conv2_num_filters=64, conv2_filter_size=(3, 3), conv2_nonlinearity=leaky_rectify, pool2_pool_size=(2, 2), dropout2_p=0.2,
    conv3a_num_filters=256, conv3a_filter_size=(3, 3), conv3a_nonlinearity=leaky_rectify,
    conv3_num_filters=256, conv3_filter_size=(3, 3), conv3_nonlinearity=leaky_rectify, pool3_pool_size=(2, 2), dropout3_p=0.2,
    hidden4_num_units=1250, dropout4_p=0.75, hidden5_num_units=1000,
    output_num_units=y.shape[1], output_nonlinearity=None,

    batch_iterator_train=AugmentBatchIterator(batch_size=180),

    update_learning_rate=theano.shared(np.cast['float32'](0.03)),
    update_momentum=theano.shared(np.cast['float32'](0.9)),

    on_epoch_finished=[
        AdjustVariable('update_learning_rate', start=0.01, stop=0.0001),
        AdjustVariable('update_momentum', start=0.9, stop=0.999),
        StoreBestModel('wb_' + out_file_name)
    ],

    regression=True,
    max_epochs=600,
    train_split=0.1,
    verbose=1,
)

conv_net.batch_iterator_train.part_flips = flip_idxs
conv_net.load_params_from('wb_keypoint_net3.pk')

conv_net.fit(X, y)

这就是我目前在 deeplearning4j 中的成果：

  int batch = 100;
    int iterations = data.getX().size(0) / batch + 1;
    int epochs = 600;
    logger.warn("Building model");
    MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
            .updater(Updater.NESTEROVS).momentum(0.9)
            .activation(Activation.RELU)
            .weightInit(WeightInit.XAVIER)
            .learningRate(0.3)
            .learningRateDecayPolicy(LearningRatePolicy.Score)
            .lrPolicyDecayRate(0.1)
            .regularization(true).l2(1e-4)
            .list()
            .layer(0, new ConvolutionLayer.Builder(7, 7).activation(Activation.LEAKYRELU).nOut(16).build()) //rectified linear units
            .layer(1, new ConvolutionLayer.Builder(5, 5).nOut(32).activation(Activation.LEAKYRELU).build())
            .layer(2, new SubsamplingLayer.Builder(SubsamplingLayer.PoolingType.MAX).kernelSize(2, 2).build())
            .layer(3, new DropoutLayer.Builder(0.1).build())
            .layer(4, new ConvolutionLayer.Builder(5, 5).nOut(64).activation(Activation.LEAKYRELU).build())
            .layer(5, new ConvolutionLayer.Builder(3, 3).nOut(64).activation(Activation.LEAKYRELU).build())
            .layer(6, new SubsamplingLayer.Builder(SubsamplingLayer.PoolingType.MAX).kernelSize(2, 2).build())
            .layer(7, new DropoutLayer.Builder(0.2).build())
            .layer(8, new ConvolutionLayer.Builder(3, 3).nOut(256).activation(Activation.LEAKYRELU).build())
            .layer(9, new ConvolutionLayer.Builder(3, 3).nOut(256).activation(Activation.LEAKYRELU).build())
            .layer(10, new SubsamplingLayer.Builder(SubsamplingLayer.PoolingType.MAX).kernelSize(2, 2).build())
            .layer(11, new DropoutLayer.Builder(0.2).build())
            .layer(12, new DenseLayer.Builder().nOut(1250).build())
            .layer(13, new DropoutLayer.Builder(0.75).build())
            .layer(14, new DenseLayer.Builder().nOut(1000).build())
            .layer(15, new OutputLayer.Builder(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD)
                    .nOut(data.getY().size(1)).activation(Activation.SOFTMAX).build())
            .setInputType(InputType.convolutional(image_size, image_size, num_channels))
            .backprop(true).pretrain(false)
            .build();

    MultiLayerNetwork model = new MultiLayerNetwork(conf);
    DataSet dataSet = new DataSet(data.getX(), data.getY());


    MiniBatchFileDataSetIterator iterator1 = new MiniBatchFileDataSetIterator(dataSet, batch);


    model.init();
    logger.warn("Train model");

    model.setListeners(new ScoreIterationListener(iterations));
    UtilSaveLoadMultiLayerNetwork uslmln = new UtilSaveLoadMultiLayerNetwork();
    for (int i = 0; i < epochs; i++) {
        logger.warn("Started epoch " + i);
        model.fit(iterator1);
        uslmln.save(model, filename);
     }

我主要感兴趣的是激活函数和配置是否相同。问题是，当我在 java 中运行神经网络时，它似乎根本没有学习，即使在 50 个 epoch 之后，分数似乎也保持在 0.2，没有明显的改进，我确信有些地方配置错误。

谢谢

Answer 1

您的数据管道是否完全相同？这也包括归一化等。使用 deeplearning4j，您无需指定输出数量。我们为您做到这一点。另外 - 您使用的 UI 服务器错误。我们的示例已经演示了如何执行这些操作： https://github.com/deeplearning4j/dl4j-examples

我不确定是什么导致您这样做，但您每次都在重新附加存储，因此神经网络的实际统计数据不会随着时间的推移而持续存在。您应该将其设置在您的 for 循环之上，而不是在期间。如果您想要像您在这里尝试做的那样进行模型快照，您可能需要考虑使用提前停止。

另外 - 您是如何获得偏向学习率的？这甚至没有出现在您的 lasange 配置中。这看起来很随意。我建议摆脱它。 您的输出层看起来也有问题。您应该使用负对数似然和 softmax（再次查看我们的示例，都在其中） 从外观上看，您也在 lasange 中使用学习率衰减。 Deeplearning4j 也支持这一点。我会查看我们的示例以了解如何做到这一点。我们支持几种学习率衰减策略。您应该能够在 javadoc (http://deeplearning4j.org/doc) 或 ide 的自动完成中找到它。