TensorFlow：使用单个 python 文件仅针对不同学习率微调全连接层

Question

我正在做一些迁移学习的实验：
我有一个脚本文件，在第一部分中，我在 mnist 数据集的子集上训练模型，然后成功保存它。 模型架构由2个C.N.N层和1个全连接层组成。

for epoch in range(1, params['epochs'] + 1):
                 shuffle = np.random.permutation(len(y_train))
                 x_train, y_train = x_train[shuffle], y_train[shuffle]


                 for i in range(0, len(y_train), params['batch_size']):
                     x_train_mb, y_train_mb = x_train[i:i + params['batch_size']], y_train[i:i + params['batch_size']]

                     sess.run(model.optimize, feed_dict={model.input: x_train_mb, model.target: y_train_mb, model.is_task1: True,
 model.is_train: True, model.learning_rate:
 temp_learning_rate_source_training})




                 valid_acc = classification_batch_evaluation(sess, model, model.metrics, params['batch_size'], True, x_valid, y=y_valid,
 stream=True)

                 print('valid [{} / {}] valid accuracy: {} learning Rate :{}'.format(epoch, params['epochs'] + 1,
 valid_acc,temp_learning_rate_source_training))
                 if valid_acc > initial_best_epoch['valid_acc']:
                     initial_best_epoch['epoch'] = epoch
                     initial_best_epoch['valid_acc'] = valid_acc
                     model.save_model(sess, epoch) 

                 if epoch - initial_best_epoch['epoch'] >= params['patience']:
                     print('Early Stopping Epoch: {}\n'.format(epoch))
                     logging.info('Early Stopping Epoch: {}\n'.format(epoch))
                     break


         print('Initial training done \n',file=f)
         logging.info('Initial training done \n')
         sess.close()      

     model.restore_model(sess) ##Restores the model after creating it .

现在我想通过保持架构相同并传输 C.N.N 层的参数并重新初始化全连接层来进行迁移学习。然后对有限的新数据集再次训练3层，使用不同的学习率和“decay_after_epoch”分析结果。现在由于大量的组合，我编写了 2 个 for - 循环来自动化该过程，如下所示：

for temp_learning_rate_target_training in (0.001,0.005,0.01):

        for decay_after_epoch in (3,5,10): 
            learning_rate = temp_learning_rate_target_training
            model.restore_model(sess) ##Restores the model after creating it .
            with open("/home/abhishek/Desktop/{}_{}_{}.txt".format(params["dataset"],params["k"],params["n"])) as f1:
                with open("/home/abhishek/Desktop/{}_{}_{}_{}_{}.txt".format(params["dataset"],params["k"],params["n"],temp_learning_rate_target_training,decay_after_epoch), "w") as f:
                    for x in f1.readlines():
                        f.write(x)
                    print("Target Training Begins",file=f)
                    for epoch in range(1, params['epochs'] + 1):
                        shuffle = np.random.permutation(len(y_train2))
                        x_train2, y_train2 = x_train2[shuffle], y_train2[shuffle]


                        if epoch%decay_after_epoch==0 and epoch <=decay_after_epoch:
                            learning_rate = learning_rate *0.1
                        elif (epoch-decay_after_epoch)%30==0:
                            learning_rate = learning_rate *0.1



                        for i in range(0, len(y_train2), params['batch_size']):
                            x_train_mb, y_train_mb = x_train2[i:i + params['batch_size']], y_train2[i:i + params['batch_size']]
                            sess.run(model.optimize, feed_dict={model.input: x_train_mb, model.target: y_train_mb, model.is_task1: False, model.is_train: True, model.learning_rate: params['learning_rate']})

                        train_acc = classification_batch_evaluation(sess, model, model.metrics, params['batch_size'], False, x_train2, y=y_train2, stream=True)
                        sess.close()

                        print('train [{} / {}] train accuracy: {} learning Rate:{} '.format(epoch, params['epochs'] + 1, train_acc,learning_rate),file=f)
                        print('train [{} / {}] train accuracy: {} learning Rate :{}'.format(epoch, params['epochs'] + 1, train_acc,learning_rate))
                        logging.info('train [{} / {}] train accuracy: {}'.format(epoch, params['epochs'] + 1, train_acc))

                        if train_acc > transfer_best_epoch['train_acc']:
                            transfer_best_epoch['epoch'] = epoch
                            transfer_best_epoch['train_acc'] = train_acc
                            test_acc = classification_batch_evaluation(sess, model, model.metrics, params['batch_size'], False, x_test2, y=y_test2, stream=True)
                            transfer_best_epoch['test_acc'] = test_acc

                        if epoch % params['patience'] == 0:
                            acc_diff = transfer_best_epoch['train_acc'] - es_acc
                            if acc_diff < params['percentage_es'] * es_acc:
                                print('Early Stopping Epoch: {}\n'.format(epoch))
                                logging.info('Early Stopping Epoch: {}\n'.format(epoch))
                                break
                            es_acc = transfer_best_epoch['train_acc']

                    print('Transfer training done \n',file=f)
                    print('TARGET test accuracy: {}'.format(transfer_best_epoch['test_acc']),file=f)

现在，在 temp_learning_rate_target_training = 0.0001 和 decay_after_epoch = 3 运行第一个循环之后，模型被训练，并且我具有测试精度，并让不同（3）层的权重和偏差由一组 S2 给出。 现在当循环再次运行时，参数 model.is_task1: False 确保全连接层被重新初始化，但 C.N.N 层的参数是从集合 S2 复制过来的。 （为什么我这么说是因为我得到了所有学习率和衰变后纪元组合的完全相同的准确度日志）。但是，我想为 S1 给出的 C.N.N 层训练具有相同初始参数的不同循环
我尝试在每个循环之后使用 sess.close() 关闭会话，然后使用 model.restore_model(sess) 恢复保存的模型（在代码的第 1 部分中进行了训练），但它仍然没有给出预期的结果。我应该如何进行？

Answer 1

如果您需要像这样调整模型，则需要知道要训练哪些变量。

所有层的变量都应该在其变量范围内。你可以得到变量：

tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope='my_scope')

对于你想重新初始化的变量，你可以这样做：

sess.run([v.initializer for v in variables_to_reset])

当你初始化 minimize method 时（它隐藏在你的 model 对象中，这就是你在训练循环中调用的 model.optimize 操作），你可以指定 var_list 这是列表您要训练的变量，其余的将保持不变。