重新启动自定义生成器函数以在 Keras 中训练新纪元

Question

我编写了一个自定义生成器，以采用我的特定文件顺序并以特定方式生成批次。生成器工作正常并根据需要返回批次。问题是：在训练时，我需要更多的 epoch 来完全训练模型，但是在第一个 epoch 耗尽后，生成器完成并且在下一个 epoch 不再返回任何内容。

使用model.fit(dataset_generator(clean_path, noisy_path, denoised_path, noise_data, batch_size=4), verbose=1, epochs = 2, batch_size=4) 使 tensorflow 按预期返回异常：

警告：tensorflow：您的输入数据已用完；中断训练。 确保您的数据集或生成器至少可以生成 steps_per_epoch * epochs 批次（在本例中为 12 批次）。你可以 构建数据集时需要使用 repeat() 函数。

解决此问题的最佳方法是什么？我的生成器还有一个种子输入来同步训练和验证集（很可能是 keras 中的生成器），如果我希望每个 epoch 有不同的种子，我是否也需要编写自定义 fit 函数？

此刻的生成器：

def dataset_generator(clean_path, noisy_path, denoised_path, noise_params, image_dimension=[512,512,3],
                      batch_size=32, shuffle=False, split=False, split_size=0.5, partition="training", seed=42):
    
    #Loading the file list for each of the datasets
    clean_list = listdir(clean_path)
    noisy_list = listdir(noisy_path)
    denoised_list = listdir(denoised_path)
    noise_data = np.load(noise_params)
    
    #Fixing the file order
    clean_list = sorted(clean_list, key=lambda x: int(x.split("."[0])[0]))
    noisy_list = sorted(noisy_list, key=lambda x: int(x.split("."[0])[0]))
    denoised_list = sorted(denoised_list, key=lambda x: int(x.split("."[0])[0]))
    
    #Setting conrol variables for trasversing the datasets
    dataset_size = len(noise_data)
    indexes = range(len(noise_data))
    index_counter = 0
    batch_index = 0
    
    #Returning infomartion on the dataset
    print("Folder " + clean_path + ": found " + str(len(clean_list)) + " files.")
    print("Folder " + noisy_path + ": found " + str(len(noisy_list)) + " files.")
    print("Folder " + denoised_path + ": found " + str(len(denoised_list)) + " files.")
    print("File " + noise_params + ": found " + str(len(noise_data)) + " entries.")
    
    #Verify if the files and the entries in noise file match   
    if(len(clean_list) != len(noisy_list) != len(denoised_list) != len(noise_data)):
        print("Datasets have different sizes. Aborting.")
        return
    
    #If shuffle was defined, permutate the indexes using a seed 
    if(shuffle):
        rng = np.random.default_rng(seed)
        shuffled_indexes = rng.permutation(len(noise_data))
        indexes = shuffled_indexes
      
    
    #If 'split' was set, return the indexes of the partition selected in 'partition'
    if(split):
        split_point = np.floor(split_size*dataset_size)
        if(partition=="training"):
            indexes = indexes[:split_point]
        elif(partition=="validation"):
            indexes = indexes[split_point:]
    
    #Calculate the number of batches given the size of the partition used
    batch_total = np.floor(len(indexes)/batch_size)
    
    clean_batch = np.empty([batch_size, image_dimension[0], image_dimension[1], image_dimension[2]])
    noisy_batch = np.empty([batch_size, image_dimension[0], image_dimension[1], image_dimension[2]])
    denoised_batch = np.empty([batch_size, image_dimension[0], image_dimension[1], image_dimension[2]])
    maps_batch = np.empty([batch_size, image_dimension[0], image_dimension[1], 1])
        
    while True:
        #check if the batch number is still in valid range
        if batch_index > batch_total:
            break
    
        for i in range(batch_size):
            
            #read the clean image for index n
            clean_image = cv.imread(clean_path + clean_list[indexes[index_counter]])
            clean_image = cv.cvtColor(clean_image, cv.COLOR_BGR2RGB)
            clean_image = clean_image[np.newaxis,:,:,:]/255
            clean_batch[i,:,:,:] = clean_image
            
            noisy_image = cv.imread(noisy_path + noisy_list[indexes[index_counter]])
            noisy_batch[i,:,:,:] = noisy_image
            
            denoised_image = cv.imread(denoised_path + denoised_list[indexes[index_counter]])
            denoised_batch[i,:,:,:] = denoised_image
            
            map_matrix = np.ones([image_dimension[0],image_dimension[1],1]) * noise_data[indexes[index_counter]]
            maps_batch[i,:,:,:] = map_matrix
            
            index_counter =+ 1
        
        yield([noisy_batch, maps_batch, denoised_batch], clean_batch)
        batch_index +=1

Answer 1

不，你错了，问题不在于生成器或时期数，因为错误是说你试图在同一时期传递相同的数据，你的生成器没有足够的数据。他不能在一个时期内传递不同的数据，你不需要更少的时期，你需要更短的时期。

我不知道你是否明白，你需要使用fit的参数steps_per_epoch来限制一个epoch的长度。

发生这种情况是因为您的生成器在 epoch 结束之前用完了新数据。