如何在tensorflow中使用变压器bert保存每个时期的最佳模型

Question

我使用 TFBertModel 和 Tensorflow 模型与 Hugging Face 转换器组合和训练。我想保存每个 epoch 的 val_accuracy 的最佳模型。我使用了“tensorflow 检查点”，但我得到了错误。如何保存最好的tensorflow 中每个 epoch 的模型与变压器 bert？

from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras.initializers import TruncatedNormal
from tensorflow.keras.losses import CategoricalCrossentropy,BinaryCrossentropy
from tensorflow.keras.metrics import CategoricalAccuracy,BinaryAccuracy
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.utils import plot_model
from transformers import AutoTokenizer,TFBertModel
import tensorflow as tf
from tensorflow.keras.layers import Input, Dense
tokenizer = AutoTokenizer.from_pretrained('bert-large-uncased')
bert = TFBertModel.from_pretrained('bert-large-uncased')
max_len = max_length
input_ids = Input(shape=(max_len,), dtype=tf.int32, name="input_ids")
input_mask = Input(shape=(max_len,), dtype=tf.int32, name="attention_mask")
# embeddings = dbert_model(input_ids,attention_mask = input_mask)[0]


embeddings = bert(input_ids,attention_mask = input_mask)[1] #(0 is the last hidden states,1 means pooler_output)
# out = tf.keras.layers.GlobalMaxPool1D()(embeddings)
out = tf.keras.layers.Dropout(0.1)(embeddings)

out = Dense(128, activation='relu')(out)
out = tf.keras.layers.Dropout(0.1)(out)
out = Dense(32,activation = 'relu')(out)

y = Dense(1,activation = 'sigmoid')(out)
    
model = tf.keras.Model(inputs=[input_ids, input_mask], outputs=y)
model.layers[2].trainable = True
#model.save_weights('path/savefile')

Answer 1

如果您使用的是 tensorflow，那么您可以在下面创建回调，它将保存每个 epoch 的模型。 filepath 是要保存模型的目录的路径。 model 是您编译模型的名称。模型将以 epoch-validation loss.h5 格式保存

class model_per_epoch(keras.callbacks.Callback):
    def __init__(self, model,filepath):
        self.filepath=filepath
        self.model=model
    def on_epoch_end(self,epoch, logs=None):
        v_loss=logs.get('val_loss') 
        name= str(epoch) +'-' + str(v_loss)[:str(v_loss).rfind('.')+3] + '.h5'
        file_id=os.path.join(self.filepath, name)
        self.model.save(file_id)
save_dir=r'c:\Temp\models'
callbacks=[model_per_epoch(model, save_dir)] 

在 model.fit 中包含 callback=callbacks 。确保您保存模型的目录存在。下面的代码是更复杂的回调版本。添加了一个附加参数 save_best_only。如果设置为 True，则仅保存验证损失最低的模型。此外，模型还加载了验证损失最低的时期的权重，因此您可以直接使用模型进行预测，而无需加载保存的模型。这可以节省大量时间。

class model_per_epoch(keras.callbacks.Callback):
    def __init__(self, model,filepath,save_best_only):
        self.filepath=filepath
        self.model=model
        self.save_best_only=save_best_only
        self.lowest_loss=np.inf
        self.best_weights=self.model.get_weights()
    def on_epoch_end(self,epoch, logs=None):
        v_loss=logs.get('val_loss')
        if v_loss< self.lowest_loss:
            self.lowest_loss =v_loss
            self.best_weights=self.model.get_weights()
            self.best_epoch=epoch +1
        if self.save_best_only==False:
            name= str(epoch) +'-' + str(v_loss)[:str(v_loss).rfind('.')+3] + '.h5'
            file_id=os.path.join(self.filepath, name)
            self.model.save(file_id)
    def on_train_end(self, logs=None):
        if self.save_best_only == True:
            self.model.set_weights(self.best_weights)
            name= str(self.best_epoch) +'-' + str(self.lowest_loss)[:str(self.lowest_loss).rfind('.')+3] + '.h5'
            file_id=os.path.join(self.filepath, name)
            self.model.save(file_id)
            print(' model is returned with best weiights from epoch ', self.best_epoch)
            
save_dir=r'c:\Temp\models'
save_best_only= True
callbacks=[model_per_epoch(model, save_dir, save_best_only)]