拥抱脸转换器将 logit 分数转换为概率答案

【问题标题】：huggingface transformers convert logit scores to probability拥抱脸转换器将 logit 分数转换为概率
【发布时间】：2022-05-06 20:06:30
【问题描述】：

我是这个领域的初学者并且被卡住了。我正在按照本教程 (https://towardsdatascience.com/multi-label-multi-class-text-classification-with-bert-transformer-and-keras-c6355eccb63a) 使用 huggingface 转换器构建多标签分类。

以下是我用来训练模型的代码。

# Name of the BERT model to use
model_name = 'bert-base-uncased'
# Max length of tokens
max_length = 100

PATH = 'uncased_L-12_H-768_A-12/'

# Load transformers config and set output_hidden_states to False
config = BertConfig.from_pretrained(PATH)
config.output_hidden_states = False

# Load BERT tokenizer
tokenizer = BertTokenizerFast.from_pretrained(PATH, local_files_only=True, config = config)
# tokenizer = BertTokenizer.from_pretrained(PATH, local_files_only=True, config = config)

# Load the Transformers BERT model
transformer_model = TFBertModel.from_pretrained(PATH, config = config,from_pt=True)

#######################################
### ------- Build the model ------- ###

# Load the MainLayer
bert = transformer_model.layers[0]

# Build your model input
input_ids = Input(shape=(None,), name='input_ids', dtype='int32')
# attention_mask = Input(shape=(max_length,), name='attention_mask', dtype='int32') 
# inputs = {'input_ids': input_ids, 'attention_mask': attention_mask}
inputs = {'input_ids': input_ids}

# Load the Transformers BERT model as a layer in a Keras model
bert_model = bert(inputs)[1]
dropout = Dropout(config.hidden_dropout_prob, name='pooled_output')
pooled_output = dropout(bert_model, training=False)


# Then build your model output
issue = Dense(units=len(data.U_label.value_counts()), kernel_initializer=TruncatedNormal(stddev=config.initializer_range), name='issue')(pooled_output)
outputs = {'issue': issue}

# And combine it all in a model object
model = Model(inputs=inputs, outputs=outputs, name='BERT_MultiLabel_MultiClass')

# Take a look at the model
model.summary()


#######################################
### ------- Train the model ------- ###

# Set an optimizer
optimizer = Adam(
    learning_rate=5e-05,
    epsilon=1e-08,
    decay=0.01,
    clipnorm=1.0)

# Set loss and metrics
loss = {'issue': CategoricalCrossentropy(from_logits = True)}
# loss = {'issue': CategoricalCrossentropy()}
metric = {'issue': CategoricalAccuracy('accuracy')}

# Compile the model
model.compile(
    optimizer = optimizer,
    loss = loss, 
    metrics = metric)

from sklearn import preprocessing
le = preprocessing.LabelEncoder()
le.fit(data['U_H_Label'])

# Ready output data for the model
y_issue = to_categorical(le.transform(data['U_H_Label']))

# Tokenize the input (takes some time)
x = tokenizer(
    text=data['Input_Data'].to_list(),
    add_special_tokens=True,
    max_length=max_length,
    truncation=True,
    padding=True, 
    return_tensors='tf',
    return_token_type_ids = False,
    return_attention_mask = True,
    verbose = True)

# Fit the model
history = model.fit(
    # x={'input_ids': x['input_ids'], 'attention_mask': x['attention_mask']},
    x={'input_ids': x['input_ids']},
    y={'issue': y_issue},
    validation_split=0.2,
    batch_size=64,
    epochs=10)

当我使用 model.predict() 函数时，我想我得到了每个类的 logit 分数，并想将它们转换为从 0 到 1 的概率分数。

我在多个博客中读到过，我必须使用 softmax 函数，但无法说明在哪里以及如何使用。如果有人能告诉我需要哪一行代码，我将不胜感激！

【问题讨论】：

您是在尝试多标签分类还是多类分类？
多标签分类
你必须使用 sigmoid 激活函数，以及二元交叉熵作为损失函数。
您能帮我在代码中的什么地方添加 sigmoid 激活函数吗？在最后一个密集层中制作激活函数会做到吗？感谢指出损失函数，我一定会改的。
是的，你必须在最后一个 Dense Layer 之后添加 sigmoid

标签： python text-classification bert-language-model multilabel-classification huggingface-transformers

【解决方案1】：

一旦你从 model.predict() 得到 logit 分数，那么你可以这样做：

from torch.nn import functional as F
import torch

# convert logit score to torch array
torch_logits = torch.from_numpy(logit_score)

# get probabilities using softmax from logit score and convert it to numpy array
probabilities_scores = F.softmax(torch_logits, dim = -1).numpy()[0]

【讨论】：