Tensorflow Transformer Decoder 输出未给出预期结果答案

【问题标题】：Tensorflow Transformer Decoder output not giving the expected resultTensorflow Transformer Decoder 输出未给出预期结果
【发布时间】：2020-01-28 17:18:29
【问题描述】：

我使用 tensorflow 设计了一个变压器模型。该模型的目的是生成一个文本序列，理想情况下是一个问题，然后是一个给定输入句子的答案。

我有格式如下的数据点（大约 15k）

SOURCE SENTENCE: <@>A man in the distance is walking past a brick wall painted with words and graffiti.<#>where<%>wall<?>brick
TARGET SENTENCE: <^>where is the man walking ?<~>A man is walking past a brick wall

我已经使用句子分词器训练了模型。

出于某种原因，即使在将模型训练到 100 个 epoch 之后，我也没有得到想要的输出。我希望网络能够接收到源句中的单词并构造一个问答对。但是，实际上，网络构建了一个问答对（这确实很好，但是它用户的单词不在源句子中。

以下是 50 Epochs 后上述源输入的网络输出，波束搜索宽度为 15。

PRED: <^>what does the woman?<~>the girls are young
PRED: <^>what was the girl holding ?<~>the girl was be
PRED: <^>what was the girl doing ?<~>the man are posing.
PRED: <^>what was the girl doing ?<~>the man are posing
PRED: <^>what was the girl holding ?<~>the girl was looking
PRED: <^>what was the girl holding ?<~>a man wearing a black shirt.
PRED: <^>what is the girls are ?<~>the girls are wearing a young man
PRED: <^>what is the girls are ?<~>the girls are wearing a
PRED: <^>what was the girl holding ?<~>the girl was be for the field
PRED: <^>what was the girl holding ?<~>the girl was holding a swing
PRED: <^>what was the girl doing ?<~>the man are tryings
PRED: <^>what is the girls are ?<~>the girls are wearing a brunette
PRED: <^>what was the girl holding ?<~>the girl was holding a peace man
PRED: <^>what is the girls are ?<~>the girls are wearing a older girl
PRED: <^>what is the girls are ?<~>the girls are wearing a older

我不确定我哪里出错了。我很确定网络正在从训练中学习，考虑到输出的构建方式，这是非常有前途的，但这里的主要问题是问题答案是由不在源句中的单词形成的。

有没有办法指示网络主要只使用源句中的单词？下面是解码器输出函数。

def symbols_to_logits_fn(model, config, decoder_tensor, debug=False):
    '''We basically need to run the complete decoder function
    :param model: namespace returned from function
    :param decoder_tensor: [batch_size * beam_size, decoded_length]
    :return new_ids: [batch_size * beam_size, vocab_size]
    '''
    print('^^^^^^ decoder_tensor: {}'.format(decoder_tensor))
        decoder_gather = tf.gather(
        model.context_embedding, decoder_tensor
    ) * (config.embedding_dim ** 0.5)
    decoder_gather += tf.gather(model.position_embedding,
                            positions_for(decoder_tensor, past_length=0))
    print('>>>>> {}'.format(decoder_gather))
    encoder_tiled = tf.tile(model.encoder_embedding, [config.beam_size, 1, 1])
    print('>>> encoder_tiled: {}'.format(encoder_tiled))
    local_decoder_pad_mask = tf.math.equal(
        decoder_tensor, config.pad_id, name='beam_decoder_pad_mask')
    print('>>>> local_decoder_pad_mask: {}'.format(local_decoder_pad_mask))
    decoder_out_func = transformer_model.decoder_fn(config=config,
                                                dec_out=decoder_gather,
                                                enc_out=encoder_tiled,
                                                encoder_pad_mask=model.encoder_pad_mask,
                                                decoder_pad_mask=local_decoder_pad_mask)  # [bs,   None, embedding_dim]
    print('>>>> decoder_out_func: {}'.format(decoder_out_func))
    # [bs, None, vocab_size]
    decoder_out = tf.matmul(decoder_out_func, model.fproj_w, transpose_b=False)
    print('>>>> decoder_out: {}'.format(decoder_out))
    decoder_out_last_step = decoder_out[:, -1, :]  # [bs, vocab_size]
    print('>>> decoder_out_last_step: {}'.format(decoder_out_last_step))
    return decoder_out_last_step

谁能帮我解决这个问题。我觉得我离放弃太近了。任何调整网络的帮助都会非常有帮助。

【问题讨论】：

标签： python tensorflow transformer attention-model

【解决方案1】：

15k 示例对于从头开始训练 Transformer 模型来说是非常少的。它的典型用途是机器翻译，其中训练语料库通常有数百万个句子对。

您可以尝试调整预训练模型：

Facebook's BART 是经过预训练的去噪 Transformer，用于语言生成和句子压缩。
Hugingface's Transformers 现在允许将预训练的编码器（如 BERT）与预训练的语言模型（如 GPT）相结合，您只能训练编码器-解码器的注意力，它们有 a tutorial on that。

【讨论】：