使用单词和字符 LSTM 进行文本预测

Question

我正在尝试实现具有字符和单词嵌入的 LSTM，如 here 所示，但我的问题不是 NER，只是简单的文本预测。现在我收到此错误：

   ValueError: Shapes (None, 135) and (None, 10, 135) are incompatible

这是我的模型摘要：

Model: "model_13"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_34 (InputLayer)           [(None, 10, 30)]     0                                            
__________________________________________________________________________________________________
input_33 (InputLayer)           [(None, 10)]         0                                            
__________________________________________________________________________________________________
time_distributed_43 (TimeDistri (None, 10, 30, 20)   2380        input_34[0][0]                   
__________________________________________________________________________________________________
embedding_16 (Embedding)        (None, 10, 128)      26887296    input_33[0][0]                   
__________________________________________________________________________________________________
time_distributed_44 (TimeDistri (None, 10, 20)       3280        time_distributed_43[0][0]        
__________________________________________________________________________________________________
concatenate_14 (Concatenate)    (None, 10, 148)      0           embedding_16[8][0]               
                                                                 time_distributed_44[0][0]        
__________________________________________________________________________________________________
spatial_dropout1d_14 (SpatialDr (None, 10, 148)      0           concatenate_14[0][0]             
__________________________________________________________________________________________________
bidirectional_14 (Bidirectional (None, 10, 100)      79600       spatial_dropout1d_14[0][0]       
__________________________________________________________________________________________________
time_distributed_45 (TimeDistri (None, 10, 135)      13635       bidirectional_14[0][0]           
==================================================================================================
Total params: 26,986,191
Trainable params: 98,895
Non-trainable params: 26,887,296
__________________________________________________________________________________________________

我的输入是X_word、X_char 和Y。 X_word 是编码单词的列表。每句10个字(2770, 10)和X.word[0]看起来是这样的：

array([[ 16871,      298,      0,      0,      0,      0,      0,      0,
             0,      0]])

这是一个有两个单词的填充句。

我的 X_char 是这些单词的字符列表：

array([[  7, 101,  16, 101,   0,   0,   0,   0,   0,   0,   0,   0,   0,
          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
          0,   0,   0,   0],
       [ 56, 102,  16,  34, 102,  61,   6, 102,  93,   0,   0,   0,   0,
          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
          0,   0,   0,   0],
       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
          0,   0,   0,   0],
       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
          0,   0,   0,   0],
       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
          0,   0,   0,   0],
       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
          0,   0,   0,   0],
       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
          0,   0,   0,   0],
       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
          0,   0,   0,   0],
       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
          0,   0,   0,   0],
       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
          0,   0,   0,   0]])

X_char 的形状为(2770, 10, 30)。

我有 135 个标签，所以 Y 的形状是 (2770,135)，我适合这样的所有东西：

history = model.fit([X_word_tr,
                    (np.array(X_char_tr)).astype('float32').reshape((len(X_char_tr), max_len, max_len_char))],
                    np.array(to_categorical(y_tr)), epochs=10, verbose=1)

我不禁认为我的逻辑在某个地方有缺陷。

Answer 1

如果您在链接的教程中定义了模型（您应该在帖子中包含该模型），那是因为它设置为返回一个序列，所以它返回接下来的 10 个字符。您需要设置 return_sequences=False 使其返回单个值，并移除包裹最后一个 Dense 层的最后一个 TimeDistributed 层。

main_lstm = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(units=50, return_sequences=False, 
                                                               recurrent_dropout=0.6))(x)
out = tf.keras.layers.Dense(n_tags + 1, activation="sigmoid")(main_lstm)

在摘要中，您将看到它返回单个值，因此可以将其与您的标签进行比较：

_______________________________________________________________________________________________
dense_5 (Dense)                 (None, 135)          13635       bidirectional_5[0][0]            
===============================================================================================