如何结合 CNN 和 LSTM？

Question

我对 LSTM 和 CNN 的概念还很陌生。所以，我目前已经分别定义了我的 LSTM 和 CNN：

LSTM：

def create_basic_rnn_model(config, output_size):      
    model = Sequential()
    model.add(LSTM(50,batch_input_shape=(config.BATCH_SIZE,config.LOOK_BACK,32),
                   return_sequences=False, stateful=config.STATEFUL, dropout=0.5, activation="softsign"))     
    model.add(Dense(output_size, activation="sigmoid"))
    
    model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
    model.summary()
    return model 

CNN：

model = Sequential()
model.add(Conv2D(filters=32, kernel_size=(5,5), padding='same', activation='relu', input_shape=(time_steps//subsample, 32, 1)))
model.add(MaxPooling2D(strides=2))
model.add(Conv2D(filters=48, kernel_size=(5,5), padding='valid', activation='relu'))
model.add(MaxPooling2D(strides=2))
model.add(Flatten())
model.add(Dense(256, activation='relu'))
model.add(Dense(84, activation='relu'))
model.add(Dense(6, activation='softmax'))

基于配置的数据输入：

class Config(object):
    #data are downsampled, each 15th data point is taken
    DOWNSAMPLING = 15
    LOOK_BACK = 35
    BATCH_SIZE = 512

    #Iterations on stateful model, epochs on non-stateful
    EPOCHS = 30
    ITERATIONS = 1
    STATEFUL = False
    SHUFFLE = not STATEFUL
    VERBOSE = 0    

模型将在哪里训练：

def train_rnn(subjects, labelToTrain, trainSeparateLabels, config, modelGenerator, callbacks): 
    #Calculating output size of trained model
    LabelsRange = [labelToTrain,labelToTrain+1]
    if(not trainSeparateLabels):
        LabelsRange = [0,6]
    OutputSize = LabelsRange[1] - LabelsRange[0]     
    #Model creation
    model = modelGenerator(config, output_size = OutputSize) 
    
    #Training
    for i in range(config.ITERATIONS):
        #For specified subjects
        for subject in subjects:
            #For series 1-7 
            for j in range(1,8):
                signals, labels = load_train_data_prepared(subject = subject,series = j) 
                #Create sequences
                X_train_signals, X_train_labels = create_sequences(
                    None,
                    None,
                    signals.iloc[::config.DOWNSAMPLING].values,
                    labels.iloc[::config.DOWNSAMPLING].values,
                    look_back=config.LOOK_BACK       
                )        

                X_train_labels = X_train_labels[:,LabelsRange[0]:LabelsRange[1]]
                croppedSize = math.floor(len(X_train_signals)/config.BATCH_SIZE)*config.BATCH_SIZE        
                #Train model on relevant label (calling fit repeatedly in keras doesnt reset the model)
                model.fit(
                    X_train_signals[0:croppedSize],
                    X_train_labels[0:croppedSize],
                    epochs=config.EPOCHS,
                    batch_size=config.BATCH_SIZE,
                    shuffle=config.SHUFFLE,
                    verbose=config.VERBOSE,
                    callbacks=callbacks
                )            
                if(config.STATEFUL):
                    model.reset_states()
    
    
    print("FITTING DONE")
    return model, LabelsRange

请问我将如何结合我的 LSTM 和 CNN？我已经搜索了几种方法，但它们都不起作用。我已经搜索了 ConvLSTM2D，但似乎无法将它们连接在一起。提前致谢。

Answer 1

1. 如果你的输入是时间序列，你应该使用 Conv1D：

   model = Sequential()

   model.add(Conv1D())

   model.add(LSTM())

   model.add(密集(softmax))

2. 如果您的输入是图像或视频序列，则应使用 ConvLSTM2D。