使用 Conv2DTranspose 输出其输入形状的双倍答案

【问题标题】：Using Conv2DTranspose to output the double of its input shape使用 Conv2DTranspose 输出其输入形状的双倍
【发布时间】：2020-05-15 10:18:17
【问题描述】：

我是 Python 3.7.7 和 Tensorflow 2.1.0 的新手，我正在努力理解 Conv2DTranspose。我试过这段代码：

def vgg16_decoder(input_size = (7, 7, 512)):
    inputs = Input(input_size, name = 'input')

    conv1 = Conv2DTranspose(512, (2, 2), dilation_rate = 2, name = 'conv1')(inputs)

    model = Model(inputs = inputs, outputs = conv1, name = 'vgg-16_decoder')

    opt = Adam(lr=0.001)
    model.compile(optimizer=opt, loss=keras.losses.categorical_crossentropy, metrics=['accuracy'])

    return model

这是它的摘要：

Model: "vgg-16_decoder"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input (InputLayer)           (None, 7, 7, 512)         0
_________________________________________________________________
conv1 (Conv2DTranspose)      (None, 9, 9, 512)         1049088
=================================================================
Total params: 1,049,088
Trainable params: 1,049,088
Non-trainable params: 0
_________________________________________________________________

但我想从conv1 输出(None, 14, 14, 512)。

我已将过滤器大小更改为(3, 3)，并得到以下摘要：

Model: "vgg-16_decoder"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input (InputLayer)           (None, 7, 7, 512)         0
_________________________________________________________________
conv1 (Conv2DTranspose)      (None, 11, 11, 512)       2359808
=================================================================
Total params: 2,359,808
Trainable params: 2,359,808
Non-trainable params: 0
_________________________________________________________________

我正在尝试使用Conv2DTranspose：

# A piece of code from U-NET implementation

up6 = Conv2D(512, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal', name = 'up6')(UpSampling2D(size = (2,2), name = 'upsp1')(drop5))

及其摘要：

drop5 (Dropout)                 (None, 16, 16, 1024) 0           conv5_2[0][0]
__________________________________________________________________________________________________
upsp1 (UpSampling2D)            (None, 32, 32, 1024) 0           drop5[0][0]
__________________________________________________________________________________________________
up6 (Conv2D)                    (None, 32, 32, 512)  2097664     upsp1[0][0]
__________________________________________________________________________________________________

它将输入上采样 2 并更改其过滤器的数量。

如何使用 Conv2DTranspose 做到这一点？

更新：

我认为，或者我想，我做到了，但我不明白我做了什么：

conv1 = Conv2DTranspose(512, (2, 2), strides = 2, name = 'conv1')(inputs)

根据前面的说法，我得到了这样的总结：

Model: "vgg-16_decoder"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input (InputLayer)           (None, 7, 7, 512)         0
_________________________________________________________________
conv1 (Conv2DTranspose)      (None, 14, 14, 512)       1049088
=================================================================
Total params: 1,049,088
Trainable params: 1,049,088
Non-trainable params: 0
_________________________________________________________________

如果您想纠正我或解释我在这里所做的事情，欢迎您。

更新 2：

顺便说一句，我正在尝试创建一个 VGG-16 解码器。这是我的 VGG-16 编码器的代码：

def vgg16_encoder(input_size = (224,224,3)):
    inputs = Input(input_size, name = 'input')

    conv1 = Conv2D(64, (3, 3), activation = 'relu', padding = 'same', name ='conv1_1')(inputs)
    conv1 = Conv2D(64, (3, 3), activation = 'relu', padding = 'same', name ='conv1_2')(conv1)
    pool1 = MaxPooling2D(pool_size = (2,2), strides = (2,2), name = 'pool_1')(conv1)

    conv2 = Conv2D(128, (3, 3), activation = 'relu', padding = 'same', name ='conv2_1')(pool1)
    conv2 = Conv2D(128, (3, 3), activation = 'relu', padding = 'same', name ='conv2_2')(conv2)
    pool2 = MaxPooling2D(pool_size = (2,2), strides = (2,2), name = 'pool_2')(conv2)

    conv3 = Conv2D(256, (3, 3), activation = 'relu', padding = 'same', name ='conv3_1')(pool2)
    conv3 = Conv2D(256, (3, 3), activation = 'relu', padding = 'same', name ='conv3_2')(conv3)
    conv3 = Conv2D(256, (3, 3), activation = 'relu', padding = 'same', name ='conv3_3')(conv3)
    pool3 = MaxPooling2D(pool_size = (2,2), strides = (2,2), name = 'pool_3')(conv3)

    conv4 = Conv2D(512, (3, 3), activation = 'relu', padding = 'same', name ='conv4_1')(pool3)
    conv4 = Conv2D(512, (3, 3), activation = 'relu', padding = 'same', name ='conv4_2')(conv4)
    conv4 = Conv2D(512, (3, 3), activation = 'relu', padding = 'same', name ='conv4_3')(conv4)
    pool4 = MaxPooling2D(pool_size = (2,2), strides = (2,2), name = 'pool_4')(conv4)

    conv5 = Conv2D(512, (3, 3), activation = 'relu', padding = 'same', name ='conv5_1')(pool4)
    conv5 = Conv2D(512, (3, 3), activation = 'relu', padding = 'same', name ='conv5_2')(conv5)
    conv5 = Conv2D(512, (3, 3), activation = 'relu', padding = 'same', name ='conv5_3')(conv5)
    pool5 = MaxPooling2D(pool_size = (2,2), strides = (2,2), name = 'pool_5')(conv5)

    opt = Adam(lr=0.001)

    model = Model(inputs = inputs, outputs = pool5, name = 'vgg-16_encoder')

    model.compile(optimizer=opt, loss=keras.losses.categorical_crossentropy, metrics=['accuracy'])

    return model

【问题讨论】：

标签： python tensorflow conv-neural-network

【解决方案1】：

当我们设计编码器-解码器架构时，我们需要一些操作来反转已经完成的操作。所以，假设在编码器中我们有 Conv2D 和池化（在 VGG 等架构中很常见）。我们使用 Conv2dTranspose（这可以认为是 Conv2D 的逆向操作）和 Upsampling2D（Pooling 的逆向操作（好吧，不严格 [pooling 是一种不可逆操作，因为信息丢失了]））。

注意：您不想使用 Conv2DTranspose 对特征图进行上采样（可以，但对于 VGG，我认为 Conv2DTranspose 不会按照您在解码器中想要的方式提供上采样的特征图），它不是设计的这样（它也学习了上采样，但它学习了稍微不同的最佳上采样参数）。您最终会得到非常大的内核，这将导致与您正在谈论的 VGG 编码器完全不同的网络。

from tensorflow.keras.layers import *
from tensorflow.keras.models import *

def encoder_decoder_conv(input_size = (224,224,3)):
    ip = Input((224,224,3))
    # encoder
    conv = Conv2D(512, (3,3))(ip) # look here, the default padding is used
    # decoder
    inv_conv = Conv2DTranspose(3, (3,3))(conv)
    # simple model
    model = Model(ip, inv_conv)
    return model

model1 = encoder_decoder_conv()
model1.summary()

def encoder_decoder_pooling(input_size = (224,224,3)):
    ip = Input((224,224,3))
    # encoder
    pool = MaxPool2D((2,2))(ip) # look here, the default padding is used
    # decoder
    inv_pool = UpSampling2D((2,2))(pool)
    # simple model
    model = Model(ip, inv_pool)
    return model

model2 = encoder_decoder_pooling()
model2.summary()

Model: "model_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         [(None, 224, 224, 3)]     0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 222, 222, 512)     14336     
_________________________________________________________________
conv2d_transpose_1 (Conv2DTr (None, 224, 224, 3)       13827     
=================================================================
Total params: 28,163
Trainable params: 28,163
Non-trainable params: 0
_________________________________________________________________
Model: "model_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_3 (InputLayer)         [(None, 224, 224, 3)]     0         
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 112, 112, 3)       0         
_________________________________________________________________
up_sampling2d (UpSampling2D) (None, 224, 224, 3)       0         
=================================================================
Total params: 0
Trainable params: 0
Non-trainable params: 0

正如您在第一个模型中看到的那样，使用 Conv2DTranspose 我们反转操作以获得与输入 (224,224,3) 完全相同的形状。

对于模型 2，我们使用上采样反转池化操作（就特征图形状而言）。

因此，当您尝试制作 VGG 解码器时，并且 VGG 主要由 Conv2D 和 Maxpooling2D 组成，您只需使用 Conv2dTranspose 和 Upsampling 反转这些操作，以便获得准确的输入形状（224、224、 3) 来自特征图形状 (7, 7, 512)。

最后，解码器部分有一些变化，但我认为您正在寻找这个 VGG-16 解码器。

def vgg16_decoder(input_size = (7,7,512)):
    inputs = Input(input_size, name = 'input')

    pool5 = UpSampling2D((2,2), name = 'pool_5')(inputs)
    conv5 = Conv2DTranspose(512, (3, 3), activation = 'relu', padding = 'same', name ='conv5_3')(pool5)

    conv5 = Conv2DTranspose(512, (3, 3), activation = 'relu', padding = 'same', name ='conv5_2')(conv5)

    conv5 = Conv2DTranspose(512, (3, 3), activation = 'relu', padding = 'same', name ='conv5_1')(conv5)

    pool4 = UpSampling2D((2,2), name = 'pool_4')(conv5)

    conv4 = Conv2DTranspose(512, (3, 3), activation = 'relu', padding = 'same', name ='conv4_3')(pool4)

    conv4 = Conv2DTranspose(512, (3, 3), activation = 'relu', padding = 'same', name ='conv4_2')(conv4)
    conv4 = Conv2DTranspose(512, (3, 3), activation = 'relu', padding = 'same', name ='conv4_1')(conv4)
    pool3 = UpSampling2D((2,2), name = 'pool_3')(conv4)

    conv3 = Conv2DTranspose(256, (3, 3), activation = 'relu', padding = 'same', name ='conv3_3')(pool3)
    conv3 = Conv2DTranspose(256, (3, 3), activation = 'relu', padding = 'same', name ='conv3_2')(conv3)

    conv3 = Conv2DTranspose(256, (3, 3), activation = 'relu', padding = 'same', name ='conv3_1')(conv3)

    pool2 = UpSampling2D((2,2), name = 'pool_2')(conv3)
    conv2 = Conv2DTranspose(128, (3, 3), activation = 'relu', padding = 'same', name ='conv2_2')(pool2)

    conv2 = Conv2DTranspose(128, (3, 3), activation = 'relu', padding = 'same', name ='conv2_1')(conv2)

    pool1 = UpSampling2D((2,2), name = 'pool_1')(conv2)

    conv1 = Conv2DTranspose(64, (3, 3), activation = 'relu', padding = 'same', name ='conv1_2')(pool1)

    conv1 = Conv2DTranspose(3, (3, 3), activation = 'relu', padding = 'same', name ='conv1_1')(conv1) # to get 3 channels

    model = Model(inputs = inputs, outputs = conv1, name = 'vgg-16_encoder')

    model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

    return model

model = vgg16_decoder()
model.summary()

Model: "vgg-16_encoder"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input (InputLayer)           [(None, 7, 7, 512)]       0         
_________________________________________________________________
pool_5 (UpSampling2D)        (None, 14, 14, 512)       0         
_________________________________________________________________
conv5_3 (Conv2DTranspose)    (None, 14, 14, 512)       2359808   
_________________________________________________________________
conv5_2 (Conv2DTranspose)    (None, 14, 14, 512)       2359808   
_________________________________________________________________
conv5_1 (Conv2DTranspose)    (None, 14, 14, 512)       2359808   
_________________________________________________________________
pool_4 (UpSampling2D)        (None, 28, 28, 512)       0         
_________________________________________________________________
conv4_3 (Conv2DTranspose)    (None, 28, 28, 512)       2359808   
_________________________________________________________________
conv4_2 (Conv2DTranspose)    (None, 28, 28, 512)       2359808   
_________________________________________________________________
conv4_1 (Conv2DTranspose)    (None, 28, 28, 512)       2359808   
_________________________________________________________________
pool_3 (UpSampling2D)        (None, 56, 56, 512)       0         
_________________________________________________________________
conv3_3 (Conv2DTranspose)    (None, 56, 56, 256)       1179904   
_________________________________________________________________
conv3_2 (Conv2DTranspose)    (None, 56, 56, 256)       590080    
_________________________________________________________________
conv3_1 (Conv2DTranspose)    (None, 56, 56, 256)       590080    
_________________________________________________________________
pool_2 (UpSampling2D)        (None, 112, 112, 256)     0         
_________________________________________________________________
conv2_2 (Conv2DTranspose)    (None, 112, 112, 128)     295040    
_________________________________________________________________
conv2_1 (Conv2DTranspose)    (None, 112, 112, 128)     147584    
_________________________________________________________________
pool_1 (UpSampling2D)        (None, 224, 224, 128)     0         
_________________________________________________________________
conv1_2 (Conv2DTranspose)    (None, 224, 224, 64)      73792     
_________________________________________________________________
conv1_1 (Conv2DTranspose)    (None, 224, 224, 3)       1731      
=================================================================
Total params: 17,037,059
Trainable params: 17,037,059
Non-trainable params: 0

它采用(7, 7, 512)特征形状并重构原始图像尺寸(224, 224, 3)。

总之，设计解码器的机械方式将在执行反向操作时沿相反的方向（相对于编码器）进行。至于Conv2DTranspose和Upsampling2D的细节，如果你想真正更深入地理解这些概念：

https://cs231n.github.io/convolutional-networks/

https://datascience.stackexchange.com/questions/6107/what-are-deconvolutional-layers

https://www.matthewzeiler.com/mattzeiler/deconvolutionalnetworks.pdf

【讨论】：

非常感谢。因此，阅读您的 N.B. 评论，我们的想法是使用 UpSampling2D 和 Conv2D 来获取解码器，不是吗？
@VansFannel 是的，我还添加了一个可能的解码器的实现，你可以检查一下。
是的，我已经检查过了，但是在你的解码器实现中你使用 Conv2DTranspose，这就是我问你使用 Conv2D 的原因。非常感谢。
在NB中我实际上提到了conv2dtranspose，而不是解码器的conv2d。从技术上讲，您也可以在 vgg 示例中的解码器中使用 Conv2D（因为您使用了相同的填充），但是对于存在有效填充的情况，在这些情况下 Conv2DTranspose 可以解决问题。
好的。我误解了你的NB。评论。对不起。

【解决方案2】：

要得到你需要的表格

conv1 = Conv2DTranspose(512, (8, 8), strides = 1, name = 'conv1')(inputs)

你可能会发现这篇关于转置卷积操作的帖子很有用https://towardsdatascience.com/types-of-convolutions-in-deep-learning-717013397f4d

【讨论】：

这就是他们所谓的反卷积，看看它是如何实现的很有趣！
帖子其实讲了转置和反卷积的区别。本质上，转置卷积是一种正常的卷积，其中输入以一种奇特的方式被大量填充。反卷积不同，它是关于找到逆矩阵。
这不是将图像大小加倍的一般方法。它仅适用于这组特定的参数。想象一下，如果他们想将 128x128 的图像转换为 256x256 的图像，他们需要使用大小为 129x129 的滤镜吗？
谢谢。但我不明白为什么kernel_size 必须是(8, 8) 而strides 是1。
基本上我是这样想的：我需要什么整数过滤器大小f 和步幅我需要得到(14 - f)/s + 1= 7。由于这里所有内容都必须是整数，因此只有两个解决方案退出：f = 8, s = 1 和 f = 2, s = 2。

【解决方案3】：

conv1 = Conv2DTranspose(512, (2, 2), strides = 2, name = 'conv1')(inputs) 有效，因为您使用的步幅为 2。在正常卷积中，这意味着仅每两步应用过滤器（每次跳过一步），这将导致输出大小为输入。然而，在转置卷积中，事情基本上是相反的，步幅为 2 会使输出大小翻倍。它通过在应用卷积之前基本上在输入中插入孔来实现这一点。

第一个 sn-p (conv1 = Conv2DTranspose(512, (2, 2), dilation_rate = 2, name = 'conv1')(inputs)) 不起作用，因为您指定的 dilation 为 2，而不是步幅。这是完全不同的。膨胀会在你的 filter 中插入“洞”，例如一个看起来像 [x1 x2 x3] 的过滤器变成了 [x1 0 x2 0 x3]，膨胀为 2。但是，这个带有孔的过滤器随后会照常应用于输入。

为什么即使使用dilation，输出大小也会发生变化？这是由于填充。通常，如果不进行填充，卷积的输出会更小。在转置卷积中，它会更大。您可以使用padding=same 来避免这种情况。

tl;dr：您可以使用 Conv2DTranspose(n_filters, filter_size, strides = 2, padding="same") 将图像大小加倍。

【讨论】：