过滤图像中补丁位置的最优化方法

Question

所以我的问题是：我有一个 RGB 图像作为尺寸为 (4086, 2048, 3) 的 numpy 数组，我将此图像尺寸拆分为 96x96 块并在 numpy 中取回这些块的 位置大批。我总是在每种情况下都得到 96x96 的补丁。如果图像的尺寸不允许我在 x 或 y 轴上创建“纯”96x96 补丁，我只需为其添加一个左填充，以便最后一个补丁与之前的补丁有一点重叠。

现在有了这些位置，我想以尽可能快的方式在所有三个通道中为补丁中的每个像素删除所有 RGB 值为 255 的所有 96x96 补丁我想取回所有没有这个值的补丁位置。

我想知道：

1. 从图像维度中提取 96x96 补丁位置的最快方法是什么？ （现在我有一个 for 循环）
2. 如何以最佳方式去除纯白色斑块（3 个通道上的值为 255）？ （现在我有一个 for 循环）

我有很多这样的图像要处理，图像分辨率高达(39706, 94762, 3)，所以我的“for 循环”在这里很快变得效率低下。谢谢你的帮助！ （我也采用了使用 GPU 的解决方案）

下面是伪代码，让您了解一下它现在是如何完成的：

patches = []
patch_y = 0
y_limit = False
slide_width = 4086
slide_height = 2048
# Lets imagine this image_slide has 96x96 patches which value is 255
image_slide = np.random.rand(slide_width, slide_height, 3)
while patch_y < slide_height:
    patch_x = 0
    x_limit = False
    while patch_x < slide_width:
        # Extract the patch at the given position and return it or return None if it's 3 RGB
        # channels are 255
        is_white = PatchExtractor.is_white(patch_x, patch_y, image_slide)

        # Add the patches position to the list if it's not None (not white)
        if not is_white:
            patches.append((patch_x, patch_y))

        if not x_limit and patch_x + crop_size > slide_width - crop_size:
            patch_x = slide_width - crop_size
            x_limit = True
        else:
            patch_x += crop_size
    if not y_limit and patch_y + crop_size > slide_height - crop_size:
        patch_y = slide_height - crop_size
        y_limit = True
    else:
        patch_y += crop_size

return patches

理想情况下，我想将我的补丁位置放在“for循环”之外，然后一旦我有了它们，我就可以测试它们是否是白色的在for循环之外以及尽可能少的对numpy的调用（所以代码在numpy的C层处理，不会往来python）

Answer 1

正如您所怀疑的那样，您可以矢量化您正在做的所有事情。它大约需要原始图像的内存需求的小整数倍。该算法非常简单：填充您的图像，使整数个补丁适合其中，将其切割成补丁，检查每个补丁是否全是白色，保留其余部分：

import numpy as np                                                                    

# generate some dummy data and shapes
imsize = (1024, 2048)
patchsize = 96
image = np.random.randint(0, 256, size=imsize + (3,), dtype=np.uint8)
# seed some white patches: cut a square hole in the random noise
image[image.shape[0]//2:3*image.shape[0]//2, image.shape[1]//2:3*image.shape[1]//2] = 255

# pad the image to necessary size; memory imprint similar size as the input image
# white pad for simplicity for now
nx,ny = (np.ceil(dim/patchsize).astype(int) for dim in imsize) # number of patches
if imsize[0] % patchsize or imsize[1] % patchsize:
    # we need to pad along at least one dimension
    padded = np.pad(image, ((0, nx * patchsize - imsize[0]),
                            (0, ny * patchsize - imsize[1]), (0,0)),
                    mode='constant', constant_values=255)
else:
    # no padding needed
    padded = image

# reshape padded image according to patches; doesn't copy memory
patched = padded.reshape(nx, patchsize, ny, patchsize, 3).transpose(0, 2, 1, 3, 4) 
# patched is shape (nx, ny, patchsize, patchsize, 3)
# appending .copy() as a last step to the above will copy memory but might speed up
# the next step; time it to find out

# check for white patches; memory imprint the same size as the padded image
filt = ~(patched == 255).all((2, 3, 4))
# filt is a bool, one for each patch that tells us if it's _not_ all white
# (i.e. we want to keep it)

patch_x,patch_y = filt.nonzero() # patch indices of non-whites from 0 to nx-1, 0 to ny-1
patch_pixel_x = patch_x * patchsize  # proper pixel indices of each pixel
patch_pixel_y = patch_y * patchsize
patches = np.array([patch_pixel_x, patch_pixel_y]).T
# shape (npatch, 2) which is compatible with a list of tuples

# if you want the actual patches as well:
patch_images = patched[filt, ...]
# shape (npatch, patchsize, patchsize, 3),
# patch_images[i,...] is an image with patchsize * patchsize pixels

如您所见，在上图中，我使用白色填充来获得一致的填充图像。我相信这符合您正在尝试做的事情的哲学。如果您想完全复制您在循环中所做的事情，您可以使用边缘附近的重叠像素手动填充图像。您需要分配一个大小合适的填充图像，然后手动切片原始图像的重叠像素，以便在填充结果中设置边缘像素。

---

由于您提到您的图像很大，因此填充会导致过多的内存使用，您可以避免使用一些肘部油脂填充。您可以使用大图像的切片（不会创建副本），但是您必须手动处理没有完整切片的边缘。方法如下：

def get_patches(img, patchsize):
    """Compute patches on an input image without padding: assume "congruent" patches

    Returns an array shaped (npatch, 2) of patch pixel positions"""
    mx,my = (val//patchsize for val in img.shape[:-1])
    patched = img[:mx*patchsize, :my*patchsize, :].reshape(mx, patchsize, my, patchsize, 3)
    filt = ~(patched == 255).all((1, 3, 4))
    patch_x,patch_y = filt.nonzero() # patch indices of non-whites from 0 to nx-1, 0 to ny-1
    patch_pixel_x = patch_x * patchsize  # proper pixel indices of each pixel
    patch_pixel_y = patch_y * patchsize
    patches = np.stack([patch_pixel_x, patch_pixel_y], axis=-1)
    return patches

# fix the patches that fit inside the image
patches = get_patches(image, patchsize)

# fix edge patches if necessary
all_patches = [patches]
if imsize[0] % patchsize:
    # then we have edge patches along the first dim
    tmp_patches = get_patches(image[-patchsize:, ...], patchsize)
    # correct indices
    all_patches.append(tmp_patches + [imsize[0] - patchsize, 0])
if imsize[1] % patchsize:
    # same along second dim
    tmp_patches = get_patches(image[:, -patchsize:, :], patchsize)
    # correct indices
    all_patches.append(tmp_patches + [0, imsize[1] - patchsize])
if imsize[0] % patchsize and imsize[1] % patchsize:
    # then we have a corner patch we still have to fix
    tmp_patches = get_patches(image[-patchsize:, -patchsize:, :], patchsize)
    # correct indices
    all_patches.append(tmp_patches + [imsize[0] - patchsize, imsize[1] - patchsize])

# gather all the patches into an array of shape (npatch, 2)
patches = np.vstack(all_patches)

# if you also want to grab the actual patch values without looping:
xw, yw = np.mgrid[:patchsize, :patchsize]
patch_images = image[patches[:,0,None,None] + xw, patches[:,1,None,None] + yw, :]
# shape (npatch, patchsize, patchsize, 3),
# patch_images[i,...] is an image with patchsize * patchsize pixels

这也将完全复制您的循环代码，因为我们明确采用边缘补丁，使它们与之前的补丁重叠（没有虚假的白色填充）。但是，如果您想按给定顺序排列补丁，则必须现在对其进行排序。