使用 gridsampler 进行 PyTorch 图像转换，优化网格时的奇怪行为答案

【问题标题】：PyTorch Image transformation with gridsampler, weird behaviour while optimizing grid使用 gridsampler 进行 PyTorch 图像转换，优化网格时的奇怪行为
【发布时间】：2019-11-13 15:44:54
【问题描述】：

我试图让每像素转换以使一个图像（+背景）适合结果。 Background Image + Input image 应该转换成desired result

为了实现这一点，我使用 PyTorch gridsampler 和 autograd 来优化网格。转换后的输入将添加到未更改的背景中。


ToTensor = torchvision.transforms.ToTensor()
FromTensor = torchvision.transforms.ToPILImage()

backround= ToTensor(Image.open("backround.png"))
pic = ToTensor(Image.open("pic.png"))
goal = ToTensor(Image.open("goal.png"))

empty = empty.expand(1, 3, empty.size()[1], empty.size()[2])
pic = pic.expand(1, 3, pic.size()[1], pic.size()[2])
goal = goal.expand(1, 3, goal.size()[1], goal.size()[2])

def createIdentityGrid(w, h):
    grid = torch.zeros(1, w, h, 2);
    for x in range(w):
        for y in range(h):
            grid[0][x][y][1] = 2 / w * (0.5 + x) - 1
            grid[0][x][y][0] = 2 / h * (0.5 + y) - 1
    return grid

w = 256; h=256 #hardcoded imagesize

grid = createIdentityGrid(w, h)
grid.requires_grad = True

for i in range(300):
    goal_pred = torch.nn.functional.grid_sample(pic, grid)[0]
    goal_pred = (empty + 0.75 * goal_pred).clamp(min=0, max=1)
    out = goal_pred

    loss = (goal_pred - goal).pow(2).sum()
    loss.backward()

    with torch.no_grad():
        grid -= grid.grad * (1e-2)
        grid.grad.zero_()

FromTensor(out[0]).show()

结果如下：

实际上是在使用这个简单的示例，但我观察到一些奇怪的行为。网格刚刚开始在一侧发生变化。为什么会这样，为什么整个网格不会立即改变？有没有我遗漏的明显部分？

【问题讨论】：

标签： python pytorch autograd

【解决方案1】：

from PIL import Image
import torch

ToTensor = torchvision.transforms.ToTensor()
FromTensor = torchvision.transforms.ToPILImage()

lr = 1
backround= ToTensor(Image.open(r"C:\Users\dj\Pictures\Saved Pictures\background.png"))
pic = ToTensor(Image.open(r"C:\Users\dj\Pictures\Saved Pictures\input.png"))
goal = ToTensor(Image.open(r"C:\Users\dj\Pictures\Saved Pictures\result.png"))

empty = backround.expand(1, 3, backround.size()[1], backround.size()[2])
pic = pic.expand(1, 3, pic.size()[1], pic.size()[2])
goal = goal.expand(1, 3, goal.size()[1], goal.size()[2])

def createIdentityGrid(w, h):
    grid = torch.zeros(1, w, h, 2);
    for x in range(w):
        for y in range(h):
            grid[0][x][y][1] = 2 / w * (0.5 + y) - 1
            grid[0][x][y][0] = 2 / h * (0.5 + x) - 1
    return grid

w = 256; h=256 #hardcoded imagesize

grid = createIdentityGrid(w, h)
grid.requires_grad = True

for i in range(9):
    goal_pred = torch.nn.functional.grid_sample(pic, grid, mode="bilinear")[0]
    goal_pred = F.relu(empty + 0.75 * goal_pred)
    out = goal_pred

    loss = (goal_pred - goal).pow(2).sum()
    loss.backward()

    with torch.no_grad():
        grid -= grid.grad * lr
        lr = lr/1.1 #learning rate a0-ing
        grid.grad.zero_()

FromTensor(out[0]).show()

实际上是在使用这个简单的示例，但我观察到一些奇怪的行为。网格刚刚开始在一侧发生变化。为什么会这样，为什么整个网格不会立即改变？

我不知道。我只是欺骗了你的例子，对我来说它是自下而上的。

【讨论】：