Kinect + Python - 填充阴影的深度

Question

Kinect 相机会返回整个视图的深度图像。由于捕获图像的方式，一些小区域对相机是不可见的。对于这些区域，返回 2047。

我想用剩下的值填充这些区域 - 这是该区域最可能的值。我的深度为 numpy uint16 数组。简单的解决方案是：

for x in xrange(depth.shape[1]):
  for y in xrange(depth.shape[0]):
    if depth[y,x] == 2047 and x > 0:
      depth[y,x] = depth[y,x-1]

每 640 x 480 帧执行 (Raspberry 2) 大约需要 16 秒。

我想出了一个使用索引的解决方案：

w = numpy.where(depth == 2047)
w = zip(w[0], w[1])
for index in w:
  if depth[index] == 2047 and index[1] > 0:
    depth[index] = depth[index[0],index[1] - 1]

执行一个测试帧大约需要 0.6 秒。更快，但仍远非完美。索引计算和 zip 只需要 0.04 秒，所以主要的性能杀手是循环。

我使用 item() 将它减少到 0.3 秒：

for index in w:
  if depth.item(index) == 2047 and index[1] > 0:
    depth.itemset(index, depth.item(index[0],index[1] - 1))

仅使用 python (+numpy/opencv) 可以进一步改进吗？相比于简单过滤的速度，应该可以快于0.05s

Answer 1

您有 islands 位于输入数组中元素为 2 的位置后面。这里的工作是在开始阴影之前用元素填充阴影。因此，一种方法是找出这些岛屿的起点和终点，并将x 和-x 分别放在这些位置，其中x 是每个岛屿开始之前的元素。然后，沿行执行cumsum，这将有效地用x 填充影子岛。这就是矢量化解决方案的全部内容！这是实现 -

# Get mask of places to be updated
mask = np.zeros(np.array(depth.shape) + [0,1],dtype=bool)
mask[:,1:-1] = depth[:,1:] == 2047

# Get differentiation along the second axis and thus island start and stops
diffs = np.diff(mask.astype(int),axis=1)
start_mask = diffs == 1
stop_mask = diffs == -1

# Get a mapping array that has island places filled with the start-1 element
map_arr = np.zeros_like(diffs)
map_arr[start_mask] = depth[start_mask]
map_arr[stop_mask] = -depth[start_mask]
map_filled_arr = map_arr.cumsum(1)[:,:-1]

# Use mask created earlier to selectively set elements from map array
valid_mask = mask[:,1:-1]
depth[:,1:][valid_mask] = map_filled_arr[valid_mask]

基准测试

定义函数：

def fill_depth_original(depth):
    for x in xrange(depth.shape[1]):
        for y in xrange(depth.shape[0]):
            if depth[y,x] == 2047 and x > 0:
                depth[y,x] = depth[y,x-1]

def fill_depth_original_v2(depth):
    w = np.where(depth == 2047)
    w = zip(w[0], w[1])
    for index in w:
      if depth[index] == 2047 and index[1] > 0:
        depth[index] = depth[index[0],index[1] - 1]

def fill_depth_vectorized(depth):

    mask = np.zeros(np.array(depth.shape) + [0,1],dtype=bool)
    mask[:,1:-1] = depth[:,1:] == 2047

    diffs = np.diff(mask.astype(int),axis=1)
    start_mask = diffs == 1
    stop_mask = diffs == -1

    map_arr = np.zeros_like(diffs)
    map_arr[start_mask] = depth[start_mask]
    map_arr[stop_mask] = -depth[start_mask]
    map_filled_arr = map_arr.cumsum(1)[:,:-1]

    valid_mask = mask[:,1:-1]
    depth[:,1:][valid_mask] = map_filled_arr[valid_mask]

运行时测试和验证输出：

In [303]: # Create a random array and get a copy for profiling vectorized method
     ...: depth = np.random.randint(2047-150,2047+150,(500,500))
     ...: depthc1 = depth.copy()
     ...: depthc2 = depth.copy()
     ...: 

In [304]: fill_depth_original(depth)
     ...: fill_depth_original_v2(depthc1)
     ...: fill_depth_vectorized(depthc2)
     ...: 

In [305]: np.allclose(depth,depthc1)
Out[305]: True

In [306]: np.allclose(depth,depthc2)
Out[306]: True

In [307]: # Create a random array and get a copy for profiling vectorized method
     ...: depth = np.random.randint(2047-150,2047+150,(500,500))
     ...: depthc1 = depth.copy()
     ...: depthc2 = depth.copy()
     ...: 

In [308]: %timeit fill_depth_original(depth)
     ...: %timeit fill_depth_original_v2(depthc1)
     ...: %timeit fill_depth_vectorized(depthc2)
     ...: 
10 loops, best of 3: 89.6 ms per loop
1000 loops, best of 3: 1.47 ms per loop
100 loops, best of 3: 10.3 ms per loop

所以，问题中列出的第二种方法看起来仍然很成功！