Kinect 1.8 colorframe 和 depthframe 不协调

Question

我的程序存在深度图像和彩色图像之间协调不佳的问题。

玩家面具与人不在同一个地方（见下图）。

void _AllFreamReady(object sender, AllFramesReadyEventArgs e)
{
    using (ColorImageFrame _colorFrame = e.OpenColorImageFrame())
    {
        if (_colorFrame == null)  //jezeli pusta ramka nie rob nic
        {
            return;
        }
        byte[] _pixels = new byte[_colorFrame.PixelDataLength]; //utworzenie tablicy pixeli dla 1 ramki obrazu o rozmiarach przechwyconej ramki z strumienia 
        _colorFrame.CopyPixelDataTo(_pixels);                   //kopiujemy pixele do tablicy
        int _stride = _colorFrame.Width * 4;                    //Kazdy pixel moze miec 4 wartosci Red Green Blue lub pusty
        image1.Source =
            BitmapSource.Create(_colorFrame.Width, _colorFrame.Height,
            96, 96, PixelFormats.Bgr32, null, _pixels, _stride);

        if (_closing)
        {
            return;
        }

        using (DepthImageFrame _depthFrame = e.OpenDepthImageFrame())
        {
            if (_depthFrame == null)
            {
                return;
            }

            byte[] _pixelsdepth = _GenerateColoredBytes(_depthFrame,_pixels);
            int _dstride = _depthFrame.Width * 4;
            image3.Source =
                BitmapSource.Create(_depthFrame.Width, _depthFrame.Height,
                96, 96, PixelFormats.Bgr32, null, _pixelsdepth, _dstride);
        }
    }               
}

private byte[] _GenerateColoredBytes(DepthImageFrame _depthFrame, byte[] _pixels)
{
    short[] _rawDepthData = new short[_depthFrame.PixelDataLength];
    _depthFrame.CopyPixelDataTo(_rawDepthData);
    Byte[] _dpixels = new byte[_depthFrame.Height * _depthFrame.Width * 4];
    const int _blueindex = 0;
    const int _greenindex = 1;
    const int _redindex = 2;

    for (int _depthindex = 0, _colorindex = 0;
        _depthindex < _rawDepthData.Length && _colorindex < _dpixels.Length;
        _depthindex++, _colorindex += 4)
    {
        int _player = _rawDepthData[_depthindex] & DepthImageFrame.PlayerIndexBitmaskWidth;

        if (_player > 0)
        {
            _dpixels[_colorindex + _redindex] = _pixels[_colorindex + _redindex]; 
            _dpixels[_colorindex + _greenindex] = _pixels[_colorindex + _greenindex];
            _dpixels[_colorindex + _blueindex] = _pixels[_colorindex + _blueindex];

        };
    }

    return _dpixels;
}

Answer 1

RGB 和深度数据未对齐。这是由于 Kinect 案例中深度传感器和 RGB 摄像头的位置：它们是不同的，因此您不能期望使用不同的视角对齐图像。

但是，您的问题很常见，并且已由 KinectSensor.MapDepthFrameToColorFrame 解决，该 KinectSensor.MapDepthFrameToColorFrame 在 SDK 1.6 之后已被弃用。现在，您需要的是CoordinateMapper.MapDepthFrameToColorFrame method。

Coordinate Mapping Basics-WPF C# Sample 展示了如何使用此方法。您可以在下面找到代码的一些重要部分：

// Intermediate storage for the depth data received from the sensor
private DepthImagePixel[] depthPixels;
// Intermediate storage for the color data received from the camera
private byte[] colorPixels;
// Intermediate storage for the depth to color mapping
private ColorImagePoint[] colorCoordinates;
// Inverse scaling factor between color and depth
private int colorToDepthDivisor;
// Format we will use for the depth stream
private const DepthImageFormat DepthFormat = DepthImageFormat.Resolution320x240Fps30;
// Format we will use for the color stream
private const ColorImageFormat ColorFormat = ColorImageFormat.RgbResolution640x480Fps30;

//...

// Initialization
this.colorCoordinates = new ColorImagePoint[this.sensor.DepthStream.FramePixelDataLength];
this.depthWidth = this.sensor.DepthStream.FrameWidth;
this.depthHeight = this.sensor.DepthStream.FrameHeight;
int colorWidth = this.sensor.ColorStream.FrameWidth;
int colorHeight = this.sensor.ColorStream.FrameHeight;
this.colorToDepthDivisor = colorWidth / this.depthWidth;
this.sensor.AllFramesReady += this.SensorAllFramesReady;

//...

private void SensorAllFramesReady(object sender, AllFramesReadyEventArgs e)
{
    // in the middle of shutting down, so nothing to do
    if (null == this.sensor)
    {
        return;
    }

    bool depthReceived = false;
    bool colorReceived = false;

    using (DepthImageFrame depthFrame = e.OpenDepthImageFrame())
    {
        if (null != depthFrame)
        {
            // Copy the pixel data from the image to a temporary array
            depthFrame.CopyDepthImagePixelDataTo(this.depthPixels);

            depthReceived = true;
        }
    }

    using (ColorImageFrame colorFrame = e.OpenColorImageFrame())
    {
        if (null != colorFrame)
        {
            // Copy the pixel data from the image to a temporary array
            colorFrame.CopyPixelDataTo(this.colorPixels);

            colorReceived = true;
        }
    }

    if (true == depthReceived)
    {
        this.sensor.CoordinateMapper.MapDepthFrameToColorFrame(
            DepthFormat,
            this.depthPixels,
            ColorFormat,
            this.colorCoordinates);

        // ...

        int depthIndex = x + (y * this.depthWidth);
        DepthImagePixel depthPixel = this.depthPixels[depthIndex];

        // scale color coordinates to depth resolution
        int X = colorImagePoint.X / this.colorToDepthDivisor;
        int Y = colorImagePoint.Y / this.colorToDepthDivisor;

        // depthPixel is the depth for the (X,Y) pixel in the color frame
    }
}

Answer 2

我自己正在解决这个问题。我同意 VitoShadow 的一种解决方案是在坐标映射中，但未发布的部分未发布匹配深度和彩色屏幕分辨率之间的比率（this.colorToDepthDivisor = colorWidth / this.depthWidth;）。这与数据移位 (this.playerPixelData[playerPixelIndex - 1] = opaquePixelValue;) 一起使用，以解决未匹配的问题。

不幸的是，这可能会在蒙版图像周围创建一个边框，其中 depthframe 没有拉伸到 color frame 的边缘。我试图不使用骨架映射，并通过使用 emgu cv 跟踪深度数据来优化我的代码，以传递一个点作为色框 ROI 的中心。我还在努力。