如何使用 SharpDX 从网络摄像头捕获帧

Question

我正在尝试实现一个网络摄像头捕获应用程序，该应用程序应该拍摄静止帧，将它们显示在屏幕上并保存到磁盘。

由于我已经在使用 SharpDX 来捕获屏幕，我认为使用该库会很好。我不确定 SharpDX 是否有任何视频捕捉功能，所以我开始搜索并找到了它看起来像网络摄像头捕捉原型的部分内容：

var attributes = new MediaAttributes(1);
attributes.Set<Guid>(CaptureDeviceAttributeKeys.SourceType, CaptureDeviceAttributeKeys.SourceTypeVideoCapture.Guid);
var activates = MediaFactory.EnumDeviceSources(attributes);

var dic = new Dictionary<string, Activate>();
foreach (var activate in activates)
{
    var uid = activate.Get(CaptureDeviceAttributeKeys.SourceTypeVidcapSymbolicLink);
    dic.Add(uid, activate);
}

var camera = dic.First().Value;

它输出带有奇怪 uid 的 camera。我不确定它是否正确。

这之后我应该做什么？

编辑

我得到了这种代码的工作方式。我还是不明白为什么输出很奇怪。

var attributes = new MediaAttributes(1);
attributes.Set(CaptureDeviceAttributeKeys.SourceType.Guid, CaptureDeviceAttributeKeys.SourceTypeVideoCapture.Guid);

var mediaSource = MediaFactory.EnumDeviceSources(attributes)[0].ActivateObject<MediaSource>();
mediaSource.CreatePresentationDescriptor(out var presentationDescriptor);

var reader = new SourceReader(mediaSource);
var mediaTypeIndex = 0;

int width, height;

using (var mt = reader.GetNativeMediaType(0, mediaTypeIndex))
{
    UnpackLong(mt.Get(MediaTypeAttributeKeys.FrameSize), out  width, out  height);
    UnpackLong(mt.Get(MediaTypeAttributeKeys.FrameRate), out var frameRateNumerator, out var frameRateDenominator);
    UnpackLong(mt.Get(MediaTypeAttributeKeys.PixelAspectRatio), out var aspectRatioNumerator, out var aspectRatioDenominator);
}

var sample = reader.ReadSample(SourceReaderIndex.AnyStream, SourceReaderControlFlags.None, out var readStreamIndex, out var readFlags, out var timestamp);

if (sample == null)
    sample = reader.ReadSample(SourceReaderIndex.AnyStream, SourceReaderControlFlags.None, out readStreamIndex, out readFlags, out timestamp);


var sourceBuffer = sample.GetBufferByIndex(0); // sample.ConvertToContiguousBuffer();
var sourcePointer = sourceBuffer.Lock(out var maxLength, out var currentLength);

var data = new byte[sample.TotalLength];
Marshal.Copy(sourcePointer, data, 0, sample.TotalLength);

var newData = new byte[width * 4 * height];

var partWidth = width / 4;
var partHeight = height / 3;

for (var i = 0; i < sample.TotalLength; i += 4)
{
    //X8R8B8G8 -> BGRA = 4
    newData[i] = data[i + 3];
    newData[i + 1] = data[i + 2];
    newData[i + 2] = data[i + 1];
    newData[i + 3] = 255; //data[i];
}

//var source = BitmapSource.Create(width, height, 96, 96, PixelFormats.Bgra32, null, data, ((width * 24 + 31) / 32) * 4);
var source = BitmapSource.Create(width, height, 96, 96, PixelFormats.Bgra32, null, newData, width * 4);

sourceBuffer.Unlock();
sourceBuffer.Dispose();

输出图像是这样的（I was showing a color spectrum to my webcam）：

图像重复4次，每个部分都有灰度图像和一半高度的彩色版本。 图像的三分之二是透明的。

Answer 1

您的输出是 NV12，这里有一些将 nv12 转换为 rgb 的示例代码

    unsafe private static void TransformImage_NV12(IntPtr pDest, int lDestStride, IntPtr pSrc, int lSrcStride, int dwWidthInPixels, int dwHeightInPixels)
    {
        uint imageWidth = (uint)dwWidthInPixels;
        uint widthHalf = imageWidth / 2;
        uint imageHeight = (uint)dwHeightInPixels;

        byte* nv12Data = (byte*)pSrc;
        byte* rgbData = (byte*)pDest;

        uint dataSize = imageWidth * imageHeight * 3;

        for (uint y = 0; y < imageHeight; y++)
        {
            for (uint x = 0; x < imageWidth; x++)
            {
                uint xEven = x & 0xFFFFFFFE;
                uint yEven = y & 0xFFFFFFFE;
                uint yIndex = y * imageWidth + x;
                uint cIndex = imageWidth * imageHeight + yEven * widthHalf + xEven;

                byte yy = nv12Data[yIndex];
                byte cr = nv12Data[cIndex + 0];
                byte cb = nv12Data[cIndex + 1];

                uint outputIndex = (dataSize - (y * imageWidth + x) * 3) - 3;

                rgbData[outputIndex + 0] = (byte)Math.Min(Math.Max((yy + 1.402 * (cr - 128)), 0), 255);
                rgbData[outputIndex + 1] = (byte)Math.Min(Math.Max((yy - 0.344 * (cb - 128) - 0.714 * (cr - 128)), 0), 255);
                rgbData[outputIndex + 2] = (byte)Math.Min(Math.Max((yy + 1.772 * (cb - 128)), 0), 255);
            }
        }
    }