如何从 VideoMediaFrame.Direct3DSurface 获取像素数据？

Question

我为 Micorsoft HoloLens 2 创建 Unity 应用程序。

该应用使用 Windows.Media.Capture.MediaCapture 和 Microsoft.MixedReality.WebRTC 捕获和共享相机视频帧。

我在 MediaFrameReader.FrameArrived 事件中得到了一个由“NV12”格式化的 VideoMediaFrame.Direct3DSurface(Windows.Graphics.DirectX.Direct3D11.IDirect3DSurface)。

但我不知道如何从 IDirect3DSurface 对象中获取像素数据。

我在下面尝试过：

C# 代码

[DllImport("Direct3DSurfaceAccess")]
private static extern void Direct3DSurfaceAccess([MarshalAs(UnmanagedType.Interface)]Windows.Graphics.DirectX.Direct3D11.IDirect3DSurface d3dSurface);

// MediaFrameReader.FrameArrived event.
private void Nv12FrameArrived(MediaFrameReader sender, MediaFrameArrivedEventArgs args) {
  using var mediaFrameReference = sender.TryAcquireLatestFrame();

  // I could't find how to get pixel data from IDirect3DSurface using C# only.
  // Use C++/WinRT for access IDirect3DSurface as below.
  Direct3DSurfaceAccess(mediaFrameReference.VideoMediaFrame.Direct3DSurface);
}

C++ 代码

void __stdcall Direct3DSurfaceAccess(const winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DSurface* d3dSurface) {
  // Here, confirmed that d3dSurface is not null.

  try {
    // Issue is here.

    winrt::com_ptr<::Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess> dxgiInterfaceAccess;
    d3dSurface->try_as(dxgiInterfaceAccess);
      // App crashes on d3dSurface->try_as.

    // Call other d3dSurface method as a test.
    winrt::Windows::Graphics::DirectX::Direct3D11::Direct3DSurfaceDescription desc = d3dSurface->Description();
      // Same above. App crashes on d3dSurface->Description.

    // Unreachable to here...
    // When succeeded to get the IDirect3DDxgiInterfaceAccess, read pixel data by below code.

    winrt::com_ptr<::IDXGISurface> nativeSurface;
    dxgiInterfaceAccess->GetInterface(IID_PPV_ARGS(nativeSurface.put()));

    ::DXGI_MAPPED_RECT rect;
    nativeSurface->Map(&rect, DXGI_MAP_READ);
    const BYTE* pixels = rect.pBits; // Read pixel data from the pointer.
    nativeSurface->Unmap();
  } catch (...) {
    // Can't catch any exceptions.
    // App freezes and downs on HoloLens 2.
  }
}

我认为如何编组是错误的IDirect3DSurface。

如何从IDirect3DSurface获取像素数据？

Reproducing sample

Answer 1

您不能在 dll 边界定义 C++/WinRT 对象指针，因为在二进制级别，它不是 COM 对象（因此 C# 传递一个映射到 C++/WinRT 对象的 COM 对象，这很糟糕） .你可以做的是这样的：

void __stdcall Direct3DSurfaceAccess(
    IUnknown* surface, // pass a raw COM object (here I use IUnknown)
    char* message,
    size_t messageLength
) {
    try {
        // map to a C++/WinRT object here
        auto d3dSurface = convert_from_abi<winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DSurface>(surface);

        winrt::com_ptr<::Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess> dxgiInterfaceAccess;
        if (!d3dSurface.try_as(dxgiInterfaceAccess)) {
            ::sprintf_s(message, messageLength, "d3dSurface->try_as failed.");
            return;
        }

        // Call other d3dSurface method as a test.
        auto desc = d3dSurface.Description();

convert_from_abi来自这里convert_from_abi function

在这种情况下，您可以像这样定义互操作函数：

[DllImport(@"Direct3DSurfaceAccess.dll")]
private static extern void Direct3DSurfaceAccess([MarshalAs(UnmanagedType.IUnknown)] object d3dSurface);