MTKView 以低于 AVCaptureVideoPreviewLayer 的分辨率显示摄像机源

Question

我正在尝试将摄像机源流式传输到 MTKView 中，以便将一些 CI 过滤器应用于实时流。 在初始化捕获会话并布局 MTKView 后，这是我设置 metal 的方式（metalView 是 MTKview）：

func setupMetal(){
    
    metalDevice = MTLCreateSystemDefaultDevice()
    metalView.device = metalDevice
    // Write when asked
    metalView.isPaused = true
    metalView.enableSetNeedsDisplay = false
    // Command queue for the GPU
    metalCommandQueue = metalDevice.makeCommandQueue()
    // Assign the delegate
    metalView.delegate = self
    // ???
    metalView.framebufferOnly = false
}

然后我从 SampleBufferDelegate 中抓取帧并获得一个 CIImage

extension ViewController: AVCaptureVideoDataOutputSampleBufferDelegate {

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    //try and get a CVImageBuffer out of the sample buffer
    guard let cvBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
        return
    }
    
    //get a CIImage out of the CVImageBuffer
    let ciImage = CIImage(cvImageBuffer: cvBuffer)
    
    self.currentCIImage = ciImage
    
    // We draw to the metal view everytime we receive a frame
    metalView.draw()
            
}}

然后我使用 currentCIImage 使用它的委托方法在 MTKView 中绘制：

extension ViewController : MTKViewDelegate {

func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
    //tells us the drawable's size has changed
}

func draw(in view: MTKView) {
    //create command buffer for ciContext to use to encode it's rendering instructions to the GPU
    guard let commandBuffer = metalCommandQueue.makeCommandBuffer() else {
        return
    }
    
    //make sure we actually have a ciImage to work with
    guard let ciImage = currentCIImage else {
        return
    }
    
    //make sure the current drawable object for this metal view is available (it's not in use by the previous draw cycle)
    guard let currentDrawable = view.currentDrawable else {
        return
    }
    
    //render into the metal texture
    // Check here if we find a more elegant solution for the bounds
    self.ciContext.render(ciImage,
                          to: currentDrawable.texture,
                          commandBuffer: commandBuffer,
                          bounds: CGRect(origin: .zero, size: view.drawableSize),
                          colorSpace: CGColorSpaceCreateDeviceRGB())
    
    //register where to draw the instructions in the command buffer once it executes
    commandBuffer.present(currentDrawable)
    //commit the command to the queue so it executes
    commandBuffer.commit()
}
}

它工作正常，我能够从 MTKView 中渲染的相机获取帧。但是，我注意到我没有获得完整的分辨率，图像在 MTKview 中被放大了。我知道这与我如何设置捕获会话无关，因为当我使用标准的 AVCapturePreviewLayer 时，一切都很好。关于我做错了什么有什么想法吗？

提前非常感谢！

P.S 此代码主要基于这个优秀的教程：https://betterprogramming.pub/using-cifilters-metal-to-make-a-custom-camera-in-ios-c76134993316 但不知何故它似乎不适合我。

Answer 1

根据捕获会话的设置，相机帧的大小将与您的MTKView 不同。这意味着您需要在渲染之前缩放和翻译它们以匹配currentDrawable 的大小。我为此使用以下代码（在draw 内，就在render 调用之前）：

    // scale to fit into view
    let drawableSize = self.drawableSize
    let scaleX = drawableSize.width / input.extent.width
    let scaleY = drawableSize.height / input.extent.height
    let scale = min(scaleX, scaleY)
    let scaledImage = input.transformed(by: CGAffineTransform(scaleX: scale, y: scale))

    // center in the view
    let originX = max(drawableSize.width - scaledImage.extent.size.width, 0) / 2
    let originY = max(drawableSize.height - scaledImage.extent.size.height, 0) / 2
    let centeredImage = scaledImage.transformed(by: CGAffineTransform(translationX: originX, y: originY))