我可以使用 RealityKit 在世界跟踪配置中进行 ARKit“连续图像跟踪”吗？

Question

更新：我的前提是“连续图像跟踪”不可能开箱即用 RealityKit ARViews 是不正确的。我需要做的就是为持续跟踪的参考图像正确创建 AnchorEntity。

需要使用init(anchor: ARAnchor) 初始化器创建锚实体。 （init(world: SIMD3<Float>) 初始值设定项适用于固定在现实世界中的锚点，但不适用于应该跟踪参考图像的锚点。）

使用带有ARWorldTrackingConfiguration 的 ARKit 和 RealityKit，我正在尝试进行“连续图像跟踪”（参考图像在每一帧都被跟踪，虚拟对象可以锚定到它上面，看起来像是附着在上面并随其移动参考图像）。因为参考图像在世界跟踪中只被识别一次（与ARImageTrackingConfiguration 不同，参考图像只要在帧中就会被持续跟踪），这不可能开箱即用。

为了在世界跟踪配置中获得相同的结果，我在 session(_:didAdd:) 委托方法中将虚拟对象锚定到参考图像，并使用 session(_:didUpdate:) 委托方法作为每次删除 ARImageAnchor 之后的机会被识别。这会导致参考图像被一遍又一遍地重新识别，从而允许将虚拟对象锚定到图像上并似乎逐帧跟踪它。

在下面的示例中，我放置了两个球标记来跟踪参考图像的位置。第一个标记仅放置一次，在最初检测到参考图像的位置。每次重新检测到参考图像时，都会重新定位另一个标记，看起来跟随着它。

这行得通。虚拟内容在 ARWorldTrackingConfiguration 中跟踪参考图像的方式与在图像跟踪配置中相同。但是，虽然 ARImageTrackingConfiguration 中的“动画”非常流畅，但世界跟踪中的动画却没有那么流畅，更加跳跃，就好像它以每秒 10 或 15 帧的速度运行一样。 （.showStatistics 报告的实际 FPS 在两种配置中都保持在 60 FPS 附近。）

我假设平滑度的差异是由于 ARKit 在每个 didAdd/didUpdate 循环中重复重新识别和删除参考图像锚点的工作所花费的时间。

我想知道是否有更好的技术可以在 ARWorldTrackingConfiguration 中获得“连续图像跟踪”，和/或是否有任何方法可以改进委托方法中的代码以实现这种效果。

import ARKit
import RealityKit

class ViewController: UIViewController, ARSessionDelegate {

    @IBOutlet var arView: ARView!
    
    // originalImageAnchor is used to visualize the first-detected location of reference image
    // currentImageAnchor should be continuously updated to match current position of ref image
    var originalImageAnchor: AnchorEntity!
    var currentImageAnchor: AnchorEntity!
    
    let ballRadius: Float = 0.02

    override func viewDidLoad() {
        super.viewDidLoad()
        
        guard let referenceImages = ARReferenceImage.referenceImages(inGroupNamed: "AR Resources",
             bundle: nil) else { fatalError("Missing expected asset catalog resources.") }
        
        arView.session.delegate = self
        arView.automaticallyConfigureSession = false
        arView.debugOptions = [.showStatistics]
        arView.renderOptions = [.disableCameraGrain, .disableHDR, .disableMotionBlur,
            .disableDepthOfField, .disableFaceOcclusions, .disablePersonOcclusion,
            .disableGroundingShadows, .disableAREnvironmentLighting]

        let configuration = ARWorldTrackingConfiguration()
        configuration.detectionImages = referenceImages
        configuration.maximumNumberOfTrackedImages = 1  // there is one ref image named "coaster_rb"

        arView.session.run(configuration)
    }

    func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
        guard let imageAnchor = anchors[0] as? ARImageAnchor else { return }

        // Reference image detected. This will happen multiple times because
        // we delete ARImageAnchor in session(_:didUpdate:)
        if let imageName = imageAnchor.name, imageName  == "coaster_rb" {

            // If originalImageAnchor is nil, create an anchor and
            // add a marker at initial position of reference image.
            if originalImageAnchor == nil {
                originalImageAnchor = AnchorEntity(world: imageAnchor.transform)
                let originalImageMarker = generateBallMarker(radius: ballRadius, color: .systemPink)
                originalImageMarker.position.y = ballRadius + (ballRadius * 2)
                originalImageAnchor.addChild(originalImageMarker)
                arView.scene.addAnchor(originalImageAnchor)
            }
            
            // If currentImageAnchor is nil, add an anchor and marker at reference image position
            // If currentImageAnchor has already been added, adjust it's position to match ref image
            if currentImageAnchor == nil {
                currentImageAnchor = AnchorEntity(world: imageAnchor.transform)
                let currentImageMarker = generateBallMarker(radius: ballRadius, color: .systemTeal)
                currentImageMarker.position.y = ballRadius
                currentImageAnchor.addChild(currentImageMarker)
                arView.scene.addAnchor(currentImageAnchor)
            } else {
                currentImageAnchor.setTransformMatrix(imageAnchor.transform, relativeTo: nil)
            }
        }
    }
    
    func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
        guard let imageAnchor = anchors[0] as? ARImageAnchor else { return }

        // Delete reference image anchor to allow for ongoing tracking as it moves
        if let imageName = imageAnchor.name, imageName  == "coaster_rb" {
            arView.session.remove(anchor: anchors[0])
        }
    }
    
    func generateBallMarker(radius: Float, color: UIColor) -> ModelEntity {
        let ball = ModelEntity(mesh: .generateSphere(radius: radius),
            materials: [SimpleMaterial(color: color, isMetallic: false)])
        return ball
    }
}

Answer 1

在世界跟踪配置中，RealityKit ARViews 可以开箱即用地进行连续图像跟踪。我的原始代码中的一个错误导致我不这么认为。

不正确的锚实体初始化（对于我试图完成的事情）：

currentImageAnchor = AnchorEntity(world: imageAnchor.transform)

由于我想跟踪分配给匹配参考图像的 ARImageAnchor，我应该这样做：

currentImageAnchor = AnchorEntity(anchor: imageAnchor)

下面的更正示例放置了一个固定到参考图像初始位置的虚拟标记，以及另一个在世界跟踪配置中平滑跟踪参考图像的虚拟标记：

import ARKit
import RealityKit

class ViewController: UIViewController, ARSessionDelegate {

    @IBOutlet var arView: ARView!
    
    let ballRadius: Float = 0.02

    override func viewDidLoad() {
        super.viewDidLoad()
        
        guard let referenceImages = ARReferenceImage.referenceImages(
            inGroupNamed: "AR Resources", bundle: nil) else {
            fatalError("Missing expected asset catalog resources.")
        }
        
        arView.session.delegate = self
        arView.automaticallyConfigureSession = false
        arView.debugOptions = [.showStatistics]
        arView.renderOptions = [.disableCameraGrain, .disableHDR,
            .disableMotionBlur, .disableDepthOfField,
            .disableFaceOcclusions, .disablePersonOcclusion,
            .disableGroundingShadows, .disableAREnvironmentLighting]

        let configuration = ARWorldTrackingConfiguration()
        configuration.detectionImages = referenceImages
        configuration.maximumNumberOfTrackedImages = 1

        arView.session.run(configuration)
    }

    func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
        guard let imageAnchor = anchors[0] as? ARImageAnchor else { return }

        if let imageName = imageAnchor.name, imageName  == "target_image" {
            
            // AnchorEntity(world: imageAnchor.transform) results in anchoring
            // virtual content to the real world.  Content anchored like this
            // will remain in position even if the reference image moves.
            let originalImageAnchor = AnchorEntity(world: imageAnchor.transform)
            let originalImageMarker = makeBall(radius: ballRadius, color: .systemPink)
            originalImageMarker.position.y = ballRadius + (ballRadius * 2)
            originalImageAnchor.addChild(originalImageMarker)
            arView.scene.addAnchor(originalImageAnchor)

            // AnchorEntity(anchor: imageAnchor) results in anchoring
            // virtual content to the ARImageAnchor that is attached to the
            // reference image.  Content anchored like this will appear
            // stuck to the reference image.
            let currentImageAnchor = AnchorEntity(anchor: imageAnchor)
            let currentImageMarker = makeBall(radius: ballRadius, color: .systemTeal)
            currentImageMarker.position.y = ballRadius
            currentImageAnchor.addChild(currentImageMarker)
            arView.scene.addAnchor(currentImageAnchor)
        }
    }
    
    func makeBall(radius: Float, color: UIColor) -> ModelEntity {
        let ball = ModelEntity(mesh: .generateSphere(radius: radius),
            materials: [SimpleMaterial(color: color, isMetallic: false)])
        return ball
    }
}