THREE.JS & Reality Capture - 3D 空间中的旋转问题摄影测量参考相机答案

【问题标题】：THREE.JS & Reality Capture - Rotation issue photogrammetry reference camera's in a 3D spaceTHREE.JS & Reality Capture - 3D 空间中的旋转问题摄影测量参考相机
【发布时间】：2022-01-19 15:42:06
【问题描述】：

感谢您抽出宝贵时间审阅我的帖子。我希望这篇文章不仅可以为自己带来成果，也可以帮助其他人！

简介

目前我正在从事一个涉及使用摄影测量生成的点云的项目。它由结合激光扫描的照片组成。用于制作点云的软件是 Reality Capture。除了点云导出之外，还可以导出“内部/外部相机参数”，提供检索用于在点云中构成某个 3D 点的照片的能力。 Reality Capture 的在线文档记录不是很好，我也在他们的论坛中发布了关于 camera variables 的帖子，也许它可以用于解决手头的问题？

相机参数文件中列出的只有少数变量（目前）与引用相机定位相关，例如 文件名、x、y、alt 表示位置、航向、俯仰和滚动作为其旋转 .

当前生成的点云被加载到与浏览器兼容的 THREE.JS 查看器中，然后加载相机参数 .csv 文件，并为每张已知照片生成一个带有绿色立方体的“PerspectiveCamera”。示例如下：

挑战

事实上，您可能已经根据上一张图片（当然也可以是这篇文章的标题 ;P）知道问题所在都错了。让我用破旧的自绘矢量为您可视化它，这些矢量初步显示了它应该面向的方向（标记为红色）以及它当前的矢量化方式（绿色） .

第 37 行，DJI_0176.jpg 是最右边的相机，红色参考线第 38 行是 177 等。最后一张图片（第 48 行是 DJI_189.jpg）并且与聚类图像的最左侧图像相对应（因为我没有在上面的图像中绘制其他两个相机参考，所以我没有包括其他）。

当您将以下数据复制到 Excel 工作表中时，它应该可以正确显示 ^^

#name   x   y   alt heading pitch   roll    f   px  py  k1  k2  k3  k4  t1  t2
DJI_0174.JPG    3.116820957 -44.25690188    14.05258109 -26.86297007    66.43104338 1.912026354 30.35179628 7.25E-03    1.45E-03    -4.02E-03   -2.04E-02   3.94E-02    0   0   0
DJI_0175.JPG    -5.22E-02   -46.97266554    14.18056658 -16.2033133 66.11532302 3.552072396 30.28063771 4.93E-03    4.21E-04    1.38E-02    -0.108013599    0.183136287 0   0   0
DJI_0176.JPG    -3.056586953    -49.00754998    14.3474763  4.270483155 65.35247679 5.816970677 30.50596933 -5.05E-03   -3.53E-03   -4.94E-03   3.24E-02    -3.84E-02   0   0   0
DJI_0177.JPG    -6.909437337    -50.15910066    14.38391206 19.4459053  64.26828897 6.685020944 30.6994734  -1.40E-02   4.72E-03    -5.33E-04   1.90E-02    -1.74E-02   0   0   0
DJI_0178.JPG    -11.23696688    -50.36025313    14.56924433 19.19192622 64.40188316 6.265995184 30.7665397  -1.26E-02   2.41E-03    1.24E-04    -4.63E-03   2.84E-02    0   0   0
DJI_0179.JPG    -16.04060554    -49.92320365    14.69721478 19.39979452 64.85507307 6.224929846 30.93772566 -1.19E-02   -4.31E-03   -1.27E-02   4.62E-02    -4.48E-02   0   0   0
DJI_0180.JPG    -20.95614556    -49.22915437    14.92273203 20.39327092 65.02028543 6.164031482 30.99807237 -1.02E-02   -7.70E-03   1.44E-03    -2.22E-02   3.94E-02    0   0   0
DJI_0181.JPG    -25.9335097 -48.45330177    15.37330388 34.24388008 64.82707628 6.979877709 31.3534556  -1.06E-02   -1.19E-02   -5.44E-03   2.39E-02    -2.38E-02   0   0   0
DJI_0182.JPG    -30.40507957    -47.21269946    15.67804925 49.98858409 64.29238807 7.449650513 31.6699868  -8.75E-03   -1.31E-02   -4.57E-03   2.31E-02    2.68E-03    0   0   0
DJI_0183.JPG    -34.64277285    -44.84034207    15.89229254 65.84203906 62.9109777  7.065942792 31.78292476 -8.39E-03   -2.94E-03   -1.40E-02   8.96E-02    -0.11801932 0   0   0
DJI_0184.JPG    -39.17179024    -40.22577764    16.28164396 65.53938063 63.2592604  6.676581293 31.79546988 -9.81E-03   -8.13E-03   1.01E-02    -8.44E-02   0.179931606 0   0   0
DJI_0185.JPG    -43.549378  -33.09364534    16.64130671 68.61427166 63.15205908 6.258411625 31.75339036 -9.78E-03   -7.12E-03   4.75E-03    -6.25E-02   0.1541638   0   0   0
DJI_0186.JPG    -46.5381556 -24.2992233 17.2286956  74.42382577 63.75110346 6.279208736 31.88862443 -1.01E-02   -1.73E-02   1.02E-02    -6.15E-02   4.89E-02    0   0   0
DJI_0187.JPG    -48.18737751    -14.67333218    17.85446854 79.54477952 63.0503902  5.980759013 31.69602914 -8.83E-03   -1.01E-02   -7.63E-03   -7.49E-03   2.71E-02    0   0   0
DJI_0188.JPG    -48.48581505    -13.79840485    17.84756621 93.43316271 61.87561678 5.110113503 31.6671977  1.99E-03    -9.40E-04   2.40E-02    -0.180515731    0.32814456  0   0   0
DJI_0189.JPG    -48.32815991    -13.88055437    17.77818573 106.3277582 60.87171036 4.039469869 31.50757712 2.84E-03    4.12E-03    8.54E-03    -1.32E-02   3.89E-02    0   0   0

目前尝试过的事情

我们发现导出的模型是从现实中镜像的，但这并不影响相机参考的放置，因为它们完美对齐。我们试图镜像引用的相机、点云和视口相机，但这似乎并没有解决手头的问题。 （因此camera.applyMatrix4(new THREE.Matrix4().makeScale(-1, 1, 1));）

到目前为止，我们尝试加载欧拉角，直接设置角度或转换并应用四元数，遗憾的是没有任何好的结果。正在使用以下逻辑解析相机参考文件：

// Await the .csv file being parsed from the server
    await new Promise((resolve) => {
      (file as Blob).text().then((csvStr) => {
        const rows = csvStr.split('\n');
        for (const row of rows) {
          const col = row.split(',');
          if (col.length > 1) {
            const suffixes = col[0].split('.');
            const extension = suffixes[suffixes.length - 1].toLowerCase();
            const validExtensions = ['jpeg', 'jpg', 'png'];
            if (!validExtensions.includes(extension)) {
              continue;
            }
            // == Parameter index by .csv column names ==
            // 0: #name; 1: x; 2: y; 3: alt; 4: heading; 5: pitch; 6: roll; 7:f (focal);
            // == Non .csv param ==
            // 8: bool isRadianFormat default false
            this.createCamera(col[0], parseFloat(col[1]), parseFloat(col[2]), parseFloat(col[3]), parseFloat(col[4]), parseFloat(col[5]), parseFloat(col[6]), parseFloat(col[7]));
          }
        }
        resolve(true);
      });
    });
  }

您将在下面找到代码 sn-p 用于实例化相机及其位置和旋转。我留下了一些额外的 cmets 来详细说明它。我也留下了注释的代码行，看看我们还在尝试什么：

private createCamera(fileName: string, xPos: number, yPos: number, zPos: number, xDeg: number, yDeg: number, zDeg: number, f: number, isRadianFormat = false) : void {
    // Set radials as THREE.JS explicitly only works in radians
    const xRad = isRadianFormat ? xDeg : THREE.MathUtils.degToRad(xDeg);
    const yRad = isRadianFormat ? yDeg : THREE.MathUtils.degToRad(yDeg)
    const zRad = isRadianFormat ? zDeg : THREE.MathUtils.degToRad(zDeg)

    // Create camera reference and extract frustum
    // Statically set the FOV and aspectratio; Near is set to 0,1 by default and Far is dynamically set whenever a point is clicked in a 3D space.
    const camera = new THREE.PerspectiveCamera(67, 5280 / 2970, 0.1, 1); 
    const pos = new THREE.Vector3(xPos, yPos, zPos); // Reality capture z = up; THREE y = up;

    /* ===
    In order to set an Euler angle one must provide the heading (x), pitch (y) and roll(z) as well as the order (variable four 'XYZ') in which the rotations will be applied 
    As a last resort we even tried switching the x,y and zRad variables as well as switching the orientation orders.
    Possible orders:
     XYZ 
     XZY
     YZX
     YXZ
     ZYX
     ZXY
       === */
    const rot = new THREE.Euler(xRad, yRad, zRad, 'XYZ');
    //camera.setRotationFromAxisAngle(new THREE.Vector3(0,))

    //camera.applyMatrix4(new THREE.Matrix4().makeScale(-1, 1, 1));
    // const rot = new THREE.Quaternion();
    // rot.setFromAxisAngle(new THREE.Vector3(1, 0, 0), zRad);
    // rot.setFromAxisAngle(new THREE.Vector3(0, 1, 0), xRad);
    // rot.setFromAxisAngle(new THREE.Vector3(0, 0, 1), yRad);
    // XYZ

    // === Update camera frustum ===
    camera.position.copy(pos);
    // camera.applyQuaternion(rot);
    camera.rotation.copy(rot);
    camera.setRotationFromEuler(rot);
    camera.updateProjectionMatrix(); // TODO: Assert whether projection update is required here
    /* ===
    The camera.applyMatrix listed below was an attempt in rotating several aspects of the 3D viewer.
    An attempt was made to rotate each individual photo camera position, the pointcloud itself aswell as the viewport camera both separately
    as well as solo. It made no difference however.
       === */
    //camera.applyMatrix4(new THREE.Matrix4().makeScale(-1, 1, 1));

    // Instantiate CameraPosition instance and push to array
    const photo: PhotoPosition = {
      file: fileName,
      camera,
      position: pos,
      rotation: rot,
      focal: f,
      width: 5120,  // Statically set for now
      height: 5120, // Statically set for now
    };

    this.photos.push(photo);
  }

在上面的 sn-p 中创建的相机然后被下一段代码抓取，该代码将相机传递给相机管理器并绘制一个CameraHelper（显示在上面的两个 3D 查看器图片中）。它是在异步函数中编写的，等待加载 csv 文件，然后再继续初始化相机。

private initializeCameraPoses(url: string, csvLoader: CSVLoader) {
    const absoluteUrl = url + '\\references.csv';

    (async (scene, csvLoader, url, renderer) => {
      await csvLoader.init(url);
      const photos = csvLoader.getPhotos(); // The cameras created by the createCamera() method
      this.inspectionRenderer = new InspectionRenderer(scene);  // InspectionRenderer manages all further camera operations
      this.inspectionRenderer.populateCameras(photos);
      for (const photoData of photos) {
        // Draw the green cube
        const geometry = new THREE.BoxGeometry(0.5, 0.5, 0.5);
        const material = new THREE.MeshBasicMaterial({ color: 0x00ff00 });
        const cube = new THREE.Mesh(geometry, material);
        scene.add(cube);

        cube.position.copy(photoData.position);
        photoData.camera.updateProjectionMatrix();
        
        // Draws the yellow camera viewport to the scene
        const helper = new CameraHelper(photoData.camera);
        renderer.render(scene, photoData.camera);    
        scene.add(helper);
      }
    })(this.scene, csvLoader, absoluteUrl, this.renderer);
  }

Marquizzo 的代码 sn-p

下面发布的 Marquizzo 代码 sn-p 似乎让我们更接近解决方案。摄像机的方向似乎正确。然而，不知何故，音高似乎有点偏离。下面我将包含 DJI_0189.jpg 的源图像。请注意，对于此示例，当前未设置 FOV，因为当每个相机位置都在渲染相机助手时，它看起来很混乱。对于这个例子，我只渲染了 DJI_0189 相机助手。

@Marquizzo 提供的反转音高编辑 (const rotX = deg2rad(photo.pitch * -1);) 将导致中点交点始终如预期的那样略低：

当间距调整为const rotX = deg2rad(photo.pitch * -.5); 时，您会看到中点交点（更接近）与源图像的交点：

不知何故，我认为解决方案触手可及，最终它会归结为一些被忽视的非常小的细节。我真的很期待看到回复。如果还有什么不清楚的地方请说出来，如果需要我会提供必要的细节^^

感谢您到目前为止阅读这篇文章！

【问题讨论】：

标签： javascript typescript three.js point-clouds photogrammetry

【解决方案1】：

乍一看，我看到了三种可能性：

如果不显示您如何使用createCamera() 方法，就很难看出问题出在哪里。你可以用heading 或类似的东西交换pitch。在 Three.js 中，heading 是绕 Y 轴旋转，绕 X 轴俯仰，绕 Z 轴滚动。
其次，您知道您的传感器以什么顺序进行heading, pitch, roll 测量？这将影响您启动THREE.Euler(xRad, yRad, zRad, 'XYZ') 的方式，因为the order in which to apply rotations 也可能是'YZX', 'ZXY', 'XZY', 'YXZ' or 'ZYX'。
最后，您必须思考“heading: 0 对传感器意味着什么？”在现实世界和 Three.js 坐标系之间可能意味着不同的东西。 Three.js 中没有旋转的摄像头正向下看-Z 轴，但您的传感器可能让它指向+Z 或+X 等。

编辑：

我在下面添加了一个演示，我认为这就是你需要的截图。请注意，我将pitch * -1 相乘，因此相机“向下看”，并将+180 添加到标题中，因此它们指向右侧...标题。

const DATA = [
{name: "DJI_0174.JPG",  x: 3.116820957,     y: -44.25690188,    alt: 14.05258109,   heading: -26.86297007,  pitch: 66.43104338, roll: 1.912026354},
{name: "DJI_0175.JPG",  x: -5.22E-02,       y: -46.97266554,    alt: 14.18056658,   heading: -16.2033133,   pitch: 66.11532302, roll: 3.552072396},
{name: "DJI_0176.JPG",  x: -3.056586953,    y: -49.00754998,    alt: 14.3474763,    heading: 4.270483155,   pitch: 65.35247679, roll: 5.816970677},
{name: "DJI_0177.JPG",  x: -6.909437337,    y: -50.15910066,    alt: 14.38391206,   heading: 19.4459053,    pitch: 64.26828897, roll: 6.685020944},
{name: "DJI_0178.JPG",  x: -11.23696688,    y: -50.36025313,    alt: 14.56924433,   heading: 19.19192622,   pitch: 64.40188316, roll: 6.265995184},
{name: "DJI_0179.JPG",  x: -16.04060554,    y: -49.92320365,    alt: 14.69721478,   heading: 19.39979452,   pitch: 64.85507307, roll: 6.224929846},
{name: "DJI_0180.JPG",  x: -20.95614556,    y: -49.22915437,    alt: 14.92273203,   heading: 20.39327092,   pitch: 65.02028543, roll: 6.164031482},
{name: "DJI_0181.JPG",  x: -25.9335097,     y: -48.45330177,    alt: 15.37330388,   heading: 34.24388008,   pitch: 64.82707628, roll: 6.979877709},
{name: "DJI_0182.JPG",  x: -30.40507957,    y: -47.21269946,    alt: 15.67804925,   heading: 49.98858409,   pitch: 64.29238807, roll: 7.449650513},
{name: "DJI_0183.JPG",  x: -34.64277285,    y: -44.84034207,    alt: 15.89229254,   heading: 65.84203906,   pitch: 62.9109777,  roll: 7.065942792},
{name: "DJI_0184.JPG",  x: -39.17179024,    y: -40.22577764,    alt: 16.28164396,   heading: 65.53938063,   pitch: 63.2592604,  roll: 6.676581293},
{name: "DJI_0185.JPG",  x: -43.549378,      y: -33.09364534,    alt: 16.64130671,   heading: 68.61427166,   pitch: 63.15205908, roll: 6.258411625},
{name: "DJI_0186.JPG",  x: -46.5381556,     y: -24.2992233,     alt: 17.2286956,    heading: 74.42382577,   pitch: 63.75110346, roll: 6.279208736},
{name: "DJI_0187.JPG",  x: -48.18737751,    y: -14.67333218,    alt: 17.85446854,   heading: 79.54477952,   pitch: 63.0503902,  roll: 5.980759013},
{name: "DJI_0188.JPG",  x: -48.48581505,    y: -13.79840485,    alt: 17.84756621,   heading: 93.43316271,   pitch: 61.87561678, roll: 5.110113503},
{name: "DJI_0189.JPG",  x: -48.32815991,    y: -13.88055437,    alt: 17.77818573,   heading: 106.3277582,   pitch: 60.87171036, roll: 4.039469869},
];

const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(
    45,
    window.innerWidth / window.innerHeight,
    1,
    1000
);
camera.position.z = 100;

const renderer = new THREE.WebGLRenderer({
    antialias: true,
    canvas: document.querySelector("#canvas")
});
renderer.setSize(window.innerWidth, window.innerHeight);
const controls = new THREE.OrbitControls( camera, renderer.domElement );

// Helpers
const axesHelper = new THREE.AxesHelper( 20 );
scene.add(axesHelper);
const plane = new THREE.Plane( new THREE.Vector3( 0, 1, 0 ), 0 );
const planeHelper = new THREE.PlaneHelper( plane, 50, 0xffff00 );
scene.add(planeHelper);

let deg2rad = THREE.MathUtils.degToRad;

function createCam(photo) {
    let tempCam = new THREE.PerspectiveCamera(10, 2.0, 1, 30);
    // Altitude is actually y-axis,
    // "y" is actually z-axis 
    tempCam.position.set(photo.x, photo.alt, photo.y);

  // Modify pitch & heading so it matches Three.js coordinates
    const rotX = deg2rad(photo.pitch * -1);
    const rotY = deg2rad(photo.heading + 180);
    const rotZ = deg2rad(photo.roll);
  
    tempCam.rotation.set(rotX, rotY, rotZ, "YXZ");
    let helper = new THREE.CameraHelper(tempCam);
  scene.add(tempCam);
    scene.add(helper);
}

for(let i = 0; i < DATA.length; i++) {
    createCam(DATA[i]);
}

function animate() {
  renderer.render(scene, camera);
  requestAnimationFrame(animate);
}

animate();

html, body { margin:0; padding:0;}

<script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/r128/three.min.js"></script><script src="https://rawgit.com/mrdoob/three.js/dev/examples/js/controls/OrbitControls.js"></script>

<canvas id="canvas"></canvas>

【讨论】：

嗨@Marquizzo，感谢您的快速回复！ createCamera() 方法是唯一应用旋转的地方。在 initializeCameraPoses() 方法中实例化的 CameraHelper 也包含在帖子中 ('const helper = new CameraHelper(photoData.camera);)。它从使用 create camera 方法创建的相机中复制所需的属性。在上面的帖子中我已经包含了关于如何从 .csv 文件 (threejs.org/docs/#api/en/helpers/CameraHelper) 解析相机引用的代码 sn-p
@DiabetiMark 您能否分享我们在您原始问题的屏幕截图中看到的位置和轮换数据？只要有红线的 12 个样本就足够了。
@DiabetiMark 谢谢，但我希望你能以文本格式粘贴它，以便我可以复制/粘贴这些值。你粘贴了一张图片，这意味着我必须手动重新输入数字。你能在代码块中添加值吗？
@DiabetiMark 我添加了一个工作演示，其中包含我认为可能是期望的结果。
@DiabetiMark 好吧，任何人都可以想出自己对“前锋”的定义。 0° 的航向可能意味着在虚构的 3D 世界中向北、向南、向东或向西。所以添加 +180 只是确保你的传感器和threejs都排成一行。请参阅我上面原始答案中的第三个要点。