FOVIS
本章提出供四旋翼無人機自主飛行使用的 RGB-D 視覺里程計，後來以 fovis 函式庫公開。演算法沿用立體視覺里程計的標準流程：灰階影像建立三層高斯金字塔，以自適應門檻的 FAST 角點擷取特徵並分格保留，從深度影像取得特徵深度；先以縮小影像直接估計初始旋轉，藉此限縮搜尋視窗，再以 9×9 影像塊描述子做雙向一致的匹配與次像素修正。內點以「剛體運動保持點間距離」建立一致性圖，並以貪婪法近似最大團挑選；位姿先以 Horn 絕對定向求解，再最小化重投影誤差，並以參考關鍵影格降低懸停時的漂移。里程計與 IMU 以 EKF 融合後在機上即時控制飛行；迴圈閉合與位姿圖最佳化則沿用作者先前的 RGB-D Mapping，在機外筆電執行，並建立 10 公分解析度的佔據體素地圖。
本頁內容
Feature-based RGB-D visual odometry for onboard MAV control, released as the fovis library: FAST features on a Gaussian pyramid with bucketing, an image-based initial rotation to constrain matching, SAD patch matching with sub-pixel ESM refinement, greedy max-clique inlier detection and reprojection-error refinement against a reference keyframe; loop closure (RGB-D Mapping, TORO) and a 10 cm occupancy map run offboard.
技術屬性
欄位內容為文獻擷取紀錄的原文用語（英文），以原文為據；「未查證」表示本研究尚未讀到該資訊，不代表該方法不具備此能力。
| 感測輸入 | RGB-D camera: stripped-down Microsoft Kinect (PrimeSense), 640 x 480 RGB-D at 30 Hz、IMU on the vehicle, fused with the visual odometry in an EKF for control (not used inside the visual odometry) |
|---|---|
| 原文測試平台 | UAV (AscTec Pelican quadrotor; motion capture room of about 11 m x 7 m x 4 m; autonomous flights around the MIT campus and at the Intel Research office in Seattle) |
| 狀態估計 | Frame-to-reference-keyframe motion from sparse 3D feature matches: Horn absolute orientation on the inliers, refined by nonlinear least-squares minimization of feature reprojection error (bidirectional, ESM), then refined again after discarding matches above a fixed reprojection threshold; the reference frame is replaced only when motion against it fails or has too few inliers. For flight control the VO is fused with IMU data in an EKF, and delayed SLAM corrections are applied retroactively to the state history. |
| 資料關聯 | FAST corners on a three-level Gaussian pyramid with an adaptive threshold and 80 x 80 pixel bucketing (25 strongest per bucket), depth read from the depth image; 80-byte descriptors from 9 x 9 intensity patches matched by sum of absolute differences with a mutual-consistency check inside a search window set by an image-based initial rotation estimate; sub-pixel refinement with ESM; inliers from a greedy approximation of the maximal clique of matches whose 3D distances are preserved |
| 時間表示 | discrete poses (frame to reference keyframe) |
| 去畸變 | 不適用 (RGB-D camera; Kinect rolling shutter named as a limitation at higher speed, Sec. 5) |
| 迴圈閉合 | Not part of the visual odometry. In the paper's full system RGB-D Mapping [14] runs offboard: keyframes every 10 deg or 25 cm, candidates limited to 90 deg and 5 m pose difference and to the 15 best vocabulary-tree matches, RANSAC over FAST keypoints with Calonder descriptors (ratio 0.6, at least 10 inliers), and a two-frame sparse bundle adjustment of the relative pose |
| 全域最佳化 | Offboard pose graph optimized with TORO after each loop closure (roughly 30 ms); corrected poses and voxel maps sent back to the vehicle; sparse bundle adjustment over all features only offline (Fig. 7) |
| 地圖表示 | Offboard 3D log-likelihood occupancy voxel grid at 10 cm resolution from depth downsampled to 128 x 96 (about 1.5 ms per frame); rendered point clouds; offline textured surfaces from sparse bundle adjustment |
| 先驗資訊 | none |
| 可輸出幾何 | real-time 6-DoF pose and velocity estimates; occupancy voxel map used for path planning; rendered RGB-D point cloud; offline textured surface model |
| 計算需求 | Visual odometry onboard on a 1.86 GHz Core2Duo flight computer with 4 GB RAM (Pixhawk project), roughly 25 ms per frame; 14.7 ms per frame in Table 1 on a laptop (written as 2.67 GHz in Sec. 4.1 and 2.6 GHz in the timing paragraph); loop closure and mapping offboard on a laptop |
使用設備
原文使用的感測器、運算硬體與載具（equipment）。型號保留原文寫法，連結到設備頁中同一型號的歸併名稱；角色依原文用途分為方法輸入、資料集感測器、執行運算平台、參考或真值量測（reference or ground truth）與比較對象設備。
| 類別 | 型號（原文寫法） | 角色 | 資料集 | 原文規格 | 出處 |
|---|---|---|---|---|---|
| 慣性量測單元（IMU） | onboard IMU (model not reported) | 方法輸入 | 未標示 | fused with visual-odometry motion estimates in an Extended Kalman Filter for position and velocity | (Huang et al., 2017, Sec. 1; Sec. 3.3) |
| RGB-D 相機 | Microsoft Kinect (stripped down) | 方法輸入 | 未標示 | 640 x 480 RGB-D image at 30 Hz; 115 g when stripped down; mounted at the base of the vehicle tilted slightly down | (Huang et al., 2017, Sec. 1; Sec. 3; Fig. 1) |
| 載具平台 | Pelican quadrotor | 方法輸入 | 未標示 | maximal dimension 70 cm, payload up to 1000 g | (Huang et al., 2017, Sec. 3; Fig. 1) |
| 運算硬體 | onboard flight computer (Pixhawk project, ETH Zurich) | 執行運算平台 | 未標示 | 1.86 GHz Core2Duo processor, 4 GB RAM; runs VO, state estimation and control; roughly 25 ms per VO frame | (Huang et al., 2017, Sec. 3; Sec. 4.1) |
| 運算硬體 | laptop computer (2.67 GHz per Sec. 4.1; 2.6 GHz per the timing paragraph; model not reported) | 執行運算平台 | 未標示 | used for the timing results of Table 1 | (Huang et al., 2017, Sec. 4.1) |
| 運算硬體 | offboard laptop (model not reported) | 執行運算平台 | 未標示 | receives RGB-D data from the MAV; detects loop closures, computes global pose corrections and builds the occupancy voxel map | (Huang et al., 2017, Sec. 3.2) |
| 其他 | motion capture system (model not reported) | 參考或真值量測 | 未標示 | 120 Hz ground truth of MAV position and attitude; room about 11 m x 7 m x 4 m | (Huang et al., 2017, Sec. 4.1; Fig. 4) |
作者報告的優勢與限制
優勢
- On a deliberately challenging motion-capture dataset the chosen configuration had 0.387 m/s mean velocity error and 3.39% gross failures at 14.7 ms per frame; in feature-rich environments the authors observed 0.08 m/s mean velocity error without gross failures (Sec. 4.1; Table 1)
- Greedy max-clique inlier detection outperformed RANSAC and preemptive RANSAC at comparable computation (Sec. 4.1; Table 1)
- Visual odometry, sensor fusion and control run onboard at roughly 25 ms per frame on the 1.86 GHz computer (Sec. 4.1; Sec. 5)
- Under visual-odometry control the vehicle held position for 90 s with 6.2 cm mean and 19 cm maximum deviation measured by motion capture (Fig. 4)
- Keyframe reference frames eliminate drift when the viewpoint does not change much, which helps while hovering (Sec. 3.1)
限制
- Performs poorly in regions with few visual features and in large open areas where structure lies beyond the Kinect range; works better in cluttered, close quarters (Sec. 5)
- Assumes a static environment and slow vehicle motion; faster flight brings motion blur and Kinect rolling-shutter artifacts (Sec. 5)
- With moving objects covering much of the image, the maximal clique may not correspond to the static scene (Sec. 5)
- Loop closing and SLAM are not fast enough to run on the onboard processor (Sec. 3.2; Sec. 5)
- Maps from the larger environments have no ground truth; their quality is judged visually from the rendered point cloud (Sec. 4.2)
營建工程相關證據
論文未在施工現場測試；實驗在約 11 × 7 × 4 公尺的動作捕捉室，以及 MIT 校園與 Intel 西雅圖辦公室的室內空間進行（Sec. 4）。fovis 後來成為 Kintinuous 的里程計前端，出現在 ICL-NUIM 的軌跡評估中（Handa et al., 2014），也是 RTAB-Map 比較的里程計選項之一（Labbé & Michaud, 2019）。作者指出特徵少的區域與超出 Kinect 量測距離的大空間會使其表現變差（Sec. 5），因此在白牆多、樓板空曠的施工中建築內單獨使用時容易失效，需要其他感測器輔助（推論）。
原文驗證環境：受控實驗、獨立參考量測、已完工建築
報告的性能數據
以下是原文作者報告的性能數值（author-reported results），不是本研究重新量測的結果。每張圖只並列同一個比較組（comparison group，同一張表、同一組實驗設定）內的方法；不同比較組之間的數值不可直接比較，也不構成排名。
本方法共出現在 6 個比較組，合計 94 筆紀錄。以下列出本方法紀錄最多的 4 組，其餘 2 組列在最後，並連到性能比較頁。
Huang et al., 2017 · Table 1 本方法 54 筆
資料集與序列authors' MAV motion-capture dataset · MAV flight in motion capture room
表格設定（擷取紀錄原文）：Ablation on a challenging MAV motion-capture dataset (motion blur, feature-poor images); each row changes one component of the authors' configuration; velocity error is the mean velocity error magnitude against differentiated motion-capture data; a gross failure is no estimate or velocity error above 1 m/s; total time per RGB-D frame on a laptop (Huang et al., 2017, Table 1)
% gross failures，authors' MAV motion-capture dataset · MAV flight in motion capture room
只並列這張表在相同設定下報告的方法；以「本方法：」開頭者為本頁方法。失敗、未執行與未報告以標記呈現，不是 0。
按 Tab 進入圖表後，用上下方向鍵逐一瀏覽各類別，Esc 關閉提示框；也可開啟表格檢視閱讀全部數值。
這些是 Huang et al., 2017 在此表設定下報告的數值（author-reported results），只能在同一個比較組內對照，不代表方法在其他資料或設定下的表現。
資料來源作者報告值(Huang et al., 2017, Table 1)
| 方法（原文寫法） | 報告值 | 出處 |
|---|---|---|
| Our approach (greedy max-clique, initial rotation, 3 pyramid levels, bidirectional ESM, 9 x 9 window, subpixel refinement, adaptive FAST threshold, grid bucketing)本方法原文提出 | 3.39% | (Huang et al., 2017, Table 1) |
| Inlier detection: RANSAC本方法 | 6.05% | (Huang et al., 2017, Table 1) |
| Inlier detection: Preemptive RANSAC本方法 | 5.91% | (Huang et al., 2017, Table 1) |
| Initial rotation estimate: None本方法 | 4.22% | (Huang et al., 2017, Table 1) |
| Gaussian pyramid levels: 1本方法 | 5.17% | (Huang et al., 2017, Table 1) |
| Gaussian pyramid levels: 2本方法 | 3.52% | (Huang et al., 2017, Table 1) |
| Gaussian pyramid levels: 4本方法 | 3.5% | (Huang et al., 2017, Table 1) |
| Reprojection error minimization: Bidir. Gauss-Newton本方法 | 3.24% | (Huang et al., 2017, Table 1) |
| Reprojection error minimization: Unidir. Gauss-Newton本方法 | 3.45% | (Huang et al., 2017, Table 1) |
| Reprojection error minimization: Unidir. ESM本方法 | 3.47% | (Huang et al., 2017, Table 1) |
| Reprojection error minimization: Absolute orientation only本方法 | 10.97% | (Huang et al., 2017, Table 1) |
| Feature window size: 3本方法 | 5.96% | (Huang et al., 2017, Table 1) |
| Feature window size: 5本方法 | 4.24% | (Huang et al., 2017, Table 1) |
| Feature window size: 7本方法 | 3.72% | (Huang et al., 2017, Table 1) |
| Feature window size: 11本方法 | 3.42% | (Huang et al., 2017, Table 1) |
| Subpixel feature refinement: No refinement本方法 | 5.13% | (Huang et al., 2017, Table 1) |
| Adaptive FAST threshold: Fixed threshold (10)本方法 | 3.12% | (Huang et al., 2017, Table 1) |
| Feature grid/bucketing: No grid本方法 | 4.02% | (Huang et al., 2017, Table 1) |
Stückler & Behnke, 2014 · Table 1 本方法 22 筆
指標median relative pose error (RPE) in mm
表格設定（擷取紀錄原文）：Incremental (frame-to-frame) registration on TUM Freiburg sequences; median translational relative pose error in mm (maximum values in brackets in the table not extracted); warp is the OpenCV reimplementation (Stückler & Behnke, 2014, Table 1)
median relative pose error (RPE) in mm，TUM RGB-D (Freiburg) · fr1 360
只並列這張表在相同設定下報告的方法；以「本方法：」開頭者為本頁方法。失敗、未執行與未報告以標記呈現，不是 0。
按 Tab 進入圖表後，用上下方向鍵逐一瀏覽各類別，Esc 關閉提示框；也可開啟表格檢視閱讀全部數值。
這些是 Stückler & Behnke, 2014 在此表設定下報告的數值（author-reported results），只能在同一個比較組內對照，不代表方法在其他資料或設定下的表現。
資料來源作者報告值(Stückler & Behnke, 2014, Table 1)
| 方法（原文寫法） | 報告值 | 出處 |
|---|---|---|
| Ours (MRSMap)原文提出 | 5.1 mm | (Stückler & Behnke, 2014, Table 1) |
| Warp [17] (OpenCV) | 5.9 mm | (Stückler & Behnke, 2014, Table 1) |
| GICP [5] | 18.8 mm | (Stückler & Behnke, 2014, Table 1) |
| 3D-NDT [7] | 7.8 mm | (Stückler & Behnke, 2014, Table 1) |
| Fovis [12]本方法 | 7.1 mm | (Stückler & Behnke, 2014, Table 1) |
Huang et al., 2017 · Text Sec.4.1 本方法 8 筆
資料集與序列authors' MAV motion-capture dataset · MAV flight in motion capture room
表格設定（擷取紀錄原文）：Per-stage timing of the chosen configuration stated in the Timing paragraph (laptop given as 2.6 GHz there), plus the approximate onboard time per frame (Huang et al., 2017, Text Sec.4.1)
Preprocessing time，authors' MAV motion-capture dataset · MAV flight in motion capture room
這張表在此指標與資料序列只列出本方法一筆，沒有可並列的其他方法，因此不畫圖，數值與出處見下表。這是 Huang et al., 2017 在此表設定下報告的數值（author-reported results），不代表方法在其他資料或設定下的表現。
| 方法（原文寫法） | 報告值 | 出處 |
|---|---|---|
| our approach (visual odometry of this chapter; the name 'fovis' is not used in the text)本方法原文提出硬體：laptop computer (2.67 GHz per Sec. 4.1; 2.6 GHz per the timing paragraph; model not reported) | 2.1 ms | (Huang et al., 2017, Sec. 4.1) |
Zhang et al., 2014 · Table I 本方法 4 筆
指標Relative position error (% of distance traveled)
表格設定（擷取紀錄原文）：Author-collected tests with the Xtion RGB-D camera and with the custom camera plus rotating Hokuyo LiDAR; the camera starts and stops at the same position and the gap between the trajectory ends divided by trajectory length is the relative position error (3D coordinates); Fovis and DVO use RGB-D input (Zhang et al., 2014, Table I)
Relative position error (% of distance traveled)，DEMO author-collected tests · Room (16 m)
只並列這張表在相同設定下報告的方法；以「本方法：」開頭者為本頁方法。失敗、未執行與未報告以標記呈現，不是 0。
按 Tab 進入圖表後，用上下方向鍵逐一瀏覽各類別，Esc 關閉提示框；也可開啟表格檢視閱讀全部數值。
這些是 Zhang et al., 2014 在此表設定下報告的數值（author-reported results），只能在同一個比較組內對照，不代表方法在其他資料或設定下的表現。
資料來源作者報告值(Zhang et al., 2014, Table I)
| 方法（原文寫法） | 報告值 | 出處 |
|---|---|---|
| Fovis本方法 | 2.72% | (Zhang et al., 2014, Table I; Sec. VII) |
| DVO | 1.87% | (Zhang et al., 2014, Table I; Sec. VII) |
| Our VO (RGB-D)原文提出 | 2.14% | (Zhang et al., 2014, Table I; Sec. VII) |
| Our VO (Lidar)原文提出 | 2.06% | (Zhang et al., 2014, Table I; Sec. VII) |
其他比較組
來源
Huang et al., 2017
（2017）Visual Odometry and Mapping for Autonomous Flight Using an RGB-D CameraRobotics Research (ISRR 2011), Springer Tracts in Advanced Robotics, vol. 100 (eds. H.I. Christensen and O. Khatib), STAR 100:235-252
DOI 10.1007/978-3-319-29363-9_14程式碼
同儕審查已出版已讀全文經典查證後修正
相關版本
- 會議版：ISRR 2011 paper, Flagstaff, Arizona, USA, Aug. 2011 (author PDF linked from the fovis project page) https://fovis.github.io
- 程式碼釋出：fovis/fovis (libfovis visual odometry library) https://github.com/fovis/fovis
程式碼：https://github.com/fovis/fovis（授權：GPL (repository contains a GPL-2 licence file, while the project page fovis.github.io states GPL version 3 or later; no SPDX licence detected by GitHub)）。有公開程式碼不等於已被重現，也不代表目前版本與論文版本相同。