TCB-VIO: Tightly-Coupled Focal-Plane Binary-Enhanced Visual Inertial Odometry
Matthew Lisondra, Junseo Kim, Takashi Glenn Shimoda, Kourosh Zareinia, Sajad Saeedi
AI summary
Problem
Existing visual-inertial odometry algorithms are not optimized for focal-plane sensor-processor arrays (FPSPs), leaving a gap in tightly-coupled, high-frame-rate pose estimation that can leverage on-sensor parallel processing while mitigating analog noise and drift.
Approach
The method extracts binary edge and corner features directly on the FPSP, processes them with a novel binary-enhanced KLT tracker on a host computer, and fuses the results with 400 Hz IMU data using a tightly-coupled MSCKF filter to estimate 6-DoF pose at 250 FPS.
Key results
- First high-frame-rate tightly-coupled VIO for FPSPs operating at 250 FPS
- Novel binary-enhanced KLT tracker enabling robust tracking on on-sensor binary data
- Outperforms ROVIO, VINS-Mono, and ORB-SLAM3 in trajectory error across 13 fast-motion tests
- Reduces computational overhead by performing feature extraction and edge processing directly on the focal plane
Why it matters
It enables low-latency, power-efficient high-speed pose estimation for agile robotics and drones operating under strict computational constraints.
Abstract
Vision algorithms can be executed directly on the image sensor when implemented on the next-generation sensors known as focal-plane sensor-processor arrays (FPSP)s, where every pixel has a processor. FPSPs greatly improve latency, reducing the problems associated with the bottleneck of data transfer from a vision sensor to a processor. FPSPs accelerate vision-based al- gorithms such as visual-inertial odometry (VIO). However, VIO frameworks suffer from spatial drift due to the vision-based pose estimation, whilst temporal drift arises from the inertial measure- ments. FPSPs circumvent the spatial drift by operating at a high frame rate to match the high-frequency output of the inertial mea- surements. In this letter, we present TCB-VIO, a tightly-coupled 6 degrees-of-freedomVIObyaMulti-StateConstraintKalmanFilter (MSCKF), operating at a high frame-rate of 250 FPS and from IMU measurements obtained at 400 Hz. TCB-VIO outperforms state-of-the-art methods: ROVIO, VINS-Mono, and ORB-SLAM3.