Event and Frame-Based Visual-Inertial Odometry with Adaptive Filtering Based on 8-DOF Warping Uncertainty

In this letter, we present an event- and frame-based visual-inertial odometry (VIO) algorithm that fuses frames, events, and inertial measurement in a robust and adaptive manner. Frames from standard cameras provide rich context of the scene at a fixed rate. Event cameras on the other hand asynchronously produce events at a pixel-level when changes in intensity occur, and thus are resilient to motion blur and have high dynamic range. To harness the advantages of the two sensors, our frontend fuses their outputs by creating brightness increment patches of each output and minimize the differences with an 8-DOF warping model. The warping model and the optimization process allow for robust feature tracking in the frontend of the algorithm. The minimized residual is then used in the multi-state filter-based backend where the measurement update is adaptively performed depending on the size of the residual for accurate estimation, reflecting the quality of the tracked features. Comparative evaluation on two publicly available datasets reveals that our method outperforms the state-of-the-art event-based VIO algorithms in pose estimation accuracy.