Real-Time Motion Segmentation with Event-Based Normal Flow
Sheng Zhong, Zhongyang Ren, Xiya Zhu, Dehao Yuan, Cornelia Fermuller, Yi Zhou
AI summary
Problem
Direct processing of sparse event data for motion segmentation is computationally prohibitive, and existing graph-based methods suffer from heavy initialization costs and slow iterative fitting that hinder real-time performance.
Approach
The system converts event neighborhoods into dense normal flow, formulates segmentation as an energy minimization problem solved via graph cuts, and uses a fast sampling and motion prediction strategy to efficiently initialize and fit motion models.
Key results
- Nearly 800× speedup over the EMSGC baseline
- State-of-the-art detection accuracy on the EED dataset
- Improved IoU segmentation on the EVIMO dataset
- Efficient initialization reducing candidate motion models via fast sampling and motion prediction
Why it matters
Provides a computationally efficient, real-time solution for dynamic scene understanding in robotics and autonomous navigation without relying on prior object knowledge or RGB imagery.
Abstract
Event-based cameras are bio-inspired sensors with pixels that independently and asynchronously respond to brightness changes at microsecond resolution, offering the po- tential to handle visual tasks in challenging scenarios. However, due to the sparse information content in individual events, directly processing the raw event data to solve vision tasks is highly inefficient, which severely limits the applicability of state-of-the-art methods in real-time tasks, such as mo- tion segmentation—a fundamental task for dynamic scene understanding. Incorporating normal flow as an intermediate representation to compress motion information from event clusters within a localized region provides a more effective solution. In this work, we propose a normal flow-based motion segmentation framework for event-based vision. Leveraging the dense normal flow directly learned from event neighborhoods as input, we formulate the motion segmentation task as an energy minimization problem solved via graph cuts, and optimize it iteratively with normal flow clustering and motion model fitting. By using a normal flow-based motion model initialization and fitting method, the proposed system is able to efficiently estimate the motion models of independently moving objects with only a limited number of candidate models, which significantly reduces the computational complexity and ensures real-time performance, achieving nearly a 800× speedup in comparison to the open-source state-of-the-art method. Extensive evaluations on multiple public datasets fully demonstrate the accuracy and efficiency of our framework. Our code is released at https://github.com/NAIL-HNU/EvMotionSeg to facilitate fur- ther research in this field.