Beyond Domain Randomization: Event-Inspired Perception for Visually Robust Adversarial Imitation from Videos
Andrea Ramazzina, Vittorio Giammarino, Matteo El Hariry, Mario Bijelic
AI summary
Problem
Visual imitation from videos fails under environmental mismatches like lighting or color shifts, and current solutions rely on computationally expensive or unavailable data augmentation techniques.
Approach
The method transforms consecutive RGB frames into a sparse, event-based representation that captures temporal intensity gradients while discarding static appearance features, then trains an adversarial imitation learning policy directly on these streams.
Key results
- Eliminates appearance-based distractors while preserving motion-critical temporal gradients
- Provides a computationally efficient alternative to domain randomization and handcrafted augmentation
- Achieves robust imitation across DeepMind Control and Adroit manipulation benchmarks under visual perturbations
- Unlocks real-world deployment potential by aligning with native event camera capabilities
Why it matters
It offers a practical, efficient pathway for deploying visually robust imitation learning in real-world robotics by bypassing costly domain adaptation techniques.
Abstract
Imitation from videos often fails when expert demonstrations and learner environments exhibit domain shifts, such as discrepancies in lighting, color, or texture. While visual randomization partially addresses this problem by augmenting training data, it remains computationally intensive and inher- ently reactive, struggling with unseen scenarios. We propose a different approach: instead of randomizing appearances, we eliminate their influence entirely by rethinking the sensory representation itself. Inspired by biological vision systems that prioritize temporal transients (e.g., retinal ganglion cells) and by recent sensor advancements, we introduce event-inspired perception for visually robust imitation. Our method converts standard RGB videos into a sparse, event-based representation that encodes temporal intensity gradients, discarding static appearance features. This biologically grounded approach disentangles motion dynamics from visual style, enabling robust visual imitation from observations even in the presence of visual mismatches between expert and agent environments. By training policies on event streams, we achieve invariance to appearance- based distractors without requiring computationally expensive and environment-specific data augmentation techniques. Ex- periments across the DeepMind Control Suite and the Adroit platform for dynamic dexterous manipulation show the efficacy of our method. Our code is publicly available at this link.