EgoMI: Learning Active Vision and Whole-Body Manipulation from Egocentric Human Demonstrations
Justin Yu, Yide Shentu, Di Wu, Pieter Abbeel, Ken Goldberg, Shiyao Wu
AI summary
Problem
Egocentric human demonstrations suffer from a severe embodiment gap when deployed on robots because humans dynamically move their heads to search and fixate, creating rapid viewpoint shifts that static or wrist-mounted robot cameras cannot replicate, leading to context loss and policy failure.
Approach
EgoMI captures synchronized egocentric head and hand trajectories using a VR headset, processes them with a training-free spatial memory algorithm called SPARKS to retain critical viewpoints, and fine-tunes a pre-trained foundation model to predict relative whole-body motions for direct robot retargeting.
Key results
- Achieves zero-shot real-world transfer without on-embodiment data or visual augmentation
- SPARKS algorithm effectively mitigates context loss from rapid head viewpoint shifts
- Policies with explicit head-motion modeling and spatial memory consistently outperform baselines
- Bridges the human-robot embodiment gap for robust bimanual manipulation on semi-humanoid platforms
Why it matters
Provides a scalable, hardware-minimal pathway for training robots with natural active vision behaviors, advancing imitation learning for complex manipulation tasks.
Abstract
Imitation learning from human demonstrations offers a promising approach for robot skill acquisition, but egocentric human data introduces fundamental challenges due to the embodiment gap. During manipulation, humans actively coordinate head and hand movements, continuously reposition their viewpoint and use pre-action visual search strategies to locate task-relevant objects. These behaviors create dynamic, task-driven head motions that static robot sensing systems cannot replicate, leading to a significant distribution shift that degrades policy performance. We present EgoMI (Egocentric Manipulation Interface), a framework that captures synchronized end-effector and active head trajectories during manipulation tasks, resulting in data that can be retargeted to compatible semi-humanoid robot embodiments. To handle rapid and wide-spanning head viewpoint changes, we introduce a memory-augmented policy that selectively incorporates context from historical observations. We evaluate our approach on a bimanual robot equipped with an actuated camera head and find that policies with explicit head-motion modeling consistently outperform baseline methods. Results suggest that coordinated hand–eye learning with EgoMI effectively bridges the human-robot embodiment gap for robust imitation learning on semi-humanoid embodiments. Project page: https: //egocentric-manipulation-interface.github.io