Augmented Reality for RObots (ARRO): Pointing Visuomotor Policies towards Visual Robustness
Reihaneh Mirjalili, Tobias Thomas Jülg, Florian Walter, Wolfram Burgard
AI summary
Problem
Visuomotor policies trained on expert demonstrations degrade significantly when deployed in new environments due to background changes, distractors, or robot appearance variations, while collecting diverse training data is costly and impractical.
Approach
ARRO uses open-vocabulary segmentation and object detection to filter out visual distractors in real time, overlaying only the robot gripper and target objects onto a structured virtual background during both training and inference.
Key results
- Consistently improves success rates across tabletop tasks under background and distractor shifts
- Enables zero-shot spatial grounding and selective object masking without retraining
- Maintains robust segmentation under partial occlusions and object deformations
- Demonstrates compatibility and performance gains with generalist policies like Octo and OpenVLA
Why it matters
Offers a plug-and-play, training-free solution to bridge the gap between lab-trained robotic policies and unpredictable real-world environments.
Abstract
Visuomotor policies trained on human expert demonstrations have recently shown strong performance across a wide range of robotic manipulation tasks. However, these policies remain highly sensitive to domain shifts stemming from background or robot embodiment changes, which limits their generalization capabilities. In this paper, we present ARRO, a novel visual representation that leverages zero-shot open- vocabulary segmentation and object detection models to effi- ciently mask out task-irrelevant regions of the scene in real time without requiring additional training, modeling of the setup, or camera calibration. By filtering visual distractors and overlay- ing virtual guides during both training and inference, ARRO improves robustness to scene variations and reduces the need for additional data collection. We extensively evaluate ARRO with Diffusion Policy on a range of tabletop manipulation tasks in both simulation and real-world environments, and further demonstrate its compatibility and effectiveness with generalist robot policies, such as Octo, OpenVLA and π0. Across all settings in our evaluation, ARRO yields consistent performance gains, allows for selective masking to choose between different objects, and shows robustness even to challenging segmentation conditions. Videos showcasing our results are available at: augmented-reality-for-robots.github.io