Distracted Robot: How Visual Clutter Undermine Robotic Manipulation
Amir Rasouli, Montgomery Tucker Alban, Sajjad Pakdamansavoji, Zhiyuan Li, Zhanguang Zhang, Yangzheng Wu, Xuan Zhao
AI summary
Problem
Current robotic evaluation protocols lack a unified way to quantify visual clutter, making it difficult to systematically assess how distractors and environmental factors degrade policy performance.
Approach
The study proposes a dual-view feature congestion (DvFC) metric to measure scene clutter and uses it to systematically benchmark five vision-language-action models across thousands of simulated and real-world manipulation tasks.
Key results
- Visual clutter lowers policy success rates by up to 34%
- Different VLA models exhibit unique failure modes and low agreement on successful scenarios
- The DvFC clutter metric reliably predicts performance degradation
- Finetuning on enhanced distraction data mitigates but does not fully resolve clutter-induced failures
Why it matters
Offers a standardized evaluation framework to help researchers and developers identify and address the specific weaknesses of manipulation policies in cluttered real-world environments.
Abstract
In this work, we propose an evaluation protocol for examining the performance of robotic manipulation policies in cluttered scenes. Contrary to prior works, we approach evaluation from a psychophysical perspective, therefore we use a unified measure of clutter that accounts for environmental factors as well as the distractors quantity, characteristics, and arrangement. Using this measure, we systematically construct evaluation scenarios in both hyper-realistic simulation and real- world and conduct extensive experimentation on manipulation policies, in particular vision-language-action (VLA) models. Our experiments highlight the significant impact of scene clutter, lowering the performance of the policies, by as much as 34% and show that despite achieving similar average performance across the tasks, different VLA policies have unique vulnerabilities and a relatively low agreement on success scenarios. We further show that our clutter measure is an effective indicator of performance degradation and analyze the impact of distractors in terms of their quantity and occluding influence. At the end, we show that finetuning on enhanced data, although effective, does not equally remedy all negative impacts of clutter on performance.