Task Robustness Via Re-Labelling Vision-Action Robot Data
Artur Kuramshin, Ozgur Aslan, Cyrus Neary, Glen Berseth
AI summary
Problem
Robot learning policies struggle to follow text instructions and generalize to new scenarios due to limited linguistic and trajectory diversity in existing datasets. Collecting diverse real-world demonstrations at scale is prohibitively expensive and slow.
Approach
TREAD leverages a pretrained Vision-Language Model to iteratively decompose long robot demonstrations into semantic sub-tasks, segment them temporally, and generate visually grounded, linguistically diverse instructions for each segment without collecting new data.
Key results
- A scalable framework for VLM-driven trajectory decomposition and relabeling
- An augmented dataset with linguistically diverse, visually grounded sub-task instructions
- Improved zero-shot motion generalization on novel scene-instruction pairs
- Enhanced language-conditioned policy robustness to varied instruction wording
Why it matters
Provides a cost-effective, scalable pathway to significantly improve robot instruction-following and generalization using existing datasets and off-the-shelf vision-language models.
Abstract
The recent trend in scaling models for robot learning has resulted in impressive policies that can perform various manipulation tasks and generalize to novel scenarios. However, these policies continue to struggle with following instructions, likely due to the limited linguistic and action sequence diversity in existing robotics datasets. This paper introduces Task Robustness via RE-Labelling Vision-Action Robot Data (TREAD), a scalable framework that leverages large Vision-Language Models (VLMs) to augment existing robotics datasets without additional data collection, harnessing the transferable knowledge embedded in these models. Our approach leverages a pretrained VLM through three stages: generating semantic sub-tasks from original instruction labels and initial scenes, segmenting demonstration videos conditioned on these sub-tasks, and producing diverse instructions that incorporate object properties, effectively decomposing longer demonstrations into grounded language-action pairs. We fur- ther enhance robustness by augmenting the data with linguisti- cally diverse versions of the text goals. Evaluations on LIBERO demonstrate that policies trained on our augmented datasets exhibit improved performance on novel, unseen tasks and goals. Our results show that TREAD enhances both planning generalization through trajectory decomposition and language- conditioned policy generalization through increased linguistic diversity. Project website: https://akuramshin.github.io/tread.