GUIDES: Guidance Using Instructor-Distilled Embeddings for Pre-Trained Robot Policy Enhancement
Minquan Gao, Xinyi Li, Qing Yan, Xiaojian Sun, Xiaopan Zhang, Chien-Ming Huang, Jiachen Li
AI summary
Problem
Pre-trained robot policies lack semantic awareness, but replacing them with foundation models is costly and risks discarding accumulated knowledge. The field needs a way to integrate foundation model capabilities into existing validated policies without modifying their core architectures.
Approach
The framework distills step-wise instructions from a fine-tuned vision-language model into compact embeddings injected into the policy’s latent space, while an LLM-based Reflector monitors confidence and retrieves past execution examples to refine actions during inference.
Key results
- Architecture-agnostic framework preserving original policy weights while adding foundation model capabilities
- LLM-based inference-time reflection mechanism that diagnoses failures and retrieves relevant execution history
- Consistent task success rate improvements across transformer and diffusion policy architectures on RoboCasa
- Real-world UR5 robot deployment demonstrating enhanced motion precision for grasping sub-tasks
Why it matters
Offers the robotics community a practical, resource-efficient pathway to upgrade validated legacy policies with semantic reasoning capabilities without prohibitive re-validation costs or architectural overhauls.
Abstract
Pre-trained robot policies serve as the foundation of many validated robotic systems, which encapsulate extensive embodied knowledge. However, they often lack the semantic awareness characteristic of foundation models, and replacing them entirely is impractical in many situations due to high costs and the loss of accumulated knowledge. To address this gap, we introduce GUIDES, a lightweight framework that augments pre- trained policies with semantic guidance from foundation models without requiring architectural redesign. GUIDES employs a fine-tuned vision-language model (Instructor) to generate con- textual instructions, which are encoded by an auxiliary module into guidance embeddings. These embeddings are injected into the policy’s latent space, allowing the legacy model to adapt to this new semantic input through brief, targeted fine-tuning. For inference-time robustness, a large language model–based Reflector monitors the Instructor’s confidence and, when confi- dence is low, initiates a reasoning loop that analyzes execution history, retrieves relevant examples, and augments the VLM’s context to refine subsequent actions. Extensive validation in the RoboCasa simulation environment across diverse policy architectures shows consistent and substantial improvements in task success rates. Real-world deployment on a UR5 robot further demonstrates that GUIDES enhances motion precision for critical sub-tasks such as grasping. Overall, GUIDES offers a practical and resource-efficient pathway to upgrade, rather than replace, validated robot policies.