Affordance RAG: Hierarchical Multimodal Retrieval with Affordance-Aware Embodied Memory for Mobile Manipulation
Ryosuke Korekata, Quanting Xie, Yonatan Bisk, Komei Sugiura
AI summary
Problem
Robots struggle to execute free-form language instructions in complex indoor environments because existing retrieval methods confuse visually similar objects and ignore physical manipulability constraints.
Approach
The method builds a hierarchical embodied memory from pre-explored images and retrieves targets by fusing regional and visual semantics, then reranks candidates using VLM-predicted affordance scores and LLM-based relevance.
Key results
- Outperforms baselines in retrieval accuracy on the WholeHouse-MM benchmark
- Achieves 85% task success rate in real-world indoor mobile manipulation
- Introduces a zero-shot hierarchical framework fusing regional and visual semantics
- Filters and reranks candidates using affordance-aware reasoning to ensure physical executability
Why it matters
Enables reliable, zero-shot robot task execution in real-world settings by bridging the gap between semantic matching and physical manipulability.
Abstract
In this study, we address the problem of open- vocabulary mobile manipulation, where a robot is required to carry a wide range of objects to receptacles based on free- form natural language instructions. This task is challenging, as it involves understanding visual semantics and the affordance of manipulation actions. To tackle these challenges, we propose Affordance RAG, a zero-shot hierarchical multimodal retrieval framework that constructs Affordance-Aware Embodied Memory from pre-explored images. The model retrieves candidate targets based on regional and visual semantics and reranks them with affordance scores, allowing the robot to identify manipulation options that are likely to be executable in real-world environ- ments. Our method outperformed existing approaches in retrieval performance for mobile manipulation instruction in large-scale indoor environments. Furthermore, in real-world experiments where the robot performed mobile manipulation in indoor envi- ronments based on free-form instructions, the proposed method achieved a task success rate of 85%, outperforming existing methods in both retrieval performance and overall task success.