Grounding Conversational Robots on Vision through Dense Captioning and Large Language Models

This work explores a novel approach to empow- ering robots with visual perception capabilities using textual descriptions. Our approach involves the integration of GPT-4 with dense captioning, enabling robots to perceive and interpret the visual world through detailed text-based descriptions. To assess both user experience and the technical feasibility of this approach, experiments were conducted with human partici- pants interacting with a Pepper robot equipped with visual capabilities. The results affirm the viability of the proposed approach, allowing to perform vision-based conversations ef- fectively, despite processing time limitations.