Lightweight Visual Reasoning for Socially-Aware Robots
Alessio Galatolo, Ronald Cumbal, Alexandros Rouchitsas, Katie Winkle, Didem Gurdur Broo, Ginevra Castellano
AI summary
Problem
Robots in shared human environments struggle to interpret complex, dynamic human behaviors due to shallow cross-modal integration in current Vision-Language Models.
Approach
The method projects an LLM’s image-token hidden states through a gated MLP back into the vision encoder, creating a two-pass reasoning loop that reinterprets visuals based on textual context.
Key results
- Improved Qwen 2.5 (7B) navigation distance by 3.3% and intention recognition accuracy by 2.93%
- Boosted Gemma 3 and LLaVA OV 1.5 intention recognition accuracy by up to 10.81%
- Increased open-ended scene description scores across all tested models
- Released a novel first-person human intention recognition dataset for robotics
Why it matters
Provides a scalable, parameter-efficient way to enhance robotic perception and social awareness without retraining foundational models.
Abstract
Robots operating in shared human environments must not only navigate, interact, and detect their surroundings, they must also interpret and respond to dynamic, and often un- predictable, human behaviours. Although recent advances have shown promise in enhancing robotic perception and instruction- following using Vision-Language Models (VLMs), they remain limited in addressing the complexities of multimodal human- robot interactions (HRI). Motivated by this challenge, we introduce a lightweight language-to-vision feedback module that closes the loop between an LLM and the vision encoder in VLMs. The module projects image-token hidden states through a gated Multi-Layer Perceptron (MLP) back into the encoder input, prompting a second pass that reinterprets the scene under text context. We evaluate this approach on three robotics- centred tasks: navigation in a simulated environment (Habitat), sequential scene description (Mementos-Robotics), and human- intention recognition (our HRI dataset). Results show that our method improves Qwen 2.5 (7B) by 3.3% (less distance), +0.057 description score, and +2.93% accuracy, with less than 3% extra parameters; Gemma 3 (4B) and LLaVA OV 1.5 (4B) show mixed navigation results but gains +0.111,+0.055 and +10.81%,+4.79% on the latter two tasks.