Investigating the Role of Implicit Signals in Adaptive User-Aware Human-Robot Interactions
Bálint Gucsi, Nguyen Tan Viet Tuyen, Bing Chu, Danesh Tarapore, Long Tran-Thanh
AI summary
Problem
Social robots struggle to personalize interactions without causing user frustration, as relying solely on explicit feedback is demanding and implicit signals like facial expressions are inherently noisy and inconsistent. This work addresses how to effectively incorporate these involuntary social cues into adaptive decision-making while accounting for their limitations.
Approach
The authors built a user-aware framework that processes facial expressions, head movements, speech, and touch to infer user intent and affective state, then applies a frustration-constrained multi-armed bandit algorithm to dynamically adapt the robot's conversational style and recommendations.
Key results
- Successfully adapted robot conversational style to individual user characteristics
- Achieved 80% positive user feedback in a cafeteria-style user study
- Users reported higher perceived usefulness compared to baseline approaches
- Released a public dataset of over 5 hours of multimodal human-robot interaction data
Why it matters
This framework advances the deployment of socially acceptable, personalized service robots by demonstrating how to safely and effectively use natural non-verbal cues for real-time behavior adaptation.
Abstract
Our work investigates how social robots can act in a user-aware manner by adapting their behaviour to users’ personal characteristics and preferences without unnecessarily exposing them to frustration through the robot’s actions. In par- ticular, we investigate how implicit social signals inadvertently exhibited by users (e.g. facial expressions) during interactions can be incorporated into user-aware decision-making models while accounting for the systematic limitations of implicit feed- back signals (e.g. inconsistency, noise, culture and individual- dependence). Doing so, we develop a user-aware adaptive decision-making and learning framework for human-robot interactions, building on implicit signal processing, cue-based intent inference, and multiarmed bandit learning techniques. Evaluating our approach, we conduct a user study where participants interact with a Pepper robot in a cafeteria style interaction scenario, with the robot providing recommendations and taking orders while adapting its behaviour to individual users. The experimental results demonstrate our proposed model’s success in adapting its behaviour (i.e. conversational style) to users with different personal characteristics, while receiving 80% positive user feedback, and user questionnaire responses reporting higher perceived usefulness than baseline approaches. Questionnaire responses also illustrate positive user impressions of implicit signal based approaches while highlighting the importance of accounting for their limitations in learning models. In addition, we provide a dataset of over 5 hours of human and robot behaviour data extracted from 1School of Electronics and Computer Science, University of Southamp- ton, UK {bg1u17,tuyen.nguyen,b.chu,d.s.tarapore} @soton.ac.uk 2Department of Computer Science, University of Warwick, UK long.tran-thanh@warwick.ac.uk multimodal recordings captured as part of our user study.