LLM-Aided Assistive Robot for Single-Operator Bimanual Teleoperation
Haolin Fei, Songlin Ma, Guanglong Du, Elmira Yadollahi, Hak-Keung Lam, Angela Faragasso, Allahyar Montazeri, Ziwei Wang
AI summary
Problem
Single-operator bimanual teleoperation imposes extreme cognitive and ergonomic burdens, while existing assistance methods rely on rigid, task-specific automation that fails to adapt to dynamic environments or varied operator needs.
Approach
The BTLA system enables a single operator to manually control one robotic arm while using natural language voice commands to direct an LLM that manages a second assistive arm with variable autonomy levels.
Key results
- 240.8% increase in task success rate over solo teleoperation
- 69.9% increase in success rate over dyadic teleoperation
- Significant reduction in operator mental workload across NASA-TLX dimensions
- 90% success rate on challenging physical dual-arm manipulation tasks
Why it matters
Makes complex dual-arm teleoperation accessible to single operators in hazardous or remote settings by drastically improving performance and reducing cognitive strain through intuitive voice control.
Abstract
Bimanual teleoperation tasks are highly demand- ing for human operators, requiring the simultaneous control of two robotic arms while managing complex coordination and cognitive load. Current approaches to this challenge often rely on rigid control schemes or task-specific automation that do not adapt well to dynamic environments or varied operator needs. This paper presents a novel large language model (LLM)-aided bimanual teleoperation assistant (BTLA) that helps operators control dual-arm robots through an intuitive voice command interface and variable autonomy. The BTLA system enables a hybrid control paradigm by combining natural language interaction for an assistive robot arm with direct teleoperation of the dominant robotic arm. Our system implements six core manipulation skills with varying autonomy, ranging from direct mirroring to autonomous object manipulation. The BTLA leverages the LLM to interpret natural language commands and select an appropriate assistance mode based on task re- quirements and operator preferences. Experimental validation on bimanual object manipulation tasks demonstrates that the BTLA system yields a 240.8% increase in success rate over solo teleoperation and a 69.9% increase over dyadic teleoperation, while significantly reducing operator mental workload. In addition, we validate our approach on a physical dual-arm UR3e robot system, achieving a 90% success rate on challenging soft-bottle handling and box-transportation tasks.