CommCP: Efficient Multi-Agent Coordination Via LLM-Based Communication with Conformal Prediction
Xiaopan Zhang, Zejin Wang, Zhixu Li, Jianpeng Yao, Jiachen Li
AI summary
Problem
Cooperative multi-robot teams must efficiently gather information to answer natural language questions in shared environments, but uncalibrated LLM communication often shares irrelevant or misleading data that hinders exploration.
Approach
CommCP employs conformal prediction to statistically calibrate the confidence of LLM-generated messages, ensuring robots only share relevant, high-confidence information to guide decentralized exploration.
Key results
- Formulation of the multi-agent multi-task Embodied Question Answering (MM-EQA) problem
- Development of CommCP, a decentralized LLM communication framework with conformal prediction calibration
- Release of a novel MM-EQA benchmark featuring photo-realistic HM3D household scenarios
- Demonstrated significant improvements in task success rate and exploration efficiency over baselines
Why it matters
Provides a reliable communication protocol for heterogeneous robot teams, accelerating the practical deployment of cooperative service robots in real-world environments.
Abstract
To complete assignments provided by humans in natural language, robots must interpret commands, generate and answer relevant questions for scene understanding, and ma- nipulate target objects. Real-world deployments often require multiple heterogeneous robots with different manipulation ca- pabilities to handle different assignments cooperatively. Beyond the need for specialized manipulation skills, effective informa- tion gathering is important in completing these assignments. To address this component of the problem, we formalize the information-gathering process in a fully cooperative setting as an underexplored multi-agent multi-task Embodied Question Answering (MM-EQA) problem, which is a novel extension of canonical Embodied Question Answering (EQA), where effective communication is crucial for coordinating efforts without redundancy. To address this problem, we propose CommCP, a novel LLM-based decentralized communication framework designed for MM-EQA. Our framework employs conformal prediction to calibrate the generated messages, thereby minimizing receiver distractions and enhancing com- munication reliability. To evaluate our framework, we intro- duce an MM-EQA benchmark featuring diverse, photo-realistic household scenarios with embodied questions. Experimental results demonstrate that CommCP significantly enhances the task success rate and exploration efficiency over baselines. The experiment videos, code, and dataset are available on our project website: https://comm-cp.github.io.