Transformer-Based Multi-Agent Reinforcement Learning for Generalization of Heterogeneous Multi-Robot Cooperation

Recent advances in multi-agent reinforcement learning (MARL) have significantly enhanced cooperation ca- pabilities within multi-robot teams. However, the application to heterogeneous teams poses the critical challenge of com- binatorial generalization—adapting learned policies to teams with new compositions of varying sizes and robots capabilities. This challenge is paramount for dynamic real-world scenarios where teams must swiftly adapt to changing environmental and task conditions. To address this, we introduce a novel transformer-based MARL method for heterogeneous multi- robot cooperation. Our approach leverages graph neural net- works and self-attention mechanisms to effectively capture the intricate dynamics among heterogeneous robots, facilitating policy adaptation to team size variations. Moreover, by treating robot team decisions as sequential inputs, a capability-oriented decoder is introduced to generate actions in an auto-regressive manner, enabling decentralized decision-making that tailored each robot’s varying capabilities and heterogeneity type. Fur- thermore, we evaluate our method across two heterogeneous cooperation scenarios in both simulated and real-world en- vironments, featuring variations in team number and robot capabilities. Comparative results reveal our method’s supe- rior generalization performance compared to existing MARL methodologies, marking its potential for real-world multi-robot applications.