Graph-Of-Constraints Model Predictive Control for Reactive Multi-Agent Task and Motion Planning
Anastasios Manganaris, Jeremy Lu, Ahmed H. Qureshi, Suresh Jagannathan
AI summary
Problem
Existing reactive TAMP methods rely on static agent assignments and totally-ordered constraint sequences, which fail to handle parallel tasks, dynamic disturbances, and real-time reassignment in multi-agent systems.
Approach
The authors introduce Graph-of-Constraints (GoC) MPC, which represents tasks as a directed acyclic graph of geometric constraints over tracked keypoints, decomposing the optimization into waypoints, agent assignments, and timing splines for real-time reactive execution.
Key results
- Supports partially ordered tasks and dynamic agent coordination without static assignments.
- Achieves higher success rates, significantly faster TAMP computation, and shorter overall paths compared to baselines.
- Enables disturbance recovery and online adaptation using only visual observations of 3D keypoints.
- Demonstrated on bimanual manipulation tasks like block stacking, liquid pouring, and tablecloth folding.
Why it matters
Provides a model-free, real-time reactive planning framework that scales multi-agent manipulation to complex, disturbance-prone real-world environments.
Abstract
Sequences of interdependent geometric con- straints are central to many multi-agent Task and Motion Plan- ning (TAMP) problems. However, existing methods for handling such constraint sequences struggle with partially ordered tasks and dynamic agent assignments. They typically assume static assignments and cannot adapt when disturbances alter task allocations. To overcome these limitations, we introduce Graph- of-Constraints Model Predictive Control (GoC-MPC), a gener- alized sequence-of-constraints framework integrated with MPC. GoC-MPC naturally supports partially ordered tasks, dynamic agent coordination, and disturbance recovery. By defining con- straints over tracked 3D keypoints, our method robustly solves diverse multi-agent manipulation tasks—coordinating agents and adapting online from visual observations alone, without relying on training data or environment models. Experiments demonstrate that GoC-MPC achieves higher success rates, significantly faster TAMP computation, and shorter overall paths compared to recent baselines, establishing it as an efficient and robust solution for multi-agent manipulation under real- world disturbances. Our supplementary video and code can be found at https://sites.google.com/view/goc-mpc/home.