Grasp-MPC: Closed-Loop Visual Grasping Via Value-Guided Model Predictive Control
Jun Yamada, Adithyavairavan Murali, Ajay Uday Mandlekar, Clemens Eppner, Ingmar Posner, Balakumar Sundaralingam
AI summary
Problem
Open-loop grasping methods lack real-time feedback and fail with prediction errors in clutter, while existing closed-loop approaches struggle to generalize to novel objects and enforce safety constraints like collision avoidance.
Approach
A vision-based value function is trained on 2 million synthetic grasp trajectories to predict success likelihood, then used as a cost term within a real-time model predictive control framework to guide reactive, collision-aware grasping.
Key results
- Up to 32.6% simulation and 33.3% real-world success improvement over baselines
- Large-scale synthetic dataset of 2M+ trajectories across 8,515 Objaverse objects
- Robust performance under noisy pre-grasp poses and predicted grasp targets
- Unifies data-driven value estimation with constraint-aware MPC for reactive control
Why it matters
Advances practical robotic manipulation by enabling safe, generalizable grasping in unstructured environments without requiring extensive real-world training data.
Abstract
Grasping of diverse objects in unstructured envi- ronments remains a significant challenge. Open-loop grasping methods, effective in controlled settings, struggle in cluttered environments. Grasp prediction errors and object pose changes during grasping are the main causes of failure. In contrast, closed-loop methods address these challenges in simplified settings (e.g., single object on a table) on a limited set of objects, with no path to generalization. We propose Grasp- MPC, a closed-loop 6-DoF vision-based grasping policy designed for robust and reactive grasping of novel objects in clut- tered environments. Grasp-MPC incorporates a value function, trained on visual observations from a large-scale synthetic dataset of 2 million grasp trajectories that include successful and failed attempts. We deploy this learned value function in an MPC framework in combination with other cost terms that encourage collision avoidance and smooth execution. We evaluate Grasp-MPC on FetchBench and real-world settings across diverse environments. Grasp-MPC improves grasp suc- cess rates by up to 32.6% in simulation and 33.3% in real-world noisy conditions, outperforming open-loop, diffusion policy, transformer policy, and IQL approaches. Videos and more at http://grasp-mpc.github.io.