Learning Geometry-Aware Nonprehensile Pushing and Pulling with Dexterous Hands
Yunshuang Li, Yiyang Ling, Gaurav Sukhatme, Daniel Seita
AI summary
Problem
High-DOF dexterous hands remain underutilized for nonprehensile manipulation due to the difficulty of modeling complex hand-object contact dynamics and planning feasible contact-rich motions.
Approach
GD2P generates a large-scale dataset of valid poses via physics-based optimization, trains a geometry-conditioned diffusion model to predict diverse pre-contact hand poses, and uses motion planning to select and execute the best action in real-world trials.
Key results
- 1.3-million-pose dataset across 2.3k objects
- Geometry-conditioned diffusion model for pose synthesis
- 840 real-world trials outperforming fixed and nearest-neighbor baselines
- Scalable applicability across Allegro and LEAP hand morphologies
Why it matters
Provides a scalable, learning-based route for manipulating complex or top-heavy objects that are difficult to grasp, advancing general-purpose dexterous manipulation.
Abstract
Nonprehensile manipulation, such as pushing and pulling, enables robots to move, align, or reposition objects that may be difficult to grasp due to their geometry, size, or relationship to the robot or the environment. Much of the existing work in nonprehensile manipulation relies on parallel- jaw grippers or tools such as rods and spatulas. In contrast, multi-fingered dexterous hands offer richer contact modes and versatility for handling diverse objects to provide stable support over the objects, which compensates for the difficulty of mod- eling the dynamics of nonprehensile manipulation. Therefore, we propose Geometry-aware Dexterous Pushing and Pulling (GD2P) for nonprehensile manipulation with dexterous robotic hands. We study pushing and pulling by framing the problem as synthesizing and learning pre-contact dexterous hand poses that lead to effective manipulation. We generate diverse hand poses via contact-guided sampling, filter them using physics simulation, and train a diffusion model conditioned on object geometry to predict viable poses. At test time, we sample hand poses and use standard motion planners to select and execute pushing and pulling actions. We perform extensive real-world experiments with an Allegro Hand and a LEAP Hand, demonstrating that GD2P offers a scalable route for generating dexterous nonprehensile manipulation motions with its applicability to different hand morphologies. Our project website is available at: geodex2p.github.io. All authors are with the Thomas Lord Department of Computer Science at the University of Southern California, USA. Correspondence: yunshuan@usc.edu