Sparse Meets Dense: Correspondence Guided Robotic Manipulation with Rigid-Deformable Interactions
Ziyu Zhu, Yue Chen, Xirui Liang, Hojin Bae, Yuran Wang, Zhen Yuan, Ruihai Wu, Hao Dong
AI summary
Problem
Existing object-centric representations fail to capture both fine-grained contact interactions and high-dimensional deformations, making rigid-deformable manipulation tasks like dressing or hanging clothes difficult to generalize from minimal data.
Approach
The method combines task-aware sparse keypoints to define interaction constraints with dense correspondences to track deformations, guiding constraint-based motion planning from a single demonstration.
Key results
- Structure-, task-, and interaction-aware sparse keypoints for constraint formulation
- Dense correspondence module for robust tracking on deforming bodies
- One-shot generalization to novel object shapes via dense feature mapping
- Validated success across four simulation and real-world rigid-deformable manipulation tasks
Why it matters
Provides a data-efficient, generalizable representation that empowers household robots to handle complex contact-rich tasks with minimal demonstrations.
Abstract
Manipulation involving rigid-deformable interac- tions, such as hanging clothes or dressing humans, is common in daily life, making it essential for household robots. Com- pared to single-object manipulation or interactions between rigid bodies, these tasks are particularly challenging due to the rich multi-point contacts and the complex dynamics of the deformable bodies during interaction. Therefore, object- centric representations such as 6D poses or structural points without task-specific information become insufficient for these interactions. In this work, we propose a hybrid correspondence- based representation tailored for rigid-deformable interactions. First, to capture intricate interaction information, we introduce structure-, task-, and interaction-aware sparse keypoints. The keypoints are generated based on the global structures of both rigid and deformable objects, and filtered by their local interac- tion contacts. However, tracking these sparse keypoints through the interaction remains difficult due to the high-dimensional dynamics of deformable objects. Therefore, we further con- struct dense correspondences on the deformable objects for accurate keypoint tracking throughout the manipulation. This hybrid design combines the advantages of both representations: sparse keypoints encode rich, task-specific information for fine-grained manipulation, while dense correspondences ensure efficient tracking and generalization to novel deformations, shapes, and scenarios. Together, they enable one-shot transfer to new tasks with minimal demonstrations. Extensive experiments demonstrate the effectiveness and broad applicability of our method. Project Page: https://sparse-meets-dense.github.io/.