RoboHitch: Learning Visual Affordance from Disordered Keypoints for Hitch Knots Tying
Jiahui Zuo, Boyang Zhang, Fumin Zhang
AI summary
Problem
Robotic knot tying of deformable linear objects fails under self-occlusion because existing methods rely on precise, ordered topological tracking that breaks during complex bending and crossings.
Approach
The framework fuses untracked 3D rope keypoints and RGB images using a dynamic Graph Autoencoder and a Convolutional Autoencoder, then applies bidirectional cross-attention to predict pick, rotate, and place affordances.
Key results
- Permutation-invariant rope state representation from disordered keypoints
- Bidirectional cross-attention mechanism fusing geometric and visual features
- Successful real-world hitch knot tying on a single-arm robot under self-occlusion
- Robust performance across crossed and uncrossed rope initializations with limited data
Why it matters
It provides a robust, data-efficient pathway for robotic manipulation of deformable objects in occluded, unstructured environments relevant to surgery and industrial automation.
Abstract
Robotic manipulation of deformable linear ob- jects (DLOs) presents significant challenges due to complex dynamics and frequent self-occlusions. Existing robotic knot tying methods typically rely on precise topological state track- ing with ordered keypoints and explicit edge connectivity. This reliance makes them prone to failures due to tracking drift and topology mismatch caused by repeated bending and crossings during knot formation. To address these limitations, we introduce RoboHitch, a novel framework that learns to perform hitch knot tying from human demonstrations using only disordered 3D keypoints and RGB images. This eliminates the need for explicit topological order, allowing for more flexible manipulation. Our method employs a dynamic Graph Autoencoder to extract geometric features from untracked keypoints, complemented by a Convolutional Autoencoder that captures essential visual context. A bidirectional cross-attention mechanism then fuses these modalities to jointly predict pick and place affordances, facilitating implicit reasoning about the rope’s state and enabling knot tying under occlusion. Real- world experiments demonstrate the effectiveness and general- izability of our approach, successfully completing hitch knots in scenarios with self-occlusions.