A Contact-Driven Framework for Manipulating in the Blind
Muhammad Suhail Saleem, Lai Yuan, Maxim Likhachev
AI summary
Problem
Robots struggle to manipulate objects in occluded or vision-inadequate spaces where visual sensing fails. Without reliable spatial awareness, they rely on sparse contact feedback but lack efficient methods to extrapolate unseen structures and plan collision-free paths.
Approach
The framework iteratively fuses torque-based contact detection with learned CNN and diffusion models to extrapolate workspace occupancy from structural priors, guiding a robust planner that accounts for localization noise while preserving path completeness.
Key results
- Torque-based contact detection and localization module using momentum observers and particle filters
- Learned CNN and diffusion predictors that extrapolate partial occupancy maps using structural priors
- Integration with Collision Hypothesis Sets and CMAX planners to maintain completeness under uncertainty
- Up to 2× reduction in task completion time on a real UR10e manipulator across domestic blind-manipulation tasks
Why it matters
Provides a practical, theoretically sound pathway for robots to operate reliably in cluttered, vision-deprived domestic and industrial environments using only proprioceptive feedback.
Abstract
Robots often face manipulation tasks in environ- ments where vision is inadequate due to clutter, occlusions, or poor lighting—for example, reaching a shutoff valve at the back of a sink cabinet or locating a light switch above a crowded shelf. In such settings, robots, much like humans, must rely on contact feedback to distinguish free from occupied space and navigate around obstacles. Many of these environments often exhibit strong structural priors—for instance, pipes often span across sink cabinets—that can be exploited to anticipate unseen structure and avoid unnecessary collisions. We present a theoretically complete and empirically efficient framework for manipulation in the blind that integrates contact feedback with structural priors to enable robust operation in unknown environments. The framework comprises three tightly coupled components: (i) a contact detection and localization module that utilizes joint torque sensing with a contact particle filter to detect and localize contacts, (ii) an occupancy estimation module that uses the history of contact observations to build a partial occupancy map of the workspace and extrapolate it into unexplored regions with learned predictors, and (iii) a planning module that accounts for the fact that contact localization estimates and occupancy predictions can be noisy, comput- ing paths that avoid collisions and complete tasks efficiently without eliminating feasible solutions. We evaluate the system in simulation and in the real world on a UR10e manipulator across two domestic tasks—(i) manipulating a valve under a kitchen sink surrounded by pipes and (ii) retrieving a target object from a cluttered shelf. Results show that the framework reliably solves these tasks, achieving up to a 2× reduction in task completion time compared to baselines, with ablations confirming the contribution of each module.