Simultaneous Extrinsic Contact and In-Hand Pose Estimation Via Distributed Tactile Sensing
Mark Van der Merwe, Kei Ota, Dmitry Berenson, Nima Fazeli, Devesh Jha
AI summary
Problem
Precise in-hand object pose and extrinsic contact estimation is critical for prehensile manipulation, but tactile sensors alone are highly ambiguous and visual feedback frequently fails due to occlusions and noise.
Approach
The authors introduce TacGraph, a factor-graph estimator that fuses object-agnostic tactile models with physical constraints—geometric consistency, non-penetration, contact kinematics, and force balance—to jointly solve for pose and contacts.
Key results
- Object-agnostic tactile models for extracting geometric and force signals
- Factor-graph framework (TacGraph) enforcing physical contact constraints
- Superior pose and contact estimation accuracy over ICP, CHSEL, and SCOPE baselines
- Robust tactile-only performance where vision-based methods fail
Why it matters
Provides a robust, vision-independent perception pipeline for contact-rich robotic manipulation tasks like assembly and tool use.
Abstract
Prehensile autonomous manipulation, such as peg insertion, tool use, or assembly, require precise in-hand under- standing of the object pose and the extrinsic contacts made during interactions. Providing accurate estimation of pose and contacts is challenging. Tactile sensors can provide local geometry at the sensor and force information about the grasp, but the locality of sensing means resolving poses and contacts from tactile alone is often an ill-posed problem, as multiple configurations can be consistent with the observations. Adding visual feedback can help resolve ambiguities, but can suffer from noise and occlusions. In this work, we propose a method that pairs local observations from sensing with the physical constraints of contact. We pro- pose a set of factors that ensure local consistency with tactile observations as well as enforcing physical plausibility, namely, that the estimated pose and contacts must respect the kinematic and force constraints of quasi-static rigid body interactions. We formalize our problem as a factor graph, allowing for efficient estimation. In our experiments, we demonstrate that our method outperforms existing geometric and contact-informed estimation pipelines, especially when only tactile information is available. Video results can be found at tacgraph.github.io.