UNIC: Learning Unified Multimodal Extrinsic Contact Estimation
Zhengtong Xu, Yuki Shirai
AI summary
Problem
Existing extrinsic contact estimation methods rely on restrictive assumptions like predefined contact types, fixed grasps, or camera calibration, limiting their generalization and deployment in unstructured environments.
Approach
UNIC directly encodes camera-frame point clouds and fuses them with proprioceptive and tactile data using a unified affordance map representation and a random masking-based multimodal fusion mechanism, eliminating the need for priors or calibration.
Key results
- 9.6 mm average Chamfer distance error on unseen contact locations
- Effective generalization to entirely unseen objects
- Robust performance under missing modalities at deployment
- Adaptation to dynamic camera viewpoints without retraining
Why it matters
Provides a practical, task-agnostic contact estimation capability that enhances robotic dexterity and safety for real-world manipulation in unstructured settings.
Abstract
Contact-rich manipulation requires reliable es- timation of extrinsic contacts—the interactions between a grasped object and its environment—which provide essential contextual information for planning, control, and policy learn- ing. However, existing approaches often rely on restrictive assumptions, such as predefined contact types, fixed grasp con- figurations, or camera calibration, that hinder generalization to novel objects and deployment in unstructured environments. In this paper, we present UNIC, a unified multimodal framework for extrinsic contact estimation that operates without any prior knowledge or camera calibration. UNIC directly encodes visual observations in the camera frame and integrates them with proprioceptive and tactile modalities in a fully data-driven manner. It introduces a unified contact representation based on scene affordance maps that captures diverse contact formations and employs a multimodal fusion mechanism with random masking, enabling robust multimodal representation learning. Extensive experiments demonstrate that UNIC performs reliably. It achieves a 9.6 mm average Chamfer distance error on unseen contact locations, performs well on unseen objects, re- mains robust under missing modalities, and adapts to dynamic camera viewpoints. These results establish extrinsic contact estimation as a practical and versatile capability for contact-rich manipulation. The overview and hardware experiment videos are here.