Tactile Hide and Seek: Bimanual Object Blind Search and Retrieval Via Tactile-Only Feedback
Xiangyu Fu, Hao Xing, Simon Armleder, Wenlan Shen, Fengyi Wang, Julio Rogelio Guadarrama Olvera, Gordon Cheng
AI summary
Problem
Vision-based object recognition fails in dark, smoky, or occluded environments, leaving a critical gap for robots that must rely solely on touch to search and identify objects autonomously.
Approach
The system uses a tactile hide-and-seek strategy involving exploratory sweeping, single-handed push-based classification, and a bimanual verification stage that combines size measurement and multimodal tactile fusion to confirm object identity and weight.
Key results
- 91.1% object and 83.1% weight classification accuracy on the new HAS dataset
- 61.4% real-world identification success rate in physical trials
- Bimanual verification corrects up to 17.6% of single-hand errors
- Introduction of the HAS dataset with 1.1 million multimodal tactile samples
Why it matters
Enables reliable autonomous object retrieval for disaster response, industrial inspection, and domestic assistance in environments where cameras are ineffective.
Abstract
Locating and identifying objects in vision-denied environments is a critical challenge for intelligent robot systems. To address the limitation of vision, we present a tactile- only method for object search and recognition using custom tactile skin sensors on robot hands. The method involves searching an object in a vision-denied environment with a tactile “hide and seek” strategy. Upon contact, the system employs a novel two-phase classification process: an initial single- handed classification by pushing the object, followed by a two- handed verification stage that incorporates size measurement to confirm the object’s identity and reduce critical errors. To support this approach, we introduce the HAS (Hide-and- Seek) dataset, a large-scale, multimodal tactile dataset of 1.1 million samples collected on a custom sensor hardware. Our system achieves an object classification accuracy of 91.1% and a weight classification accuracy of 83.1% on the HAS dataset, with a strict joint accuracy of 79.6%. The full online pipeline attains a 61.4% success rate in real-world identification, with the bimanual verification stage further correcting up to 17.6% of single-hand errors. Comprehensive ablation studies validate the contribution of individual sensor modalities and demonstrate the effectiveness of our tactile-only method for autonomous operation in a non-vision environment. Our project page is available at https://tactile-hide-and-seek. github.io/.