Grasp Independent Indirect Tool Force Estimation using Vision-based Tactile Sensors
LUCHEN LI, Thomas George Thuruthel
AI summary
Problem
Robotic systems struggle to adaptively use general-purpose tools because they cannot accurately estimate forces transmitted through the tool to its tip, relying instead on rigid, tool-specific physics models.
Approach
The method uses vision-based tactile sensors on a two-fingered gripper to capture high-resolution contact deformations, which a CNN with a global attention mechanism processes to implicitly predict indirect normal and shear forces.
Key results
- First demonstration of grasp-independent indirect force estimation for robotic tool use
- Accurate normal and shear force prediction across diverse tools and grasp poses with a single model
- Effective isolation of force-induced deformations using zero-force reference images and dual-sensor attention
- Validated generalization across varying tool geometries, weights, and surface textures
Why it matters
Enables robots to safely and adaptively use everyday human tools, advancing dexterous manipulation and real-world robotic applications.
Abstract
Humans possess the capability to seamlessly inte- grate tools into their body schema, enabling precise and adaptive interactions with the environment. This touch-mediated ability allows us to dexterously use tools in everyday tasks, an ability currently lacking in robotic systems. In this work, we propose a novel method for indirect force estimation in robotic tool use, a prerequisite for advanced tool use, leveraging vision-based tactile sensing (VTS) and deep learning techniques. By capturing high- resolution spatial deformations from tactile images, our model implicitly infers force transmission dynamics without requiring explicit knowledge of tool properties or material characteristics. We validate our approach across multiple tool types using a single trained machine learning model, demonstrating its generalization capability. This work represents the first demonstration of indirect force estimation for tool-mediated robotic interactions, offering a pathway toward more dexterous and adaptive robotic tool use in real-world applications.