Tactile-Driven Dexterous In-Hand Writing Via Extrinsic Contact Sensing
Can Zhao, Lingzi Xie, Bidan Huang, Shuai WANG, Daolin Ma
AI summary
Problem
Dexterous in-hand manipulation with extrinsic contact remains challenging because existing methods often assume isolated objects or rely on fixed gripper contacts, limiting flexibility in real-world scenarios.
Approach
The authors combine reinforcement learning for finger motion with compliant wrist control, using fingertip tactile sensors to dynamically estimate object pose and external contact forces without visual input.
Key results
- Tactile-driven factor graph estimates object pose and contact states without vision
- Hybrid RL and compliant control framework enables stable in-hand and extrinsic contact
- Successful sim-to-real transfer via systematic calibration and domain randomization
- Real-world adaptation to writing tools with varying radius, mass, and friction
Why it matters
Advances robotic manipulation in unstructured environments by enabling reliable tool use and contact-heavy tasks without relying on vision or fixed grasps.
Abstract
Dexterous in-hand manipulation, especially involv- ing interactions between grasped objects and external environ- ments, remains a formidable challenge in robotics. This study tackles the complexities of in-hand manipulation under extrinsic contact through a representative three-finger handwriting task. We propose a hybrid arm-hand coordination framework that combines reinforcement learning with compliance control, offer- ing both flexibility and robustness. Leveraging tactile sensors embedded in each finger, our tactile-driven estimation model dynamically predicts in-hand object pose and external contact, eliminating the need for fixed contact states. The proposed frame- work is first validated in simulation, where it successfully executes diverse writing tasks with accurate contact sensing. Sim-to-Real transfer is achieved through systematic calibration of finger joints and tactile sensors, supported by domain randomization. Real- world experiments further demonstrate the system’s adaptability to writing tools with varying physical properties—such as radius, length, mass, and friction—while maintaining stability across dif- ferent trajectories. Also see https://inhandwriting.github.io/. This work advances robotic manipulation capabilities in unstructured environments.