In-The-Wild Compliant Manipulation with UMI-FT
Hojung Choi, Yifan Hou, Chuer Pan, Seongheon Hong, Austin Patel, Xiaomeng Xu, Mark Cutkosky, Shuran Song
AI summary
Problem
Existing handheld data collection systems lack integrated force sensing, relying on bulky, expensive, or fragile commercial sensors that hinder scalable, force-aware policy learning for delicate or forceful manipulation.
Approach
The authors introduce UMI-FT, a modified handheld gripper with compact per-finger six-axis force/torque sensors and an iPhone for vision, to collect multimodal demonstration data. They train an Adaptive Compliance Policy on this data to predict position, grasp force, and stiffness targets for standard compliance controllers.
Key results
- Open-sourced hardware and software for scalable per-finger force/torque data collection
- 92% success rate in whiteboard wiping across diverse unseen environmental variations
- Reliable regulation of both external contact and internal grasp forces in contact-rich tasks
- Significant performance gains over baselines lacking compliance or force sensing in lightbulb insertion and zucchini skewering
Why it matters
Enables scalable, cost-effective learning of compliant manipulation from in-the-wild demonstrations, advancing safe and robust robot interaction with fragile or dynamic environments.
Abstract
Many manipulation tasks require careful force modulation. With insufficient force the task may fail, while excessive force could cause damage. The high cost, bulky size and fragility of commercial force/torque (F/T) sensors have limited large-scale, force-aware policy learning. We introduce UMI-FT, a handheld data-collection platform that mounts compact, six-axis force/torque sensors on each finger, enabling finger-level wrench measurements alongside RGB, depth, and pose. Using the multimodal data collected from this device, we train an adaptive compliance policy that predicts position targets, grasp force, and stiffness for execution on standard compliance controllers. In evaluations on three contact-rich, force-sensitive tasks (whiteboard wiping, skewering zucchini, and lightbulb insertion), UMI-FT enables policies that reliably regulate external contact forces and internal grasp forces, outperforming baselines that lack compliance or force sensing. UMI-FT offers a scalable path to learning compliant ma- nipulation from in-the-wild demonstrations. We open-source the hardware and software to facilitate broader adoption at: https://umi-ft.github.io/.