Instrumentation for Imitation Learning: Enhancing Training Datasets for Clothes Hanger Insertion
Remko Proesmans, Thomas Lips, Francis wyffels
AI summary
Problem
Robotic imitation learning requires massive datasets and struggles with complex, unstructured tasks like cloth manipulation, while large behavior models lack the physical data needed for real-world deployment.
Approach
The researchers instrument a clothes hanger with embedded IR sensors to provide privileged state information during teleoperation, then train diffusion policies with and without this data, and finally use the instrumented policy's successful rollouts to augment training for a vision-only student policy.
Key results
- Instrumented policies outperform vision-only counterparts by 14–25 percentage points in success rate
- Black-box imitation learning automatically prioritizes sensor signals without explicit guidance
- Augmenting teleoperation data with instrumented expert rollouts boosts vision-only policy performance by 12 percentage points
- Instrumentation enhances policy task awareness and reduces critical failure modes like dropping the garment
Why it matters
Provides a scalable, low-cost method to overcome data scarcity and improve robustness in real-world robotic manipulation.
Abstract
Large behaviour models have transformed the field of robotic manipulation, but prohibitive data requirements have thus far prevented a revolution similar to vision language models. We believe that instrumentation, i.e. sensor integration in objects, can provide invaluable state information and enable efficient learning for robotic manipulation. In this paper, we present instrumented imitation learning of clothes hanger inser- tion. Using 180 teleoperated demonstrations, we train diffusion policies with and without access to instrumentation data. Re- sults show that policies leveraging instrumentation outperform vision-only counterparts by 14–25 %pt and exhibit greater task awareness. Crucially, a black-box imitation learning policy learns to prioritise instrumentation signals without explicit guidance. In addition, enhancing the teleoperation dataset with rollouts from an instrumented “expert” policy, enables a vision- only “student” policy to achieve performance comparable to the instrumented expert, thereby surpassing the original vision-only policy. These findings establish instrumentation as a promising strategy to enhance imitation learning for robotic manipulation. Datasets are available on Zenodo [10.5281/zenodo.17122216].