FD-VLA: Force-Distilled Vision-Language-Action Model for Contact-Rich Manipulation
Ruiteng Zhao, Wenshuo Wang, Yicheng Ma, Xiaocong Li, Francis TAY, Marcelo H Ang Jr, Haiyue Zhu
AI summary
Problem
Vision-only and existing force-aware Vision-Language-Action models struggle with contact-rich tasks due to sensor fragility, high hardware costs, or disrupted pretrained semantics. There is a need for a robust, sensor-free approach that effectively integrates physical interaction cues without compromising model stability.
Approach
FD-VLA uses a Force Distillation Module to predict a latent force token from visual and proprioceptive inputs during training, which is then injected into a frozen pretrained VLM during inference. Directional attention masking preserves the model's vision-language alignment while enabling tight force-vision-state fusion for action generation.
Key results
- Proposes FD-VLA framework injecting distilled force tokens into frozen VLMs
- Designs FDM to predict latent force representations from vision and state inputs
- Enables sensor-free deployment while improving cross-modal alignment and robustness
- Achieves higher success rates on plug insertion, whiteboard cleaning, and button pressing tasks than baselines
Why it matters
Eliminates the need for expensive force-torque sensors in robotic systems while improving performance in contact-rich tasks, making advanced manipulation more accessible and cost-effective for industry and research.
Abstract
Force sensing is a crucial modality for Vision- Language-Action (VLA) frameworks, as it enables fine-grained perception and dexterous manipulation in contact-rich tasks. We present Force-Distilled VLA (FD-VLA), a novel framework that integrates force awareness into contact-rich manipulation without relying on physical force sensors. The core of our approach is a Force Distillation Module (FDM), which distills force by mapping a learnable query token, conditioned on visual observations and robot states, into a predicted force token aligned with the latent representation of actual force signals. During inference, this distilled force token is injected into the pretrained VLM, enabling force-aware reasoning while preserving the integrity of its vision-language semantics. This design provides two key benefits: first, it allows practical deployment across a wide range of robots that lack expensive or fragile force-torque sensors, thereby reducing hardware cost and complexity; second, the FDM introduces an additional force- vision-state fusion prior to the VLM, which improves cross- modal alignment and enhances perception-action robustness in contact-rich scenarios. Surprisingly, our physical experiments show that the distilled force token outperforms direct sensor force measurements as well as other baselines, which highlights the effectiveness of this force-distilled VLA approach.