IMPACT: Intelligent Motion Planning with Acceptable Contact Trajectories Via Vision-Language Models
Yiyang Ling, Karan Owalekar, Oluwatobiloba Adesanya, Erdem Bıyık, Daniel Seita
AI summary
Problem
Traditional collision-free motion planning is often too restrictive in cluttered environments where some contact is necessary, but characterizing which contacts are benign versus dangerous is difficult.
Approach
The framework uses a VLM to assign semantic costs to objects and generates an anisotropic cost map encoding directional push safety for a contact-aware A* planner.
Key results
- 78% success rate in simulation, outperforming RRT, RRT*, and LAPP
- Lower object displacement (2.51 cm) compared to other VLM-based planners
- Validated across 3200 simulation trials and 200 real-world trials
- Higher human preference rankings over alternative methods
Why it matters
This allows robots to operate in highly cluttered spaces where strict collision avoidance would make tasks infeasible.
Abstract
Motion planning involves determining a sequence of robot configurations to reach a desired pose, subject to movement and safety constraints. Traditional motion planning finds collision-free paths, but this is overly restrictive in clutter, where it may not be possible for a robot to accomplish a task without contact. In addition, contacts range from relatively benign (e.g., brushing a soft pillow) to more dangerous (e.g., toppling a glass vase), making it difficult to characterize which may be acceptable. In this paper, we propose IMPACT, a novel motion planning framework that uses Vision-Language Models (VLMs) to infer environment semantics, identifying which parts of the environment can best tolerate contact based on object properties and locations. Our approach generates an anisotropic cost map that encodes directional push safety. We pair this map with a contact-aware A* planner to find stable contact-rich paths. We perform experiments using 20 simulation and 10 real-world scenes and assess using task success rate, object displacements, and feedback from human evaluators. Our results over 3200 simulation and 200 real-world trials suggest that IMPACT enables efficient contact-rich motion planning in cluttered settings while outperforming alternative methods and ablations. Our project website is available at https://impact-planning.github.io/.