Bridging Large-Model Reasoning and Real-Time Control Via Agentic Fast-Slow Planning
Jiayi Chen, Shuai Wang,∗, Guangxu Zhu,∗, and Chengzhong Xu
AI summary
Problem
Translating high-level semantic intent from large foundation models into reliable, real-time control remains challenging due to latency, brittleness, and blurred reasoning-control interfaces in existing methods.
Approach
Agentic Fast-Slow Planning hierarchically separates perception, reasoning, planning, and control, using a VLM-to-LLM bridge for symbolic directives and a semantic-guided A* planner with agentic hyperparameter tuning tracked by real-time MPC.
Key results
- Hierarchical framework decoupling slow reasoning from fast control
- Perception2Decision module compresses scenes to ego-centric topologies for low-latency cloud reasoning
- Semantic-Guided A* planner incorporates language-derived soft costs with agentic hyperparameter refinement
- Up to 45% reduction in lateral deviation and over 12% faster completion time in CARLA simulations
Why it matters
Enables safe, interpretable, and real-time integration of large language models into autonomous systems without sacrificing control stability or latency.
Abstract
Large foundation models enable powerful rea- soning for autonomous systems, but mapping semantic intent to reliable real-time control remains challenging. Existing ap- proaches either (i) let Large Language Models (LLMs) gener- ate trajectories directly—brittle, hard to verify, and latency- prone—or (ii) adjust Model Predictive Control (MPC) objec- tives online—mixing slow deliberation with fast control and blurring interfaces. We propose Agentic Fast–Slow Planning, a hierarchical framework that decouples perception, reasoning, planning, and control across natural timescales. The framework contains two bridges. Perception2Decision compresses scenes into ego-centric topologies using an on-vehicle Vision–Language Model (VLM) detector, then maps them to symbolic driving directives in the cloud with an LLM decision maker—reducing bandwidth and delay while preserving interpretability. De- cision2Trajectory converts directives into executable paths: Semantic-Guided A∗embeds language-derived soft costs into classical search to bias solutions toward feasible trajectories, while an Agentic Refinement Module adapts planner hyperpa- rameters using feedback and memory. Finally, MPC tracks the trajectories in real time, with optional cloud-guided references for difficult cases. Experiments in CARLA show that Agentic Fast–Slow Planning improves robustness under perturbations, reducing lateral deviation by up to 45% and completion time by over 12% compared to pure MPC and an A∗-guided MPC baseline. Code is available at https://github.com/ cjychenjiayi/icra2026_AFSP.