A Large Language Model-Based Mission Manager for Autonomous UAV Control
Milos Cihlar
AI summary
Problem
Traditional UAV control relies on rigid behavior trees or state machines that lack adaptability in dynamic environments and require manual reprogramming for new tasks, creating a scalability bottleneck.
Approach
The system uses a local LLM in an iterative think-act loop to interpret natural language commands and dynamically select from a constrained registry of ROS 2 actions, guided by real-time telemetry and perception data.
Key results
- Successfully executed 18 out of 20 simulation missions across four complexity levels
- Automatically decomposed natural language commands into multi-step ROS 2 action sequences
- Demonstrated reactive adaptation to real-time telemetry and perception feedback
- Maintained predictable execution metrics with bounded LLM and tool call counts
Why it matters
Enables scalable, adaptive UAV autonomy for dynamic missions without requiring mission-specific programming, benefiting researchers and developers in autonomous robotics.
Abstract
Autonomous unmanned aerial vehicles (UAVs) are traditionally controlled using behavior trees or state machines, which provide deterministic execution but limited adaptability in dynamic environments. Extending these conventional systems to handle new tasks requires manual specification of additional nodes or transitions, creating a scalability challenge as mission complexity increases. This work introduces a high-level mission manager leveraging local Large Language Models (LLMs) for autonomous UAV control. The system allows operators to issue high-level commands in natural language, which the LLM interprets and decomposes into sequences of ROS 2 actions, such as takeoff, navigation, object localization, and landing, without mission-specific programming. The LLM does not directly control the UAV but selects from a constrained set of tools mapped to ROS 2 actions or services. Real-time robot state is injected into the model context, ensuring that decisions are based on actual system status and environment perception.