ModuLoop: Low-Level Code Generation Using Modular Synthesizer and Closed-Loop Debugger for Robotic Control
GINA Yoon, SUMIN LEE, Joo Yong Sim
AI summary
Problem
LLMs struggle with low-level robotic control due to their open-loop nature, lack of real-time physical feedback, and poor generalization in precise, environment-dependent tasks.
Approach
The framework decomposes natural language commands into modular Python code, executes it on a robot, and iteratively refines the code using a closed-loop debugger that analyzes runtime errors and accuracy metrics to insert diagnostic probes and apply fixes.
Key results
- 96.67% code generation success and 86.67% calibration accuracy on a UR3 arm
- 100% accuracy in simulation-based workspace reachability validation
- Probing-based debugging significantly accelerated convergence over open-loop baselines
- Successfully extended to a real-world pick-and-place manipulation task
Why it matters
It provides a practical, generalizable pathway for deploying LLMs as autonomous agents in real-world robotics without task-specific fine-tuning.
Abstract
Large Language Models (LLMs) have demonstrated impressive performance across various domains, including code generation and problem solving. However, their application in robotic control—particularly in low-level tasks that require precise manipulation, real-time feedback, and environment- dependent execution—remains limited. To address this challenge, we propose the Closed-Loop Modular Code Synthesizer frame- work. This framework leverages a pre-trained LLM without any task-specific fine-tuning to perform modular code planning and generation, and iteratively executes the generated code while inserting debugging probes to observe its behavior. This closed- loop structure facilitates systematic debugging and refinement, ultimately producing executable control programs. We apply the proposed framework to the calibration of an RGB-D camera and a robotic arm, validating its effectiveness in real-world settings. Furthermore, through a subsequent pick-and-place task, we demonstrate not only the accuracy of the calibration but also the potential extensibility of the framework. Across both tasks, the framework achieved high execution accuracy and autonomy, illustrating the practicality and scalability of LLM-based robotic control using our framework.