Mixed-Reality Digital Twins: Leveraging the Physical and Virtual Worlds for Hybrid Sim2Real Transition of Multi-Agent Reinforcement Learning Policies
Chinmay Samak, Tanmay Samak, Venkat Krovi
AI summary
Problem
Multi-agent reinforcement learning training suffers from excessive wall-clock times and requires costly, unsafe physical testbeds for real-world deployment, while existing parallelization and domain randomization methods lack smart isolation and real-time synchronization.
Approach
The authors introduce a modular simulation parallelization framework that selectively isolates agent interactions and a bi-directional mixed-reality digital twin that immerses a few physical agents within a virtual environment to enable efficient training and hybrid sim2real transfer.
Key results
- Up to 76.3% reduction in training time via selective parallelization
- Sim2real transfer gap minimized to 2.9% through systematic domain randomization
- Open-source mixed-reality digital twin enabling hybrid deployment with minimal physical agents
- Validated across cooperative intersection traversal and competitive autonomous racing scenarios
Why it matters
Provides a scalable, cost-effective pathway for training and deploying multi-agent reinforcement learning policies in autonomous systems without relying on large-scale physical testbeds.
Abstract
Multi-agent reinforcement learning (MARL) for cyber-physical vehicle systems usually requires a significantly long training time due to their inherent complexity. Furthermore, de- ployingthetrainedpoliciesintherealworlddemandsafeature-rich environment along with multiple physical embodied agents, which may not be feasible due to monetary, physical, energy, or safety constraints. This work seeks to address these pain points by pre- senting a mixed-reality (MR) digital twin (DT) framework capable of: (i) boosting training speeds by selectively scaling parallelized simulation workloads on-demand, and (ii) immersing the MARL policies across hybrid simulation-to-reality (sim2real) experiments. Theviabilityandperformanceoftheproposedframeworkarehigh- lighted through two representative use cases, which cover coopera- tive as well as competitive classes of MARL problems. We study the effect of: (i) agent and environment parallelization on training time, and (ii) systematic domain randomization on zero-shot sim2real transfer, across both case studies. Results indicate up to 76.3% reduction in training time with the proposed parallelization scheme and sim2real gap as low as 2.9% using the proposed deployment method.