Depth Transfer: Learning to See Like a Simulator for Real-World Drone Navigation
Hang Yu, Christophe De Wagter, Guido de Croon
AI summary
Problem
Sim-to-real transfer for vision-based drone navigation is hindered by the perception gap between ideal simulated depth maps and noisy, artifact-laden real-world stereo depth images.
Approach
A Variational Autoencoder encodes simulated ground-truth depth into a latent space for RL training, then uses adversarial domain adaptation to refine the encoder so real-world stereo depth aligns with that space.
Key results
- Nearly doubles obstacle avoidance success rate when switching from ground-truth to stereo depth in simulation
- Outperforms state-of-the-art baselines in the photo-realistic AvoidBench simulator without using its training data
- Successfully transfers to real-world indoor and outdoor drone navigation environments
- ResNet-based VAE with min-pooling dilation significantly improves latent representation quality and obstacle detail preservation
Why it matters
Enables safe, efficient deployment of reinforcement learning policies for autonomous drones by bridging the sim-to-real visual gap without costly real-world training or fine-tuning.
Abstract
Sim-to-real transfer is a fundamental challenge in robot learning. Discrepancies between simulation and reality can significantly impair policy performance, especially if it receives high-dimensional inputs such as dense depth estimates from vision. We propose a novel depth transfer method based on domain adaptation to bridge the visual gap between simulated and real-world depth data. A Variational Autoencoder (VAE) is first trained to encode ground-truth depth images from simulation into a latent space, which serves as input to a reinforcement learning (RL) policy. During deployment, the encoder is refined to align stereo depth images with this latent space, enabling direct policy transfer without fine-tuning. We apply our method to the task of autonomous drone navigation through cluttered environments. Experiments in IsaacGym show that our method nearly doubles the obstacle avoidance success rate when switching from ground-truth to stereo depth input. Furthermore, we demonstrate successful transfer to the photo-realistic simulator AvoidBench using only IsaacGym- generated stereo data, achieving superior performance com- pared to state-of-the-art baselines. Real-world evaluations in both indoor and outdoor environments confirm the effectiveness of our approach, enabling robust and generalizable depth-based navigation across diverse domains.