3DGS-Holo-Inspector: A Mixed Reality UAV Controller with 3D Gaussian Splatting Localization for Infrastructure Inspection
Syed Muhammad Raza Rizvi, Huaiyuan Weng, Chul Min Yeum
AI summary
Problem
Traditional UAV controllers and existing mixed reality interfaces are unintuitive, lack depth perception, and often rely on external markers or infrastructure, making safe and precise navigation in cluttered inspection environments difficult.
Approach
The system uses a LiDAR-RGB 3D Gaussian Splatting localization backbone to establish persistent, markerless alignment between a mixed reality headset and a UAV, allowing operators to define inspection goals via hand gestures that the drone autonomously executes.
Key results
- Markerless headset-UAV spatial alignment using 3D Gaussian Splatting rendering
- Holographic goal definition and pre-flight path visualization via natural hand gestures
- Autonomous UAV navigation to MR-defined viewpoints with real-time telemetry overlay
- Experimental validation achieving 0.090 m indoor and 0.119 m outdoor positional RMSE
Why it matters
Provides a scalable, markerless mixed reality control paradigm that reduces operator workload and crash risks while improving inspection data quality for civil infrastructure monitoring.
Abstract
Unmanned aerial vehicles (UAVs) are increasingly used for infrastructure inspection, but conventional joystick and first-person-view (FPV) controllers remain unintuitive, error-prone, and cognitively demanding, particularly in clut- tered or safety-critical environments. We present 3DGS-Holo- Inspector, a Mixed Reality (MR) UAV controller that combines holographic goal-setting with autonomous UAV navigation. Using natural hand gestures, operators can define and preview navigation goals directly in MR before flight, ensuring precise and safe data capture at inspection viewpoints. The system complements existing inspection pipelines by leveraging pre- built 3D maps (e.g., photogrammetry or LiDAR reconstruc- tions) to enable refinement of Regions of Interest (ROIs) where coverage is incomplete or the detail is insufficient. Robust headset–UAV alignment is achieved through a LiDAR–RGB 3D Gaussian Splatting (3DGS) localization backbone, which provides dense, markerless, and persistent spatial registration in both indoor and outdoor settings. Once goals are placed, the UAV autonomously navigates to the specified pose, with real-time telemetry and live video overlaid in MR to enhance situational awareness. Experimental validation using a ModalAI Starling UAV and Microsoft HoloLens 2 demonstrated accurate UAV-goal alignment, achieving a positional Root Mean Square Error (RMSE) of 0.090 m (median = 0.084 m) indoors and 0.119 m (median = 0.118 m) outdoors, with orientation (yaw) RMSEs of 1.491◦(median = 1.400◦) and 2.233◦(median = 2.268◦), respectively. These results confirm that 3DGS-Holo- Inspector provides reliable MR-based UAV control, augmenting inspection workflows by enabling safe, intuitive, and high- precision UAV operations in real-world environments.