SplatCtrl: Perception�Action Coupling Via Gaussian Scene Representations and Reactive Robot Control
Siddarth Jain, Ho Jin Choi
AI summary
Problem
Robotic manipulators struggle in unstructured, dynamic settings where traditional scene representations lack the fidelity, efficiency, and adaptability required for real-time reactive control.
Approach
The framework extends 3D Gaussian Splatting with voxel-based filtering and dynamic Gaussian relocation for real-time scene updates, then derives continuous signed distance functions from these Gaussians to integrate with control barrier functions for reactive motion generation.
Key results
- Comparable reconstruction quality to standard 3D-GS with a fixed Gaussian budget
- Real-time, differentiable collision probability estimation via Gaussian Process Distance Fields
- Successful simulation, physical robot, and human-robot workspace validation for reactive control
- First real-time system achieving full 6-DoF collision-free control alongside dynamic RGB-D scene reconstruction
Why it matters
It provides a computationally efficient, unified perception-control pipeline that allows robots to safely and adaptively operate in unpredictable, real-world settings without relying on offline training or prior maps.
Abstract
Robotic manipulators excel in structured envi- ronments but face substantial challenges in unstructured and dynamic settings. This paper presents SplatCtrl, a unified framework for real-time scene reconstruction and reactive robot motion generation to enable collision-free robotic arm control in previously unseen and continuously changing environments. Building on 3D Gaussian Splatting (3D-GS), we introduce a hybrid voxel-based filtering and dynamic Gaussian relocation strategy that supports efficient scene reconstruction from RGB- D streams while accommodating environmental changes. For safe and reactive control, we further propose a method for deriving continuous signed distance functions from isotropic Gaussians, providing stable and differentiable collision prob- ability estimates that bridge classical distance fields with the modern implicit representation. These continuous distance met- rics are incorporated into control barrier functions, resulting in a unified perception–action coupling framework that supports smooth and reliable real-time motion generation in response to scene changes. Experimental validation in simulation, on physical robot, and within shared human–robot workspace demonstrates the framework’s effectiveness, achieving inte- grated scene reconstruction and reactive control in uncertain, and dynamic environments.