ROOM: A Physics-Based Continuum Robot Simulator for Photorealistic Medical Datasets Generation
Salvatore Esposito, Matias Mattamala, Daniel Rebain, Francis Xiatian Zhang, Kev Dhaliwal, Mohsen Khadem, Subramanian Ramamoorthy
AI summary
Problem
Developing autonomous bronchoscopy algorithms is hindered by the scarcity of realistic training data due to ethical constraints, patient safety risks, and the high cost of clinical data collection.
Approach
The authors present ROOM, an automated pipeline that reconstructs patient airway geometries from CT scans, simulates physics-based continuum robot navigation, and renders synchronized multi-modal sensor streams with clinically accurate lighting and noise.
Key results
- Automated pipeline converting patient CT scans into photorealistic, multi-modal bronchoscopy datasets
- State-of-the-art pose and depth estimation methods struggle with bronchoscopy-specific challenges like specularities and low texture
- Fine-tuning depth models on ROOM synthetic data significantly improves performance on external clinical datasets
- Open-source release of the ROOM framework and generated datasets
Why it matters
Provides a scalable, safe, and cost-effective solution for training and evaluating autonomous bronchoscopy algorithms across diverse patient anatomies.
Abstract
Continuum robots are advancing bronchoscopy procedures by accessing complex lung airways and enabling targeted interventions. However, their development is limited by the lack of realistic training and test environments: Real data is difficult to collect due to ethical constraints and patient safety concerns, and developing autonomy algorithms requires realistic imaging and physical feedback. We present ROOM (Realistic Optical Observation in Medicine), a comprehensive simulation framework designed for generating photorealistic bronchoscopy training data. By leveraging patient CT scans, our pipeline renders multi-modal sensor data including RGB images with realistic noise and light specularities, metric depth maps, surface normals, optical flow and point clouds at medically relevant scales. We validate the data generated by ROOM in two canonical tasks for medical robotics: multi-view pose estimation and monocular depth estimation, demonstrating diverse challenges that state-of-the-art methods must overcome to transfer to these medical settings. Furthermore, we show that the data produced by ROOM can be used to fine-tune existing depth estimation models to overcome these challenges, also enabling other downstream applications such as navigation. We expect that ROOM will enable large-scale data generation across diverse patient anatomies and procedural scenarios that are challenging to capture in clinical settings. Code and data: https://iamsalvatore.io/room/.