Sonar-MASt3R: Real-Time Opti-Acoustic Fusion in Turbid, Unstructured Environments
Amy Phung, Richard Camilli
AI summary
Problem
Optical cameras fail in turbid underwater conditions, while existing opti-acoustic fusion methods cannot produce dense 3D reconstructions in real-time or handle high turbidity.
Approach
The method uses MASt3R to extract dense optical features in real-time and continuously rescales them using geometric cues from a wrist-mounted imaging sonar, switching to acoustic-only mode when visibility drops.
Key results
- Real-time dense metric-scale 3D reconstruction
- Adaptive optical-acoustic switching in high turbidity
- Experimental validation across <0.5 to >12 NTU turbidity
- New opti-acoustic dataset with calibrated turbidity levels
Why it matters
Provides a reliable perception solution for underwater intervention, inspection, and manipulation tasks where visibility is frequently compromised.
Abstract
Underwater intervention is an important capa- bility in several marine domains, with numerous industrial, scientific, and defense applications. However, existing percep- tion systems used during intervention operations rely on data from optical cameras, which limits capabilities in poor visibility or lighting conditions. Prior work has examined opti-acoustic fusion methods, which use sonar data to resolve the depth ambiguity of the camera data while using camera data to resolve the elevation angle ambiguity of the sonar data. However, existing methods cannot achieve dense 3D reconstructions in real-time, and few studies have reported results from applying these methods in a turbid environment. In this work, we propose the opti-acoustic fusion method Sonar-MASt3R, which uses MASt3R to extract dense correspondences from optical camera data in real-time and pairs it with geometric cues from an acoustic 3D reconstruction to ensure robustness in turbid conditions. Experimental results using data recorded from an “opti-acoustic eye-in-hand” configuration across turbidity values ranging from <0.5 to >12 NTU highlight this method’s improved robustness to turbidity relative to baseline methods.