DepthMesh: A Dual-End Complementary Online Depth Estimation and Mesh Reconstruction
Jiaqi Yang, Dazhao Fan, Xingbin Yang, Jiabin Yang, Song Ji, Yang Dong, Ming Li, Aosheng Wang
AI summary
Problem
Existing online 3D reconstruction methods struggle to balance computational efficiency and depth/mesh accuracy on mobile devices, often sacrificing one for the other or relying on complex, slow deep learning models.
Approach
The method uses incremental TSDF raycasting to generate depth and normal priors, which guide a planar-prior-constrained matching cost calculation and segmentation-aware cost aggregation to optimize depth estimation and mesh reconstruction simultaneously.
Key results
- Achieves <3 cm depth accuracy on mobile devices
- Enables real-time online mesh reconstruction via coupled TSDF fusion
- Reduces computational overhead through planar-prior-constrained cost aggregation
- Outperforms SimpleRecon on ScanNetV2 depth benchmarks (0.0784 m abs error, 78.11% δ<1.05)
Why it matters
Enables efficient, high-fidelity 3D reconstruction for resource-constrained applications like robotic navigation and mixed reality.
Abstract
We present a novel dual-end complementary method for online depth estimation and mesh reconstruction, termed DepthMesh. Unlike most existing state-of-the-art methods that produce either only depth online or surface mesh offline, our method tightly couples online multiview depth estimation and Truncated Signed Distance Function (TSDF) reconstruction to achieve fast online mesh reconstruction. For each keyframe from 6DoF tracking, we first obtain the prior depth and normal maps via ultra-fast raycasting from TSDF, which is incrementally fused from historical keyframe depths. Then, these priors, combined with segmentation results, are used to generate local planar hypotheses that optimize both depth accuracy and computational efficiency. Finally, the optimized depth estimates further enhance the accuracy of mesh reconstruction. Through this dual-end complementary mechanism, our system achieves high accuracy and efficiency. Experiments with qualitative and quantitative evaluations on the ScanNetV2 and self-collected datasets demonstrate the effectiveness of our method. Our method can generate depth and mesh online with accuracy (< 3 cm) on mobile devices, which is useful for robotic autonomous navigation and mixed reality applications such as real-time occlusion and collision handling.