How Many Views Are Needed to Reconstruct an Unknown Object Using NeRF?

Neural Radiance Fields (NeRFs) are gaining sig- nificant interest for online active object reconstruction due to their exceptional memory efficiency and requirement for only posed RGB inputs. Previous NeRF-based view planning methods exhibit computational inefficiency since they rely on an iterative paradigm, consisting of (1) retraining the NeRF when new images arrive; and (2) planning a path to the next best view only. To address these limitations, we propose a non-iterative pipeline based on the Prediction of the Required number of Views (PRV). The key idea behind our approach is that the required number of views to reconstruct an object depends on its complexity. Therefore, we design a deep neural network, named PRVNet, to predict the required number of views, allowing us to tailor the data acquisition based on the object complexity and plan a globally shortest path. To train our PRVNet, we generate supervision labels using the ShapeNet dataset. Simulated experiments show that our PRV-based view planning method outperforms baselines, achieving good recon- struction quality while significantly reducing movement cost and planning time. We further justify the generalization ability of our approach in a real-world experiment. ⋆These authors contributed equally to this work. Sicong Pan and Maren Bennewitz are with the Humanoid Robots Lab, Liren Jin and Marija Popovi ́c are with the Institute of Geodesy and Geoinformation, University of Bonn, Germany. Maren Bennewitz is additionally with the Lamarr Institute for Machine Learning and Artificial Intelligence, Germany. Hao Hu is with Intel Asia-Pacific Research & Development Ltd. This work has partially been funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant 459376902 – AID4Crops and under Germany’s Excellence Strategy, EXC- 2070 – 390732324 – PhenoRob. Corresponding author: span@uni-bonn.de