Abstract

Novel view synthesis using implicit neural functions such as Neural Radiance Field (NeRF) has achieved significant progress recently. However, it is very computationally expensive to train a NeRF due to the disordered frequency optimization. In this paper, we propose the Pyramid NeRF, which guides the NeRF training in a 'low-frequency first, high-frequency second' style using the image pyramids and could improve the training and inference speed at 15× and 805×, respectively. The high training efficiency is guaranteed by (i) organized frequency-guided optimization could improve the convergency speed and efficiently reduce the training iterations and (ii) progressive subdivision, which replaces a single large multi-layer perceptron (MLP) with thousands of tiny MLPs, could significantly decrease the execution time of running MLPs. Experiments on various synthetic and real scenes verify the high efficiency of the Pyramid NeRF. Meanwhile, the structure and perceptual similarities could be better recovered.

Pyramid NeRF

The proposed Pyramid NeRF comprises two parts, i.e., the organized frequency-guided optimization and the progressive subdivision. The former focuses on improving the convergency speed by guiding the training following the rule of 'low-frequency first, high-frequency second'. At the same time, the latter could decrease the running time of the network by reducing the MLP size.

Here, we show how radiance field change as training processes.

Our organized frequency-guided optimization solves the problem of disordered optimization and organizes the NeRF training following the orders of 'low-frequency first, high-frequency second'. Each input image is represented as a multi-scale image pyramid, and the INR fitting is conducted from the largest scale to the lowest one. Apart from this, the progressive subdivision is proposed to to reduce the execution time of running networks on each scale.

We can see that when training processes, the higher frequency features would be added to the radiance field. Difference of Gaussian contents between neighboring scales are shown in the second line of video.

Results

We train NeRF, Torch-ngp, KiloNeRF and Pyramid NeRF on datasets Synthetic NeRF, Synthetic NSVF and BlendedMVS. Because the organized frequency-guided optimization is adopted, the proposed Pyramid NeRF could provide better structure similarity and perceptual similarity.

Qualitative comparison on synthetic scenes. Considering the proposed frequency-guided optimization, our Pyramid NeRF is outperformed compared to state-of-the-art methods on the detail textures and occlusion boundaries.

Qualitative comparison on real scenes.

Citation

Acknowledgements

The website template was borrowed from Michaël Gharbi.