ASSR-NeRF: Arbitrary-Scale Super-Resolution on Voxel Grid for High-Quality Radiance Fields Reconstruction

¹National Taiwan University, ²NVIDIA Research Taiwan

Abstract

NeRF-based methods reconstruct 3D scenes by building a radiance field with implicit or explicit representations. While NeRF-based methods can perform novel view synthesis (NVS) at arbitrary scale, the performance in high-resolution novel view synthesis (HRNVS) with low-resolution (LR) optimization often results in oversmoothing. On the other hand, single-image super-resolution (SR) aims to enhance LR images to HR counterparts but lacks multi-view consistency. To address these challenges, we propose Arbitrary-Scale Super-Resolution NeRF (ASSR-NeRF), a novel framework for super-resolution novel view synthesis (SRNVS). We propose an attention-based VoxelGridSR model to directly perform 3D super-resolution (SR) on the optimized volume. Our model is trained on diverse scenes to ensure generalizability. For unseen scenes trained with LR views, we then can directly apply our VoxelGridSR to further refine the volume and achieve multi-view consistent SR. We demonstrate quantitative and qualitatively that the proposed method achieves significant performance in SRNVS.

Overview of ASSR-NeRF

For any distilled feature field reconstructed from LR training views (grey part), VoxelGridSR module (orange part) performs self-attention on the volumetric representation and generate refined appearance feature and density for every sampled point. The refined features and densities are then aggregated to pixels, leading to novel view with rich and clean details.

Distilled Feature Field

We distill features from pre-trained feature extractor into volumetric representation so that VoxelGridSR module can utilize the queried SR priors and benefit self-attention. In a student-teacher setting, features extracted from training views are distilled into a 3D student network. The student network is trained by minimizing the difference between rendered features and features from pre-trained image feature extractor, in addition to rendered colors and ground-truth pixel colors. FeatureNet turn voxel feature into view-dependent distilled features, and a pre-trained decoder maps view-dependent features RGB color

Qualitative Results

Qualitative results on Synthetic-NeRF and BlendedMVS dataset: All other baselines are first reconstructed from LR training views, then perform HRNVS. For ASSR-NeRF, pre-trained VoxelGridSR model is applied to achieve SRNVS. The results show that ASSR-NeRF generates cleaner edges as well as richer details than other baselines.

Rich details in 4K

In this experiment, we compare other NeRF-based methods with ASSR-NeRF. While both ASSR-NeRF and other methods reconstruct a scene of training views in 1K resolution, ASSR-NeRF render novel views in 4K resolution with richer details.

BibTeX

@misc{huang2024assrnerfarbitraryscalesuperresolutionvoxel, title={ASSR-NeRF: Arbitrary-Scale Super-Resolution on Voxel Grid for High-Quality Radiance Fields Reconstruction}, author={Ding-Jiun Huang and Zi-Ting Chou and Yu-Chiang Frank Wang and Cheng Sun}, year={2024}, eprint={2406.20066}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2406.20066}, }