Novel view synthesis from a sparse set of input images is a challenging problem of great practical interest, especially when camera poses are absent or inaccurate. Direct optimization of camera poses and usage of estimated depths in neural radiance field algorithms usually do not produce good results because of the coupling between poses and depths, and inaccuracies in monocular depth estimation. In this paper, we leverage the recent 3D Gaussian splatting method to develop a novel construct-and-optimize method for sparse view synthesis without camera poses. Specifically, we construct a solution progressively by using monocular depth and projecting pixels back into the 3D world. During construction, we optimize the solution by detecting 2D correspondences between training views and the corresponding rendered images. We develop a unified differentiable pipeline for camera registration and adjustment of both camera poses and depths, followed by back-projection. We also introduce a novel notion of an expected surface in Gaussian splatting, which is critical to our optimization. These steps enable a coarse solution, which can then be low-pass filtered and refined using standard optimization methods. We demonstrate results on the Tanks and Temples and Static Hikes datasets with as few as three widely-spaced views, showing significantly better quality than competing methods, including those with approximate camera pose information. Moreover, our results improve with more views and outperform previous InstantNGP and Gaussian Splatting algorithms even when using half the dataset.

Overview of our method for sparse view synthesis. We first back-project the first view and sequentially register, adjust and back-project the remaining views in sequence to obtain a coarse solution. This coarse solution is then refined by standard optimization to reproduce fine details.

We assume the first k views have already been registered, and illustrate the registration, adjustment and back-projection of the k + 1 view. We first initialize the camera pose of the k + 1 view, denoted as P_k + 1, as the k view’s camera pose. 2D correspondences are detected between ground-truth image I_k + 1 and the rendered result I_render(P_k + 1) at P_k + 1. Correspondence points on I_render(P_k + 1) are denoted as κ′, while those on I_k + 1 are denoted as κ. Green points denote correct correspondences, while red points denote wrong correspondences. We can use perspective-n-points (PnP) to solve the camera pose but it results in an erroneous solution. We then apply our optimization pipeline to estimate the camera pose for registration. For now, the monocular depth D_k + 1 of the k + 1 view deviates significantly from the rendered depth D_render(P_k + 1) at P_k + 1. Afterwards, we apply our optimization pipeline to adjust all previous registered camera poses and monocular depths along with P_k + 1 and D_k + 1. It can be seen that I_render(P_k + 1) and D_k + 1 are much close to I_k + 1 and D_render(P_k + 1). Finally, we back-project pixels in the k + 1 view into world space as 3D Gaussians based on D_k + 1.

Citation

@article{COGS2024,
    title={A Construct-Optimize Approach to Sparse View Synthesis without Camera Pose},
    author={Jiang, Kaiwen and Fu, Yang and Varma T, Mukund and Belhe, Yash and Wang, Xiaolong and Su, Hao and Ramamoorthi, Ravi},
    journal={SIGGRAPH},
    year={2024}
}