Neural Lens Modeling

CVPR 2023

Wenqi Xian Aljaž Božič Noah Snavely Christoph Lassner
Cornell Tech Meta Reality Labs Cornell Tech Meta Reality Labs
StayPositive Method Intro
paper-snapshot
[Paper] [Code]

Abstract

Recent methods for 3D reconstruction and rendering in creasingly benefit from end-to-end optimization of the entire image formation process. However, this approach is currently limited: effects of the optical hardware stack and in particular lenses are not being modeled in a differentiable way. This limits the quality that can be achieved for camera calibration as well as the fidelity of results of 3D reconstruction. In this paper, we propose a neural lens model for distortion and vignetting that can be used for point projection and raycasting and can be optimized through both operations. This means that it can (optionally) be used to perform pre-capture calibration using classical calibration targets, and can later be used to perform calibration or refinement during 3D reconstruction; for instance, while optimizing a radiance field. To evaluate the performance of the proposed model, we propose a comprehensive dataset assembled from the Lensfun database with a multitude of lenses. Using this and other real-world datasets, we show that the quality of our proposed lens model outperforms standard packages as well as recent approaches while being much easier to use and extend. The model generalizes across many lens types and is trivial to integrate into existing 3D reconstruction and rendering systems.

StayPositive Framework
Our key insight is to use an invertible neural network (INN) to model ray distortion, combined with standard camera intrinsics and extrinsics. This means that we model the camera lens as a mapping of two vector fields using a diffeomorphism (i.e., a bijective mapping where both the mapping and its inverse are differentiable), represented by an INN.


We also proposed: A new marker that can be jointly optimzed with a keypoint detector to increase the robustness of pattern-based calibration;
A large scale camera lens benchmark for evaluating the performance of marker detection and camera calibration.

Result Visualizations

Our model can also be optimized using point correspondences for keypoint pairs obtained using calibration targets, as well as gradients from a set of 2D image observations. Here we show a timelapse of the optimization process. Blue: ground truth keypoint positions, orange: predicted keypoint positions. For lens distortion visualization, Hue: offset direction, saturation: offset magnitude.
We integrate our neural lens model into a neural rendering framework such that the camera poses, pinhole intrinsics and lens dis- tortion are optimized together with the appearance model, given only RGB observations.
flow-airplane-4

Acknowledgements


Part of the work done during an internship at Meta Reality Labs. We thank Guandao Yang and Zhaoyang Lv for help and discussion.

Citation

@InProceedings{Xian_2023_CVPR, author = {Xian, Wenqi and Bo{\v{z}}i{\v{c}}, Alja{\v{z}} and Snavely, Noah and Lassner, Christoph}, title = {Neural Lens Modeling}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, year = {2023}, }