Researchers from MPI Informatik and University College London trained a neural network that changes the angle of view of a scene in an image. The neural network takes into account the change in lighting. To interpolate the viewing angle, time and lighting from a set of 2D images, which have coordinates marked, the neural network learns to draw each image from the others.
On inference, the model takes as input a set of coordinates that describe the parameters of the viewing angle, time and lighting. At the output, the model generates a 2D image in real time with the specified coordinates. At the same time, additional parameters can be added to the model, which will be taken into account when generating the image.
X-Fields — Video Demonstration
Approach and Benchmarking
The researchers propose to represent a set of 2D images of one scene using a neural network to correlate the parameters of view, lighting and time with images. The model can estimate for each pixel in the image how it will change when viewing angle, time, or lighting changes. Technically, this is the Jacobi matrix for pixels in relation to viewing angle, time and lighting.
This X-Field is trained for one scene per minutes. This allows you to use real-time views to change the viewing angle, time, and lighting parameters in the input image. Convolutional Neural Network (CNN) was used as the network architecture.
If you found this article helpful, click the💚 or 👏 button below or share the article on Facebook so your friends can benefit from it too.