GauGAN VR

An in-house prototype

The task of the project:
To design a neural renderer

Specific of the project:
We used neural network as a final element of rendering in graphic pipeline in Unity

What we did and how we did it:
A Unity scene, consisting of objects, is rendered with the help of a common graphic pipeline. At the end of the rendering process we get masks, designed specially for each of the objects. A mask exists as an image. The mask is sent to the stage of neural network processing, where the final image is generated. The generated image is the result of the final render, which appears on the screen.
The neural network operates on a server. Unity serves as a client application, allowing us to use even the less fast computers for work. This is especially important for standalone VR equipment. From the VR perspective, we used HTC Vive and StreamVr, but you may use any other PC VR device.

Project team
Alexander Kryuchkov, Apollinariya Orlova, Evgeny Chomutov

Time-frame: 5 days

Year of the project implementation: 2019