With a Convolutional Neural Network, GTA V Comes Alive

The new convolutional neural network developed by Intel Labs employees is able to do fantastic things. Vladlen Koltun, Stephan Richter, and Hassan Abu AlHaija published the work where they explained the principles of the new method used to make synthesized images look more realistic.

To illustrate how it works, the developers took footage from the famous GTA V – a fantastically drawn game that doesn’t, though, even pretend to be photorealistic. But so it becomes when processed with the new convolutional network. The team took high-resolution pictures of urban landscapes (mostly the streets of German cities) and used them as material for the neural network to learn what makes an image look realistic. Given that, the resulting images are consistent with the originals in both geometrical and semantical senses.

The methods described look sophisticated. In short, first, the network extracts the geometrical information (shape of objects, distance to them, lighting, and so on) from the artificial images. Then objects are separated, processed, and reimaged using the network, which in the meanwhile learns from the real photos how these objects should look in real life. Then it picks out visual properties of real objects and applies them to already recognized artificial ones. The result is really astounding, even next to color transfer and other methods of processing images.

The principles of convolutional networks were formulated in the late 1980s. It is purposely developed to analyze images and process them. But hardware available in the 1980s was incomparable to what we have today. The results can be seen on the video the team published along with the article. If you are a GTA V fan, you have probably seen visual enhancement mods, but what these three developers are showing is incomparably more impressive.