Nvidia researchers use artificial intelligence to transform 2D video clips into detailed 3D graphics

Neuralangelo, a new AI model from Nvidia Research, uses AI to capture two-dimensional video clips and transform them into detailed 3D graphic structures.

With this technology, the researchers were able to generate realistic virtual replicas of real-world buildings, sculptures and other objects.

Like Michelangelo sculpting breathtaking, lifelike visions from blocks of marble, Neuralangelo generates 3D structures with intricate detail and textures, Nvidia said. Creative professionals can then import these 3D objects into design applications, further modifying them for use in art, game development, robotics, and industrial digital twins.

Neuralangelo’s ability to translate the textures of complex materials including tile, sheet glass, and smooth marble from 2D video to 3D assets significantly surpasses previous methods. High fidelity makes its 3D reconstructions easier for developers and creative professionals to quickly create usable virtual objects for their projects using captured smartphone footage.

The 3D reconstruction capabilities offered by Neuralangelo will be a huge boon for creators, helping them recreate the real world in the digital world, Ming-Yu Liu, senior director of research and co-author of the paper, said in a statement. This tool will eventually allow developers to import detailed objects, be they small statues or huge buildings, into virtual environments for video games or industrial digital twins.

In a demo, Nvidia researchers showed how the model could recreate objects as iconic as Michelangelo’s David and as commonplace as a flatbed truck. Neuralangelo can also reconstruct the interiors and exteriors of buildings demonstrated with a detailed 3D model of the Nvidias Bay Area campus park.

The neural rendering model sees in 3D

Previous AI models for reconstructing 3D scenes have struggled to accurately capture repetitive texture patterns, smooth colors and strong color variations, Nvidia said. Neuralangelo adopts instant neural graphics primitives, the technology behind NVIDIA Instant NeRF, to help capture these finer details.

Using a 2D video of an object or scene shot from various angles, the model selects several frames that capture different points of view like an artist considering a subject from multiple angles to get a sense of depth, dimension and shape.

Once the camera position of each frame is determined, Neuralangelos AI creates a rough 3D representation of the scene, like a sculptor starting to carve out the shape of the subject.

The model then optimizes rendering to sharpen details, just as a sculptor painstakingly cuts stone to mimic the texture of fabric or a human figure. The end result is a large-scale 3D object or scene that can be used in applications of virtual reality, digital twins or robotic development.

Neuralangelo is one of nearly 30 Nvidia Research projects to be presented at the Conference on Computer Vision and Pattern Recognition (CVPR), taking place June 18-22 in Vancouver. The papers cover topics including pose estimation, 3D reconstruction, and video generation.

One such project, DiffCollage, is a diffusion method that creates large-scale content including a long horizontal orientation, a 360-degree panorama, and looping moving images. When fed a training dataset of standard aspect ratio images, DiffCollage treats these smaller images as sections of a larger visual as collage pieces. This allows syndication models to generate large, consistent-looking content without being trained on images of the same scale.