NVIDIA’s Neuralangelo AI Turns iPhone Video into Detailed 3D Structures
Neuralangelo is the latest artificial intelligence (AI) from NVIDIA Research that is capable of turning simple 2D iPhone video clips into detailed 3D structures, fully replicating buildings, sculptures, or other real world objects or spaces virtually.
The name of the platform is a nod to Michelangelo’s ability to sculpt life-like renditions out of blocks of marble. Likewise, NVIDIA Research says that Neuralangelo generates 3D structures with intricate details and textures from simple video clips.
These 3D objects and spaces can then be imported into design applications to be edited for the purposes of art, video game development, robotics, and the creation of digital twins.
The concept behind Neuralangelo might sound familiar because it is very similar to other projects that have come out of NVIDIA Research. For example, last year NVIDIA researchers created a new tool that easily turned photos into 3D objects. Called NVIDIA 3D MoMa, the company said it can allow architects, designers, concept artists, and game developers to quickly import objects into a graphics engine for digital manipulation.
While similar, Neuralangelo builds on that concept to allow for far larger and more detailed spaces and objects to be imported. What makes Neuralangelo special is that it can accurately capture repetitive texture patterns, homogenous colors, and strong color variations — areas where prior AI models have struggled.
NVIDIA says that Neuralangelo adopts instant neural graphics primitives — the technology behind NVIDIA Instant NeRF — to help capture these finer details.
The tool is powerful enough that drone video of a large space can be converted into a massive 3D object.
The company explains that by using a video of an object or a scene filmed from various angles, Neuralangelo selects several of the frames that capture different viewpoints. It then determins the camera position in each frame and then creates a rough 3D representation of the scene. The model then then optimizes the render to sharpen the details before producing the final 3D object.
“The 3D reconstruction capabilities Neuralangelo offers will be a huge benefit to creators, helping them recreate the real world in the digital world,” Ming-Yu Liu, senior director of research and co-author on the paper describing the research, says.
“This tool will eventually enable developers to import detailed objects — whether small statues or massive buildings — into virtual environments for video games or industrial digital twins.”
NVIDIA says that Neuralangelo’s ability to translate the textures of complex materials — which can include objects like roof shingles, panes of glass, and smooth marble — from 2D videos to 3D assets significantly surpasses prior methods.
“The high fidelity makes its 3D reconstructions easier for developers and creative professionals to rapidly create usable virtual objects for their projects using footage captured by smartphones,” the company says.
Image credits: NVIDIA
