NVIDIA’s AI-Powered ‘Content-Aware Fill’ is Mind-Blowing

NVIDIA is taking Photoshop-style “Content-Aware Fill” to the next level. The company has introduced a new state-of-the-art technique that uses AI to realistically reconstruct photos. This 2-minute video introduces the technology and contains examples that may blow your mind.

What’s amazing about NVIDIA’s system and the current “Content-Aware Fill” in Photoshop is that NVIDIA’s tool doesn’t just glean information from surrounding pixels to figure out what to fill holes with — it understands what the subject should look like.

For example, if you were to run Content-Aware Fill on a blank space where a subject’s eye should be, the result will be the space filled in with surrounding features — things like the skin, eyebrow, or nose. NVIDIA uses deep learning and knows that the space should be filled with an eye, so it adds a computer-generated eye.

For example, when giving this portrait photo with huge sections missing…

A portrait with many, many pixels missing.

This is what Photoshop’s Content-Aware Fill gives you:

Photoshop’s Content-Aware Fill result

And here’s what NVIDIA’s technique creates:

NVIDIA’s result

While it may not be perfect, it’s significantly more similar to what the original photo looked like than Photoshop’s reconstruction based on surrounding pixels.

The original photo

The feature can be used to both fill in holes in photos and remove existing things in a photo (just like Content-Aware Fill).

NVIDIA researchers have published a new paper detailing their creation, titled “Image Inpainting for Irregular Holes UsingPartial Convolutions.”

“Our model can robustly handle holes of any shape, size location, or distance from the image borders,” the researchers write. “Previous deep learning approaches have focused on rectangular regions located around the center of the image, and often rely on expensive post-processing. Further, our model gracefully handles holes of increasing size.”

The researchers trained their AI using 55,116 random masks of holes and streaks that were applied to huge collections of photos. By examining both the versions with missing areas and the original photos, the neural network learned how to reconstruct missing pixels from photos.

An additional 25,000 masks were used for the testing phase to validate accuracy.

Here are some additional before-and-after examples of heavily erased photos reconstructed by NVIDIA’s system:

“To the best of our knowledge, we are the first to demonstrate the efficacy of deep learning image inpainting models on irregularly shaped holes,” the researchers write.

No word yet on if or when we’ll see this technology released in a real-world software program.