Google Researchers Add Powerful Denoise Tool to NeRF AI Program
Google researchers have unveiled a new artificially intelligent (AI) program that has incredible denoising ability allowing users to “see in the dark.”
RawNeRF is able to combine images taken from many different camera viewpoints to jointly denoise and reconstruct the scene.
It works in tandem with Google’s Neural Radiance Fields,known as NeRF, a view synthesis that creates 3D renderings of a scene from multiple photos.
Ben Mildenhall, a research scientist, gives an example of a table lit only by candlelight that he took on his smartphone. Attempts to brighten the RAW file gave an awful amount of noise.
But, after Mildenhall ran it through RawNeRF the image came back without almost any noise and with far more detail in the scene.
“And of course, RawNeRF is not just a denoiser, which means we can vary the camera position to view the scene from different angles,” Mildenhall explains.
“RawNeRF goes beyond this, we actually reconstruct the scene and render new views in a linear hdr color space. This means we can also do things, like vary the exposure, by rescaling renderings in linear space before they’re post-processed.”
RawNeRF can swap out the basic gamma curve for a more complex tone mapping algorithm that brings out detail in shadows.
“We can even take advantage of the linear color values to render synthetic defocus with accurate bokeh effects,” Mildenhall says.
Moving on from the candlelit table scene, Mildenhall takes the technology to a photo of a dark road lit only by street lights.
Again, the RawNeRF performs remarkably well and this is because it was trained on RAW image data rather than post-process jpegs.
Mildenhall then shows further examples of badly lit, noisy images that are rescued incredibly well by the RawNeRF.
“Despite the noise, render is able to recover the fine geometry of this fence and bicycle,” he explains over one image.
The software is fed a collection of five to 200 photos of the same scene, which is easily shot in seconds using today’s camera technology.
From the data, the AI software can manipulate focus, exposure, and tone mapping after the fact.
“Although a single raw image appears significantly more noisy than a post-processed one, we show that NeRF is highly robust to the zero-mean distribution of raw noise,” writes the researchers.
“When optimized over many noisy raw inputs (25-200), NeRF produces a scene representation so accurate that its rendered novel views outperform dedicated single and multi-image deep raw denoisers run on the same wide baseline input images,” they add.
“As a result, our method, which we call RawNeRF, can reconstruct scenes from extremely noisy images captured in near-darkness.”
The team’s paper can be read here.
Image credits: All photos by Ben Mildenhall.
