New Canon AI Image Processor Fixes Most Major Digital Photo Issues

Canon Japan

Canon has showcased a neural network, deep learning-powered image processing technology that can dramatically improve image quality in three areas: noise, color, and lens imperfections such as aberrations or blur.

Using what it describes as “cutting-edge technology,” the company says that it is able to take images that suffer from a range of issues and correct them thanks to a deep learning algorithm that has been trained on a massive database of images that it has accumulated through the development of cameras and lenses. Canon says this database covers “all imaginable subjects” and the algorithm has access to both RAW files (which have a significant amount of information) as well as JPEGs.

Canon says it combined this large data set with its “unique know-how” to train the software to understand not only how camera settings affect image quality, but also how to correct them.

“A camera captures wonderful moments, such as a spectacular view that you have never seen before, or a moving moment that brings back vivid memories when you look back at it later… However, there were some unavoidable problems with the image quality of the photos. For example, noise that makes a photo look grainy, moiré that makes a pattern that should not exist, and blurring of an image caused by a lens,” Canon says.

“Canon is committed to the unavoidable ‘left behind in the camera’ principle of photography. In order to solve this problem, IT directly tackled the development of deep learning image processing technology himself.

Deep Learning with Three Main Targets

Canon’s deep-learning image processing technology — described as a neural network, which is a type of artificial intelligence (AI) — covers three areas: noise reduction, color interpolation, and aberration diffraction correction.

Noise reduction or removal technology isn’t new, but Canon says that when noise is removed, typically details are lost in exchange and — in most cases — the image quality deteriorates. Using deep learning, Canon says that it was able to obtain clear, high-definition results with little deterioration after removing noise.

Canon Japan
Noise Reduction showing skin tone detail reproduction. On the left, original photo. On the right, Canon’s deep learning result.

“We have established a Neural Network Noise Reduction function that enables clear, high-quality images to be obtained,” Canon explains.

“In high-sensitivity photography, it is now possible to amplify light information while simultaneously removing amplified noise and expressing smooth skin texture (skin tone), which was hindered by the occurrence of noise.”

Canon Japan

Canon says that capturing images using current sensor technology has fundamental problems.

“In a digital camera, each pixel in the image sensor is arranged in a regular pattern, so when you shoot stripes or checkered patterns, the regular patterns are superimposed, creating moire patterns that should not exist,” the company says.

“There was a fundamental problem with doing so. In addition, each pixel of the image sensor detects only one of the three primary colors of light, red (R), green (G), and blue (B), and refers to information from surrounding pixels for the remaining two colors. RGB data for one pixel is generated by estimating the color of the pixel (this is called color interpolation processing).”

Because of this, Canon says it was hard or impossible to stop false colors or color moire from occurring, but there were processes that could suppress it at the cost of image degradation — resolution and color reproducibility would deteriorate. But thanks to the deep learning technology it developed, Canon says that it has a new processing capability for color interpolation called Neural Network Demosiac.

“Erroneous interpolation is suppressed by focusing on subjects that are difficult to guess in color interpolation processing,” Canon says.

Canon Japan
Color interpolation is designed to improve color reproduction. On the left is the original image. On the right is the corrected image.

“False colors on striped shirts, jaggies that often stand out in oblique lines, and moire and false colors that stand out in pet photos can be accurately interpolated to improve resolution and color reproducibility.”

Canon Japan
Color interpolation is designed to reduce color moire. On the left is the original image. On the right is the corrected image.

Canon says that its deep learning technology can “strongly suppress” false colors that occur when shooting striped shirts, for example.

Finally, Canon’s neural network deep learning is able to correct for problems that occur due to issues with lenses. Even with high-quality optics that use multiple concave and convex lenses, aspherical lenses, and special materials, Canon says that it is impossible to correct for all aberrations completely. To make up for whatever can’t be corrected, its Neural Network lens Optimizer is able to correct these image degradations and can greatly improve how much resolution an image appears to have.

“Using simulation technology based on the design values ​​for each lens, we created a large number of high-precision pairs of student images with aberrations and diffraction blur and teacher images without them,” Canon explains.

Canon Japan
Aberration diffraction correction improves perceived resolution. On the left is the original image. On the right is the corrected image.

“Using these as training data for deep learning, we succeeded in correcting various blurring that occurs in details when we want to clearly depict the periphery of an image, such as when shooting landscapes or astronomical objects. In addition, it is now possible to correct only the bokeh without increasing the noise that was amplified when correcting the bokeh.”

Aberration diffraction correction can compensate for subjects that have overexposed areas or that blur at the extremes of a photo. On the left is the original image. On the right is the corrected image.
Aberration diffraction correction can also remove color fringing. On the left is the original image. On the right is the corrected image.

This technology can also fix blurring that can occur at edges of a frame, especially with photos of the night sky. While chromatic aberration is one of the easiest problems to fix, Canon’s neural network also addresses this problem.

“By combining the three areas of image correction realized by deep learning image processing technology, it is possible to perform a higher level of correction than by performing them individually, achieving a significant improvement in image quality such as the depiction of details and the three-dimensional effect of photographs,” Canon says.

Canon doesn’t clarify if this technology will have to be applied to images after they have already been shot on a computer or if it has plans to build this technology into its cameras. It also doesn’t specify when photographers might have access to the technology. Canon’s full explanation of its deep learning technology can be found on its website, though the entire piece is in Japanese and, at the time of publication, not available in any other language.

Image credits: Canon