Can JPEGs Be Improved to RAW Quality?
Over the years, I have come across this question often: “Can a JPEG image be enhanced to a RAW image quality?” I understand where this question comes from. Photographers are taught to “shoot in RAW” from day one. These days, so many images are only in JPEG format, from phone photos to stock photos, to those priceless old pictures. It would be really nice if we could enhance them to RAW image quality so they could fit into today’s photographer’s RAW image editing workflow!
Unfortunately, the answer has always been “No.” JPEG format was introduced in 1994 mainly to reduce file size and conserve memory space. Images saved as JPEG are a lot smaller in file size and have also lost a lot of information. From the information theory of point of view, that lost data is unrecoverable and the damage is irreversible. There are changes you can make to the image to make it look a little better, as we did with our Photoshop plugin “Topaz DeJPEG”, which used an algorithm to remove blocky artifacts in near “optimal” sense (optimal Mean Squared Error). However, even DeJPEG simply cannot recover the lost details.
However, I recently became hopeful that my “No” could eventually become a “Yes.” Before I explain further, let’s dig a little deeper into the difference between a JPEG and a RAW image.
RAW images emerged when high-quality DLSR digital cameras started to replace film cameras. A RAW image usually comes directly from a DLSR camera by saving imaging sensor data with minimum alteration so that maximum image quality can be achieved during the post editing. This is possible because DSLR cameras use the best image sensors and high-quality optics, resulting in RAW images that are high quality.
Optics/Lens → Imaging Sensor → 10-14bit per RGB → save to RAW file in 10-14 bits per
JPEG images typically come from small cameras, like your phone, or are saved after editing to reduce image size for the internet. Phone or low-cost point-and-shoot cameras have very small image sensors (the red one in figure 2 is an iPhone imager size, as opposed to DLSR full frame), which will produce much higher noise. In-camera pre-processing has to be used to suppress the noise to make the image looks good. This noise suppression also removes fine details in the images. Then the images are converted to 8-bit sRGB colorspace to further reduce the amount of data in the image. This is a shame because sRGB can only represent about 50% of the colors that your eye can see, as illustrated in Figure 3. Finally, the JPEG encoder compresses the image into an even smaller file size which results in more loss of information.
lens/sensor → pre-processing → (8bit sRGB) → JPEG encoder → JPEG file
To summarize the difference between JPEG and RAW:
Don’t get me wrong. When a JPEG image is saved with “Quality > 70”, it “looks” almost as good as a RAW image. The problems only manifest when you edit the image in a strong way.
With the information losses in JPEG’s encoding and the fact that jpeg cameras tend to have small lenses and sensors, it is easy to conclude that it is impossible to enhance JPEG images into raw quality… well, until recently, that is.
In the last few years, there has been some revolutionary development in Artificial Intelligence and its application in image enhancement. Using “deep learning”, artificial neural networks have achieved some amazing results (see our AI Gigapixel and AI Clear). With these developments, it was natural for us to try to use A.I. to enhance a JPEG image to (hopefully) RAW image quality!
But wait, I just said that it is theoretically impossible to recover the lost information. How can an artificial neural network do the impossible?
The easiest way to explain it is that A.I. approaches this problem from a very different angle. Instead of trying to reverse the information loss, which is impossible, we train a neural network to “remember” what the RAW image should look like before it’s converted to a JPEG image. We train the neural network on a large number of high-quality RAW images together with their corresponding JPEG version so that it learns and remembers the correspondence. After training the network on enough images, the neural network will “remember” to output a higher quality image even when given a new “unseen” JPEG image as input. It’s just like when you see a friend from far away. Though they are too far away for you to make out the details with your eyes, you can imagine or remember what your friend looks like in vivid detail.
To make this approach work, we faced the following challenges:
RAW image dataset for training. We needed raw images of all subjects and types – lots of them. We searched the internet, we begged people to allow our machine to see their RAW images, we dug out all of our vacation photos, we shot more. Eventually, we built the most comprehensive RAW image dataset that I know of in deep learning fields.
A proper neural network architecture and the method to train it. For this, we took advantage of our previous research in image upsampling (Topaz A.I. Gigapixel) and noise reduction (Topaz A.I. Clear) and used our proven neural network architecture. We went above and beyond though and made it even deeper – over 30 layers of neurons. We also invested in large GPU servers and ran them for months at a time to let the network slowly learn from our large RAW image dataset.
Run the deep neural network fast enough on a PC. We continued to refine our own proprietary GPU based neural network inference engine to allow it to run acceptably on at least mid-end personal computers.
After months of training and a lot of late nights for our team, we are proud to introduce our new software “Topaz JPEG to RAW AI.” Look back at Figure 1: the right image is a high-quality RAW image, the left image is JPEG compressed with “quality=15”, the middle one is the JPEG after being enhanced by “Topaz JPEG to RAW AI”. This is an extreme case and the output is not quite “RAW” image quality, but the A.I. does a pretty impressive job given the JPEG is extremely compressed and has lost almost all details.
Overall, my scorecard for our current attempt to use AI to enhance JPEG is summarized below:
Here are a few more images to compare.
Remove compression artifacts:
Recover details and sharpen:
Remove noise:
Are the enhanced JPEGs now RAW quality? That’s for you to judge. We will continue to work on improving it. We encourage you to try JPEG to RAW AI on your own images by downloading the free trial. Let us know what you think – we’re always open to feedback.
At Topaz Labs, we are so excited about the possibilities of A.I. for photo processing. We are just starting to scratch the surface of A.I.’s potential for photography. In due time, we believe this technology will bring many great tools to assist artists and photographers in their creative pursuit.
Disclaimer: New JPEG standard does allow you to save the image in higher quality, e.g. encoded in 16-bit. However, my discussion above applies to the majority (I suspect > 95%) of the existing JPEG images. Example images used are pretty extreme cases to illustrate the points. Most JPEG images you have probably are not that bad! :-)
About the author: Dr. Yang founded Topaz Labs 12 years ago due to his life-long passion for signal and image enhancement. His knowledge and skills have been instrumental in leading Topaz Labs to be an industry frontrunner in using new technology in their photography products – from innovative masking (Topaz ReMask), best quality photo noise reduction (Topaz Denoise), to Artificial Intelligence-based products, such as A.I. Gigapixel and A.I. Clear. Dr. Yang is also a long-time entrepreneur, co-founder and CTO of Techwell Inc. (video enhancement semiconductors, went IPO in 2006) and the co-founder and CTO of Fortemedia Inc (small microphone array technology for voice enhancement). Dr. Yang received his Ph.D in System Design Engineering, University of Waterloo, Canada.
