Researchers Use People’s Brain Activity to Generate AI Images
A group of Japanese scientists used people’s brain activity to generate images on the artificially intelligent (AI) model Stable Diffusion.
Researchers at the Graduate School of Frontier Biosciences in Osaka University, Japan, were able to reconstruct high-resolution and highly accurate images from brain activity by using the popular Stable Diffusion model.
The team, which was led by researchers Yu Takagi and Shinji Nishimoto, presented their findings in a paper that was published in December.
The scientists presented individuals with a set of images and took fMRI (functional magnetic resonance imaging) scans of their brains while they concentrated on the image.
The team used a combination of the fMRI image output and Semantic Decoder to create the resulting image. However, they found that the addition of Stable Difusion to the process enabled the final generated images to look more similar to the original images that the participants were shown.
In the figures above, the images on the top red rows show the original pictures that were presented to the individuals in the study. Meanwhile, the images on the bottom gray rows are the images reconstructed from human brain activity using Stable Diffusion, Semantic Decoder, and fMRI image generation.
According to Vice, the team of researchers says that they first predicted a latent representation, which is a model of the image’s data, from fMRI signals.
Then, the model was processed and “noise” was added to it through the diffusion process. Finally, the researchers decoded text representations from fMRI signals within the higher visual cortex and used them as input to produce a final constructed image.
The study is the latest example of how scientists are attempting to discover how AI models can work with the human brain to recreate images.
Last year, PetaPixel reported on how researchers at Radboud University in the Netherlands were developing “mind-reading” technology that can translate a person’s brainwaves into photographic images.
The team of researchers showed photos of faces to two volunteers inside a powerful brain-reading functional magnetic resonance imaging (fMRI) scanner. As the volunteers looked at the images of faces, the fMRI scanned the activity of neurons in the areas of their brain responsible for vision.
The researchers then fed this information into a computer’s AI algorithm and the system was able to identically reconstruct the original images that the volunteers were shown.
Image credits: All photos sourced from the paper entitled “High-resolution image reconstruction with latent diffusion models from human brain activity” by Yu Takagi and Shini Nishimoto.
