Fastest-Ever ‘Mind Reading’ AI Model Can Reconstruct Images From Your Brain

Scientists have developed an AI model that can “read” a person’s brain activity and reconstruct what they’re looking at with striking speed and accuracy. The astonishing results can be seen above — with the original image that the person sees on the left and the AI’s reconstruction on the right.

The AI model, which is called Brain-IT, was developed in the lab of Prof. Michal Irani at the Weizmann Institute of Science. Brain-IT has been trained to recognize similar patterns of brain activity that occur in different people when they look at images. Unlike earlier AI models, it can then recreate those images with a high level of accuracy, completing the process in about an hour instead of the several hours previously required.

“There exist nowadays models that translate brain activity into images, and they can even produce impressive reconstructions that preserve the semantic meaning of the image reasonably well,” Irani says in a statement. “However, they tend to make mistakes in basic features such as composition and color.”

Irani explains: “The new model we developed outperforms them in reconstructing both the content of the image and its details. What’s more, while every other model requires dozens of hours of brain scans to learn to ‘read’ a new person, our model needs only one hour.”

Researchers trained the system using more than 70,000 images shown to eight people while their brains were scanned. Because eight people is a relatively small dataset for this kind of research, they developed an “encoder” that could identify patterns shared across different people’s brains and predict brain activity from an image.

To build the encoder, the researchers divided each brain scan into about 40,000 tiny sections, called voxels, and measured how each responded to different images. They also used AI to identify visual features in those images, such as colors and locations, as well as what the images depicted, such as faces or food.

By comparing these features with brain activity across the eight participants, they were able to identify patterns that appeared across different brains, despite individual brains responding differently. The researchers found that these shared patterns could be grouped into 128 functional regions involved in processing images.

“During training, the encoder naturally identified 128 functional regions that are shared by all people and perform specific roles in image processing,” Irani explains. “Some of them are familiar to neuroscientists, but others are entirely new. For example, we discovered a division of roles within the brain region that processes images of places – the PPA – with one part responding to indoor scenes and another to outdoor scenes.

According to a report by MIT Technology Review, the team then used the encoder alongside a “decoder” to see whether these patterns could be used to reconstruct images from brain activity. The encoder predicts what a person’s brain activity should look like when they view a particular image, while the decoder works in the opposite direction, using brain activity to reconstruct the image. By repeatedly training the two models together, the researchers were able to make the reconstructions increasingly accurate.

The researchers say the technology behind Brain-IT could eventually help people with paralysis communicate and give scientists a more efficient way to study how the brain processes images.