Researchers are Working on Sketch-Based Image Search

In the not-so-distant future, finding the photo you’re looking for online may not require you to describe it or know the location or even the photographer… just as long as you can doodle something that looks like it on a scrap of paper.

Computer scientists at the Georgia Institute of Technology in Atlanta have developed a program that can do just that: take a drawing and find a photo that looks just like it. “For some types of images you want to find, it would be very hard to express that thing with just language,” GIT computer scientist James Hays told New Scientist. “What if you could just draw what you want?”

The program was developed by putting 664 people to work making sketches. Each person was shown one of several thousand photos for a total of two seconds, then tasked with sketching that photo from memory. Those sketches were then fed into a program that used two neural networks—one to analyze the sketches and the other the photos, trying to match the two up.

37% of the time, the AI got it exactly right: matching the drawing to the exact photo it was supposed to be of. And while that might not seem all that great, consider that human subjects could only match the two up 54% of the time. “Computationally,” says Hays, “we might be able to beat the human baseline.”

As for how this tech could be used once perfected, some commercial and even police-related ideas have already being thrown around… but we’re curious: how would you as a photographer use it? To find beautifully composed images? To identify visual ideas that are rare or haven’t been tried before?

We can think of a few more ideas, but let us know your thoughts in the comments.

(via Digital Trends)