Making Visual Search Smarter: How AI Understands Creative Intent
Most visual creations start with a search—for images, colors, fonts, and inspiration — but search has always felt disconnected from the creative process. It can be tedious and time-consuming to translate brilliant, imaginative ideas into words.
Search terms rarely convey the aesthetics and emotions at the heart of a creative idea – which can make image search become a mind-numbing task when it should be inspiring. However, artificial intelligence (AI) and machine learning technology can fundamentally change the nature of search and help make creative visions a reality.
What AI-Powered Search Can Do, and What It Means For Creativity
With deep learning, search algorithms can be trained to better understand images to recognize objects—like cars, cats, humans, or even the Eiffel Tower—as well as colors, composition, style, and mood. Then, it can be broken down so people can search for any of these aspects, or combine search terms and components from multiple images. The result is a search that doesn’t just find something similar to a keyword or image—it finds something similar to the exact elements that are most meaningful to the searcher. Ultimately, search captures nuance and creative intent.
Here’s an example of how it works. Imagine you’ve just done a keyword search for an image of a gazelle. You have a trove of interesting gazelle photos, but none of them is quite right.
To hone in on what you need, you choose an image that’s close. Then, you can use image similarity search to find gazelle photos with similar colors, content, and composition to your original. You find the perfect lone gazelle, looking to the right, with a warm color palette.
But what if gazelles aren’t the only piece of your puzzle? You might also need a German shepherd in the same pose and location, so you change your keyword, but keep the composition constraints, which quickly takes you to images of a German shepherd that look remarkably like your gazelle — your dog is front and center, looking right with warm colors.
As you can see, the combination of keywords with visual search for subtle aspects of an image allows the search engine to understand the nuances of your aesthetic and creative vision—far beyond what it could ever do with keywords alone.
Now, imagine that you want an image with a particular kind of object. Maybe you have a photo of a tent in the woods, but you want the tent further to the left, and out of the woods. You can click the tent, an object that machine learning technology can recognize, slide it where you need it, and search images with a tent in exactly the spot and scenery you desire. Eventually, you will even be able to engage several images in your search, choosing an object from one, the colors from another, and the composition from a third.
Looking ahead: How Search Will Create the Things We Imagine
The journey with AI and visual search is just beginning. As the tools become smarter, search and creation will come together in a complimentary way. Eventually, AI won’t just help us find things, it will generate what we’re actually seeking. The evolution of smarter search is like a dance—humans are
creating the data from which AI learns. Smarter tools, combined with the indelible spark of human creativity, will usher in new avenues for creative expression that we haven’t previously imagined.
For example, imagine you’re searching for an image of a person with an umbrella on a sunny day, but all the images you find are rainy days. Search will be able to apply machine learning to blend assets and create an image that never existed before—an image that’s exactly what you had in your imagination. Think of layering Photoshop-like magic to your search query and applying that to find the image you envisioned.
For creatives, visual search harnessing the power of AI means cutting out a lot of the grunt work and the frustrating, clumsy keywords. By doing this, the search becomes more deeply integrated into the creative side of the work, and people have more time free to devote to the innately human side of creativity—developing and exploring new ideas.
About the author: Scott Prevost is Vice President of Engineering for Adobe Sensei, where he oversees development of the company’s AI and machine learning (ML) technology powering the design and delivery of digital experiences. He has a long history of delivering AI/ML features and solutions for both consumer and enterprise products – from web-scale search to computer vision, and was a pioneer in the field of intelligent agents. Prior to Adobe, he was vice president of product management at eBay, responsible for the global search and buying experience. Scott began working in search at Powerset, a startup specializing in semantic search, which was acquired by Microsoft in 2008. Before eBay, Scott spent several years at Microsoft managing the integration of Powerset into Bing.
Image credits: Header image licensed via Deposit Photos.
