Google’s New AI Mode Will Answer ‘Nuanced’ Questions About Your Photos

Google’s new AI mode can now answer complex questions about a user’s photos. The new image-analyzing functionality is powered by Google Lens’ multimodal capabilities, the company says.

This new multimodal understanding in AI mode allows users to take a photo or upload one, and ask questions about it to receive a “rich, comprehensive response with links to dive deeper.”

The function is powered by Google Lens and a custom version of its AI image program Gemini. Together, Google says this new AI Mode can “understand the entire scene in an image, including the context of how objects relate to one another and their unique materials, colors, shapes, and arrangements.”

The system uses a “fan-out technique”, that queries objects it sees in the scene so it can provide a response “that’s incredibly nuanced and contextually relevant.”

“AI Mode builds on our years of work on visual search and takes it a step further,” Robby Stein, VP of product for Google Search, tells The Verge. “With Gemini’s multimodal capabilities, AI Mode can understand the entire scene in an image, including the context of how objects relate to one another and their unique materials, colors, shapes, and arrangements.”

In the video above, Google gives the example of books on a shelf and the user wants to find similar recommendations that are highly rated. AI Mode provides that list as well as links where the users can learn more and purchase. The user can also ask it follow-up questions.

AI Mode was previously exclusive to Google One AI Premium subscribers but after the image-analyzing functionality started rolling out yesterday (Monday), millions more can now access it via the Google app on iOS and Android. However, you must be enrolled in Labs, Google’s experimental platform.

Google has been adding AI features to its search engine since 2024, but AI Mode is one step further. The search giant believes that AI Mode is key to maintaining its monopoly as the first port of call for people who want to search the internet.

Last month, PetaPixel reported on Google’s “Project Astra” which a step toward a personalized AI assistant.

Image credits: Header photo licensed via Depositphotos.