Apple Researchers Make AI that Can Read Between the Pixels
Sometimes, artificial intelligence feels so smart it’s scary. But when it misses details that should feel obvious, AI feels like it still has a ways to go. Apple’s researchers may have found a way to jump over one more hurdle.
Researchers detail the ability for large language models to essentially parse what’s on screen, referred to as reference resolution. The findings are explained in a paper, which reveals Apple’s goals to keep up with the competition and to continue upgrading its virtual voice assistant Siri.
The paper highlighted the use of this research to make assistants more conversational. By reading between the lines and better understanding the context of what’s on screen, assistants like Siri can respond in a way that feels more natural.
“Human speech typically contains ambiguous references such as ‘they’ or ‘that,’ whose meaning is obvious (to other humans) given the context,” researchers write. “Being able to understand context, including references like these, is essential for a conversational assistant that aims to allow a user to naturally communicate their requirements to an agent, or to have a conversation with it.”
Notably, the study made it clear how closely Apple is watching its competitors. The paper said it measured its models against GPT-3.5 and GPT-4. Further, its smallest model was on par with GPT-4 and its largest models outperformed it “substantially,” per Apple.
All of this lines up with Apple’s recent messaging regarding AI. At an earnings call, CEO Tim Cook coyly announced in February that AI features are expected to come “later this year.”
“Our M.O., if you will, has always been to to do work and then talk about work, and not to get out in front of ourselves,” Cook said at the time. “And so we’re going to hold that to this as well. But we have got some things that we’re incredibly excited about, that we’ll be talking about later this year.”
In fact, Apple has already started delivering on that promise. The announcement of its new M3-chip-powered MacBook Air focused heavily on AI.
This study likely marks what is just the start of Apple’s AI endeavors.
Image credits: Header photo licensed via Depositphotos.
