Siri is going to get smarter, it may soon even understand how iPhone apps work

Siri is going to get smarter, it may soon even understand how iPhone apps work

If you are a 90s kid, chances are that the first ever AI (artificial intelligence) tool you encountered was Siri. The AI-powered voice assistant was unveiled as part of iPhone 4S’ features in 2011. Be it by helping us answer a call or setting up the alarm, Siri made lives easier and was quite fun to interact with. However, in the last few years, we haven’t really seen any major announcements with regards to Siri. Now that AI is in the limelight ever since OpenAI’s chatbot ChatGPT’s launch, it is being reported that Siri might also get smarter in the future.Â

Reports of Apple working on generative AI features for Siri have been doing rounds for quite some time. Now, a research paper published by Cornell University talks about a new MLLM (Multimodal Large Language Model) that might understand how a phone’s interface works. The paper, titled Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs, explains how the technology has come a long way but still has shortcomings when it comes to interacting with user interface of screens.

However, Ferret UI (which was launched in October last year) is an MLLM being developed to understand UI screens and potentially understanding how apps in a phone work. The MLLM, as per the paper, is also like to have “referring, grounding, and reasoning capabilities.”

One of the primary challenges in enhancing AI’s understanding of app screens lies in the diverse aspect ratios and compact visual elements found in smartphone displays. Ferret-UI tackles this hurdle by magnifying details and leveraging enhanced visual features to understand even the smallest icons and buttons. The paper also mentions that through meticulous training, Ferret-UI has surpassed existing models in its ability to understand and interact with app interfaces. If Ferret-UI gets incorporated into Apple’s voice assistant Siri, we can expect it to make the tool smarter.Â

The digital assistant could execute complex tasks within apps in future. Imagine instructing Siri to book a flight or make a reservation, and seamlessly, Siri interacts with the corresponding app to fulfil the request.

Talking about Ferret, it is an open-source, multimodal large language model that was released between Apple and Cornell University, resulting from extensive research on how large language models could recognise and understand elements within pictures. This means that a user interface with Ferret underneath could handle queries like those for ChatGPT or Gemini. Ferret was launched for research purposes in October last year.Â