Apple’s iPhone is generally not seen as an AI-infused smartphone, but that may be changing soon. A newly released research paper by Cornell University researchers working with Apple details a multimodal large language model called Ferret-UI that can understand what is displayed on a mobile user interface. This model can pick out icons, find text, describe elements on the screen, and interact with the display based on open-ended instructions and prompts. This upgrade could potentially enhance Siri and offer various benefits to iPhone users.
Ferret, released in October 2023, was originally designed to parse photos and images to recognize objects. The Ferret-UI can now be utilized with iPhones, providing a richer avenue for accessibility apps and simplifying complex tasks. Users could ask Siri to navigate through different apps and find obscure options or functions. Additionally, developers can use Ferret-UI as a testing tool by simulating various user scenarios, such as a teenager or an elderly individual, to evaluate user experience with different applications.
While Google made significant AI advancements on Android devices, Apple has been relatively quiet about its AI improvements until now. Apple’s Worldwide Developer Conference in June will likely shed light on their AI plans and set the stage for the launch of the iPhone 16 and iPhone 16 Pro in September. It is expected that Apple will showcase its AI capabilities and how they will enhance their upcoming smartphones.
The release of the Ferret-UI highlights Apple’s focus on integrating AI into their devices and services to keep up with the industry trends. By incorporating AI technologies into their platforms, Apple aims to improve user experience, accessibility, and functionality of their products. This move by Apple indicates a shift towards AI-first smartphones, challenging the competition and setting higher expectations for future devices.
The inclusion of Ferret-UI in Apple’s AI strategy signifies a significant step towards enhancing the capabilities of the iPhone and Siri. With the ability to understand and interact with mobile user interfaces, Apple aims to provide a more intuitive and personalized user experience for iPhone users. This approach aligns with the trend of AI-first smartphones and showcases Apple’s commitment to improving their AI capabilities.
As Apple prepares for the launch of the iPhone 16 and iPhone 16 Pro, the integration of Ferret-UI and other AI advancements will likely disrupt the current industry standards for smartphone specifications. With a focus on enhancing user experience, accessibility, and functionality, Apple continues to innovate and push boundaries in the AI space. The upcoming iPhone models are expected to showcase Apple’s commitment to AI technology and set a new standard for AI-infused smartphones.
