In a newly released research paper by researchers from Cornell University working with Apple, details of an upgrade to Apple’s AI capabilities have been revealed. The paper titled “Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs” introduces a large language model that can understand elements displayed on a mobile user interface, such as an iPhone screen. This advancement, stemming from the earlier Ferret model, offers benefits for accessibility apps, Siri enhancements, and testing tools for developers.
This new AI technology, Ferret-UI, allows for the description of the screen regardless of the app being used, enabling users to perform complex tasks or access obscure options on their phones with ease. Additionally, developers can utilize Ferret-UI as a testing tool to simulate different user scenarios, such as a novice or elderly user, to ensure a seamless user experience. The upgrade to Ferret-UI is expected to provide a richer avenue for interaction with iPhones and improve the user’s overall experience.
Further findings suggest that new server-side tools, labeled as “Safari Browsing Assistant” and “Encrypted Visual Search,” have been uncovered, pointing towards potential additional AI features on individual iPhones. However, it is important to note that these discoveries are based on server-side code and may not necessarily indicate definitive future features. These findings are part of a growing body of evidence hinting at Apple’s approach to integrating AI into their products.
Apple’s AI capabilities have lagged behind Google’s AI-first approach to smartphones, with the latter beginning the rush for AI with the launch of Google Pixel phones. Apple has not made significant announcements regarding AI improvements, allowing Google to take the lead in AI innovations. The upcoming Worldwide Developer Conference in June will provide Apple with an opportunity to engage with the public and discuss its AI plans, which could potentially shape the development of iPhone 16 and iPhone 16 Pro set to be launched in September.
As Apple continues to enhance its AI capabilities, the iPhone’s approach to AI is disrupting the specifications for the upcoming iPhone 16 and iPhone 16 Plus. With the introduction of Ferret-UI and potential server-side tools, Apple is poised to catch up with competitors in the realm of AI-infused smartphones. The upcoming developments in AI technology by Apple are expected to revolutionize the user experience on iPhones and pave the way for future advancements in AI integration within Apple products.
