Google says running AI models on phones is a huge RAM hog

In early March, Google made the odd announcement that solely one of its two newest smartphones, the Pixel 8 and Pixel 8 Professional, would have the ability to run its newest AI mannequin, known as “Google Gemini.” Regardless of having very comparable specs, the smaller Pixel 8 would not get the brand new AI mannequin, with the corporate citing mysterious “{hardware} limitations” as the explanation. It was an odd assertion contemplating the truth that Google designed and marketed the Pixel 8 to be AI-centric after which designed a smartphone-centric AI mannequin known as “Gemini Nano” but nonetheless could not make the 2 work collectively.

A number of weeks later, Google is backtracking considerably. The corporate introduced on the Pixel Phone Help discussion board that the smaller Pixel 8 really will get Gemini Nano within the subsequent large quarterly Android launch, which ought to occur in June. There is a catch, although—whereas the Pixel 8 Professional will get Gemini Nano as a user-facing function, on the Pixel 8, it is solely being launched “as a developer possibility.” Meaning you’ll flip it on solely through the hidden Developer Choices menu within the settings, and most of the people will miss out on it.

Google’s Seang Chau, VP of units and providers software program, defined the choice on the corporate’s in-house “Made by Google” podcast. “The Pixel 8 Professional, having 12GB of RAM, was an ideal place for us to place [Gemini Nano] on the system and see what we might do,” Chau stated. “Once we regarded on the Pixel 8 for example, the Pixel 8 has 4GB much less reminiscence, and it wasn’t as simple of a name to only say, ‘all proper, we will allow it on Pixel 8 as effectively.'” In response to Chau, Google’s trepidation is as a result of the corporate does not need to “degrade the expertise” on the smaller Pixel 8, which solely has 8GB of RAM.

Chau went on to explain what it is prefer to have a big language mannequin like Gemini Nano in your cellphone, and it seems like there are large trade-offs concerned. Google needs a number of the AI fashions to be “RAM-resident” so that they’re at all times loaded in reminiscence. One such function is “good reply,” which tries to auto-generate textual content replies.

Chau informed the podcast, “Good Reply is one thing that requires the fashions to be RAM-resident in order that it is obtainable on a regular basis. You do not need to await the mannequin to load on a Gboard reply, so we hold it resident.” For that purpose, Google is retaining the Gemini-powered good reply behind a developer flag for each the Pixel 8 and Pixel 8 Professional. The “Good reply” choices within the regular keyboard settings aren’t Gemini-powered.

So in contrast to an app, which will be loaded and unloaded as you employ it, working one thing like Gemini Nano might imply completely shedding what is seemingly an enormous chunk of system reminiscence. The baseline of 8GB of RAM for Android telephones might must be elevated once more sooner or later. The excessive mark we have seen for telephones is 24GB of RAM, and the larger flagships often have 12GB or 16GB of RAM, so it is definitely doable.

Google’s Gemini Nano mannequin can also be delivery on the Galaxy S24 lineup, and the bottom mannequin there has 8GB of RAM, too. When Google initially cited {hardware} limitations on the Pixel 8 for the function’s absence, its rationalization was complicated—if the base-model S24 can run it, the Pixel 8 ought to have the ability to as effectively. It is all about how a lot of a trade-off you are prepared to make in obtainable reminiscence for apps, although. Chau says the group is “nonetheless doing system well being validation as a result of even when you’re a developer, you would possibly need to use your cellphone each day.”

The elephant within the room, although, is that as a person, I do not even know if I need Gemini Nano on my cellphone. We’re on the peak of the generative AI hype cycle, and Google has its own internal reasons (the inventory market) for pushing AI so exhausting. Whereas visiting ChatGPT and asking it questions will be helpful, that is simply an app. Truly helpful OS-level generative AI options are few and much between. I do not really want a keyboard to auto-generate replies. If it is simply going to make use of up a bunch of RAM that could possibly be utilized by apps, I’d need to flip it off.

This publish was up to date March 30 after receiving extra knowledge from Google PR. Google informed us neither the Pixel 8 or Pixel 8 Professional is retaining Gemini in reminiscence proper now, except you activate the developer flag. Meaning by default, neither system’s good reply function is powered by Gemini.