Dictating a message on a plane, with no signal, is a small test of on-device AI. If the words appear, the speech model is running on the phone. If it stalls, the request was waiting for a server. As more features of this kind arrive, it is worth knowing which design a product is using, because the answer affects privacy, speed and what the feature can do.
Two approaches from the largest phone makers
Apple has published detailed descriptions of its design. Its Apple Intelligence system pairs a model that runs on the device, described in Apple’s 2024 technical report as having roughly three billion parameters, with larger server models that run on what it calls Private Cloud Compute. Apple’s research papers describe optimizing the on-device model for Apple’s chips, including quantization-aware training at very low numeric precision, which shrinks the memory the model needs. Apple says that when Private Cloud Compute handles a request, the user’s personal data is not stored or made accessible to Apple, and that outside experts can verify the claim.
Google’s route on Android runs through a system service called AICore, which hosts a compact model named Gemini Nano. Google’s developer documentation says on-device generative AI executes prompts locally, with no calls to a server, which keeps sensitive data on the phone, allows offline use and reduces inference costs. The same documentation says AICore has no direct internet access of its own and is built not to keep a record of inputs or outputs after processing them.
How a model gets small enough
Models for phones are shrunk in several ways. Quantization stores a model’s numbers at lower precision, so it fits in less memory and runs faster. Pruning removes parts that contribute little, and Apple has described a sparse on-device model built with a technique it calls instruction-following pruning. Phones also carry dedicated processors, often called neural processing units, designed for the arithmetic involved while using far less battery than a general-purpose chip would.
What local processing buys you
Privacy is the headline benefit, with a caveat. Processing a request on the device removes the need to send it away, but it does not stop an app from sending data elsewhere for other reasons, so the product’s own privacy terms still matter.
Speed is the second. With no round trip to a server, responses for suitable tasks can appear almost instantly, and they keep working without a connection. For companies, every request handled locally is one they do not pay to run in a data center.
Local models can also work on personal material without uploading it. A phone can search your own notes or messages and give the model relevant passages, a pattern similar to retrieval-augmented generation with the files staying on the device.
What you give up
Google’s documentation is candid about the limits: Gemini Nano has constraints such as token limits, device hardware limitations and boundaries on what the model can do. Its guidance for heavier work is hybrid inference, splitting jobs between the device and the cloud. Support is also uneven. On-device generative AI needs compatible hardware, and older phones may not get new features.
Smaller models make mistakes of the kind described in our explainer on AI hallucinations, and they tend to be less capable on long, complex requests than the largest cloud models. Heavy local use also draws power and warms the device.
Reading a product’s claims
Most products now mix both approaches, and the labels can blur the difference. Useful questions are specific: which features run locally, which need the cloud, what is sent when a request does leave the device, and whether the model will keep improving after purchase or only with a system update. A product that can answer those plainly is describing its design. One that says only “powered by on-device AI” may be describing its marketing.
