On-device AI: what it is genuinely good at now

Two years ago "AI on your phone" meant a request to a server with a spinner in front of it. That is still how most of it works, but a meaningful set of tasks now runs entirely on the device in your hand — no account, no upload, no connection — and the list is specific enough to be worth knowing precisely.

On-device AI is machine learning that runs on the phone or computer's own processor rather than in a data centre. The trade is fixed: smaller models, so lower ceiling on the hardest inputs, in exchange for privacy, offline operation, no per-use cost and no wait for a round trip.

What is on-device AI good at today?

On-device AI is dependable across five categories today, and they share a shape: bounded input, bounded output, and a task the model has seen a great deal of.

• Speech to text. Dictation and transcription run locally on current phones and laptops with accuracy that is fine for clear speech in a quiet room. This is the most mature category by some distance.
• Text to speech. Reading text aloud with natural prosody, entirely offline.
• Image understanding. Recognising text in a photo, identifying objects, finding "the picture with the receipt in it", lifting a subject from its background.
• Translation. Common language pairs, with downloadable packs, at a quality that is fine for comprehension and not for publication.
• Small text tasks. Summarising a short passage, suggesting a reply, cleaning up formatting. The ceiling is low, but the tasks are frequent.

Ask whether the task is bounded. On-device AI is strong when the input is one page, one image or one minute of audio, and weak when it is a book.

Notice what is missing from that list: long-document reasoning, code generation of any substance, image generation at quality, and anything requiring current knowledge of the world. Those remain server work, and will for a while.

What you actually gain

On-device AI buys you four things, and the second is the one people underestimate.

Privacy that is structural rather than promised. A recording that never leaves the phone cannot be retained, breached or subpoenaed from a service. For medical notes, legal matters, journalism and unreleased work, this is the difference between usable and not — which is why local transcription in apps like TapMemo is a category decision rather than a feature.

Latency you can build a habit on. No network round trip means the result appears in the same interaction. A dictation that returns in 200 milliseconds is a different tool from one that returns in two seconds, even though both are fast.

No cost per use. Nothing meters it, so it can run on everything rather than on what you decide is worth spending on.

It works with no signal. On a plane, in a basement, abroad without roaming.

The latency point compounds. Features that are cheap and instant get used casually, and casual use is where most of the value in these tools actually accumulates.

What you give up

The honest ledger, because the trade is real.

Accuracy falls off at the edges. Noisy audio, heavy accents, technical vocabulary, unusual handwriting. The average case is close to server quality; the difficult case is not, and averages hide this.

Battery and heat are real. Sustained local processing warms the device and drains it, and a long transcription is noticeably more expensive than a short one.

Storage. Models take space — language packs, speech models, image models — often a gigabyte or more once you have a few.

Improvement is slower. A server model gets better without you doing anything. On-device AI improves when you update the OS or the app, which is a slower cadence you do not control.

How to tell which one an app is using

Vendors are inconsistent about saying whether a feature uses on-device AI, and the wording is often deliberately ambiguous. Four checks settle it.

1. Turn on airplane mode and try the feature. If it works, it is local. This is definitive and takes five seconds.
2. Look for a download step. An app that downloaded a model pack is running it locally; there is no other reason to fetch several hundred megabytes.
3. Read the privacy label for data collection. An app sending audio or images off the device generally has to say so.
4. Watch the first run versus later runs. A local model is slow the first time — loading into memory — then fast. A server model is roughly constant.

The first check is the one to use. Marketing language about "processing on your device" sometimes describes only part of a pipeline, and airplane mode does not negotiate.

What does it need from your hardware?

The phrase "runs on the device" hides a wide range, and the same feature can be instant on one phone and absent on another.

Three constraints decide it:

• A neural accelerator. Current phones and laptops carry dedicated hardware for this work, and it is perhaps an order of magnitude more efficient than doing the same arithmetic on the main processor. Devices without one can often still run the model, slowly and hot.
• Memory headroom. A model occupies memory for as long as it is loaded, which is why some features refuse to run while another heavy app is open, and why older devices with less RAM simply do not offer them.
• Free storage. Language packs and speech models are downloaded, and a phone that is nearly full will silently decline to fetch them — a common and confusing reason for a feature to be missing.

This is why availability is uneven in a way that looks arbitrary. A feature present on a two-year-old flagship may be missing on a newer mid-range device, because the deciding factor is the accelerator and the memory rather than the release date.

The practical consequence for choosing software: check the requirements before paying, and if a feature matters to you, test it on your own device rather than trusting a demo. And if a feature you had disappears after an update, look at free storage first — an evicted model pack is a far more common cause than a removed feature.

Choosing sensibly, task by task

A rule of thumb that holds up in practice: use local for anything frequent, private or short, and server for anything hard or long.

Voice notes and dictation — local, always. Frequent, private, and the local quality is sufficient.

Transcribing a two-hour interview — depends. Local is free and private but slow and warm; a server is faster and better on crosstalk. If the content is sensitive, the answer is local regardless of the inconvenience.

Translating a menu — local. Bounded, instant, and works without a signal, which is when you need it.

Translating a contract — server, then a human. The failure cost is too high for a small model.

Summarising a chat thread — local is fine. Summarising a research paper is not.

Reading text aloud — local, unless you need a specific studio-quality voice for publication, where a dedicated tool such as SpeakFile gives you control the system voice does not.

Before installing anything, check what your device already does. Dictation, live text in photos, subject lifting, and offline translation packs are built into current phones, and a surprising number of paid apps are wrappers around exactly those APIs.

More on the tools in AI tools, the general picks in best tools, and comparisons in comparisons. Apple documents the on-device speech and vision APIs that most of these features are built on.

The short version

On-device AI is reliable today for speech to text, text to speech, image understanding, common-pair translation and short text tasks — bounded jobs where the input fits in a page, a photo or a minute of audio.

Use it for anything frequent, private or short, and reach for a server when the input is long or genuinely hard. To find out which one an app uses, turn on airplane mode and try the feature — that answers in five seconds what a marketing page will not.