Nobody Is Deploying On-Device AI for Privacy — Here’s What They’re Actually Buying

Every pitch for on-device AI leads with privacy. Privacy is not the reason it is being deployed.

Here is the reason. Run a language model on your own hardware and the cost per request is brutal and visible. Chitral Patil’s June 2026 measurement study found that on identical H100 silicon, effective cost spans $0.21 to $15.25 per million output tokens — a 72-fold spread — and that the variable driving it is not the model, the hardware, or the optimisation stack. It is the request rate actually being served. Public cost calculators assume utilisation as an input, so they systematically understate true cost by exactly the inverse of it. Near idle, the penalty reaches 36x. Every inference you serve is a line item that grows with usage and shows up in a budget review.

Move that inference onto the customer’s phone and the line item disappears. Not because the compute got cheaper — it did not. Because somebody else paid for the silicon before you arrived.

This is a cost transfer dressed as an architecture decision, and it explains almost everything about how the platform vendors are positioning the shift. Apple and Google are not shipping on-device inference as a privacy feature with a performance benefit. They are shipping it as the mechanism that converts a brand’s variable cost into a fixed cost already sunk into a device the customer bought, and will replace, on their own schedule.

Which is why the defaults suddenly matter so much. Firebase AI Logic exposes four routing modes — prefer on-device, prefer cloud, only on-device, only cloud. Whoever picks the default decides whose budget absorbs the inference. That is not a technical parameter. That is a negotiating position.

The transfer is not free, and the research is unusually blunt about where the bill lands. A July 2026 study by Guégain and Coignion measured 18 models across two smartphones and a server and found on-device inference is, on average, three times less energy-efficient per generated token than batched server inference. Worse for the sustainability story: 88–90% of the environmental impact of local inference traces to the device’s embodied carbon — manufacturing already done — rather than the electricity it burns. The handset does not amortise that cost. It inherits it.

And the hardware is not the hero of the story either. A May 2026 stage-level benchmark by Li and colleagues found CPUs beating NPUs in the compute-heavy prefill stage by up to 1.6x, with NPUs delivering only 1.05–1.2x in the memory-bound decode stage — and more NPU offloading pushing energy consumption up by as much as 51%. The chip marketed as the AI engine is not reliably the efficient one.

Counterpoint’s analysts noticed the same gap from the market side: GenAI capability is now standard above roughly $400 wholesale, yet a clear divide remains between phones that can run AI and phones whose owners regularly use it.

Put those together and you get the actual insight. The reason on-device AI is spreading is not that it is better, cheaper, or greener. It is that it relocates the meter. Brands get near-zero marginal cost at the moment of use; customers get the electricity bill, the battery degradation, and a slightly earlier upgrade. That trade is defensible. It is just not the trade being advertised — and the brands that say it out loud will be the ones trusted when the bill arrives.