Tokens per second is two numbers

2026-09-20ENTR

Twelve years ago I built an app that read electricity meters. The field crews carried industrial Android devices.

One request from those days stayed with me. Build something that does not go back to a server.

I could not. There was no model that ran on the device, and no phone silicon that could carry one. The request made sense. The technology did not exist.

When I decided to start writing again, I went back to the problems I never solved and asked how reachable they are now.

Picture the same job today. The technician photographs the panel. The app confirms that it really is an electrical panel and that the meter really is an electricity meter. Nothing leaves the device.

I could put an API behind it now. I would rather not. The number of requests is not under my control, and abuse turns into a bill. Connectivity in the field is usually bad. Sometimes you are in a basement, sometimes on a mountain with no signal. And the technician needs to finish and move on.

So the job that was impossible twelve years ago now looks solvable on the device itself.

Before building it I needed three answers. Is this faster on the device or through an API. What does it cost. And how many seconds does the user actually wait.

To answer the first one I looked at model cards. Every one of them showed a single speed number.

That was the wrong number.

There is no single number

A language model runs in two phases. First it reads the prompt. Then it writes the answer. The two phases hit different hardware limits, so they produce different speeds. Across the nine configurations I measured, the gap between them ran from 5x to 100x. Where you land depends on the device, the runtime and the length of the prompt.

The single number on a model card is usually the second one, the speed at which the answer is written. If your job is to make the model read a long document, that number tells you very little.

Why the two are so different

In prefill the whole prompt is processed at once. The hardware sees one large matrix multiplication. It fetches a weight from memory once and does a lot of arithmetic with it. That makes prefill compute bound.

In decode the model produces one token at a time and feeds what it produced into the next step. The work is sequential and cannot be parallelised. Every single token requires every weight in the model to travel through memory. Little arithmetic, heavy memory traffic. That makes decode memory bandwidth bound.

One rule follows from this. The ceiling on decode speed is roughly memory bandwidth divided by the bytes actually read per token. The denominator is not the size of the model file. A mixture of experts model reads only a fraction of its weights per token, which is why one of them decoded three times faster than its file size predicted in my own desktop runs. The rule still gives you the direction. Lower the bytes read per token and decode speeds up, while prefill barely moves.

How I measured

Three devices. An iPhone 15 Pro, a Pixel 10 Pro XL, and a MacBook Pro M3 Max for scale. Both phones are real hardware, no emulators.

In the benchmarks I wrote myself, each measurement starts with a warm-up run that is thrown away, then three timed runs, median reported. Every prompt carries a random prefix, and I will explain why in a moment. Token counts come from the runtime’s own report, not from my estimate.

The method is not identical across sources, and I say so under each table. The llama.cpp rows use that tool’s own repetitions and averages. The two rows that come from Google’s app are the app’s own averages, not my medians.

I measured three workload shapes, because not every job on a phone is a chat.

Verifying a panel photo is the first row. Short prompt, very short output. In that job the thing that decides the wait is not how fast the model writes.

Three jobs on one phone

iPhone 15 Pro, Qwen3 1.7B in 4-bit, through MLX.

Same model, same phone. The wait for the first token moves between 196 ms and 6919 ms. Thirty five times.

Here is where I have to be honest. Length does not explain all of it. Look at the third row. A 54 token prompt takes 573 ms while a 51 token prompt finishes in 196 ms. Almost the same prompt, three times the wait. The only other difference is the generation budget, 400 tokens against 16. Space reserved for the cache probably lands inside that figure, but I did not isolate it, so I will say that I do not know.

The table holds a second thing. Decode slows down as the answer grows. 36.6 tok/s over fifty tokens, 29.6 over four hundred.

My first explanation was the growing context, and my own data refuses it. The extract run carries a 1741 token context and decodes at 36.6. The generate run carries 454 tokens, four times smaller, and decodes at 29.6. The run with the smaller context is the slower one. That leaves thermal drift as the likely candidate, and I did not record the thermal state, so I have not measured that either. One thing is certain. A model’s decode speed is not one number, it is a curve that depends on how long the answer is.

What the TPU actually buys you

This part surprised me the most.

On the Pixel 10 Pro XL I ran Gemma 4 E2B twice through Google’s AI Edge Gallery. One run on the CPU, one on the device’s TPU. Same settings, 256 prompt tokens and 256 generated tokens, three runs.

Let me state the flaw before the numbers. The app downloads a separate model file for the TPU, 4.0 GB, while the CPU file is 2.6 GB. So I could not run one file on two cores, I compared two builds. The accelerator is not the only thing that changed. These two figures also come from the app’s own benchmark screen, not from my stopwatch.

On prefill the TPU is almost ten times faster. On decode the gain is seven percent. I call that noise for a concrete reason. I measured the same CPU twice and the two runs differ from each other by five and a half percent. The claimed speedup is the size of one device’s run to run spread.

The direction fits the mechanism. A TPU exists to multiply matrices faster, and prefill is matrix multiplication. Decode pulls every weight through memory for every token, and the TPU does not change the memory bandwidth of this phone. That is not a general law. An accelerator with enough on-chip memory can speed up decode as well. It does not happen here.

There is a bill attached. On the TPU path each session took 3045 ms to become ready, on the CPU path 700 ms. If you keep the model warm the TPU wins by a wide margin. If you reload the model for every photo you hand back in warm-up what you gained in prefill.

Two phones and a desktop

Look at the two Pixel CPU rows. Same device, same model file. Only the thread count changes. More cores lift prefill and lower decode. The argument of this post is visible in one table. This device does not have a single speed number, and the best setting depends on which phase you care about.

The rows use different runtimes, so racing the devices against each other would be dishonest. Read each row on its own. One more row is worth a look.

The general purpose GPU path on the Pixel, Vulkan, came out 4.5 times slower than the CPU on prefill and 7 times slower on decode. The driver prints its own capabilities at startup and reports no integer dot product support and no matrix cores. That explains the prefill gap. It does not explain the 7x on decode, because decode is not compute bound and this GPU shares memory with the CPU. I do not know what causes that one. Driver maturity or shader efficiency are candidates, and I did not measure it.

Google’s own app offers no GPU option on this device either, only CPU and TPU. That is not proof, it is a hint pointing the same way.

Which side does a smaller model help

On the desktop I measured the same model at two quantization levels.

The file drops from 1.70 GB to 1.03 GB, a 39 percent reduction. Decode gains 38 percent. Prefill goes the other way and loses 4.6 percent, so the smaller model is slightly behind there. The memory bound and the compute bound in one pair of rows.

A pure bandwidth calculation would have predicted a 65 percent gain here, and the measurement says 38. The rule gives the direction, not the magnitude.

The same ordering is sharper on the phone. At six threads on the Pixel the 4-bit build read 107.6 tok/s on prefill while the 8-bit build read 138.1. On decode the order flips, 17.4 against 10.8.

The likely explanation is the extra work of unpacking 4-bit weights before they can be used. Prefill is compute bound, so that cost shows up there, while decode is memory bound, so the smaller file wins. I write this as a hypothesis, because I did not build an experiment that isolates it.

So the sentence “smaller quantization is always faster” is wrong too. It depends on which phase you are talking about.

Traps I fell into while measuring

All of these happened while producing the numbers above.

Sending the same prompt twice. The server reuses the cache from the previous request and skips prefill completely. With a repeated prompt the first token arrived in 98 ms. With a random prefix on the prompt, 599 ms. A six fold overstatement. The honest number is the second one.

Not checking what the build produced. My first Android build was not using the instructions the CPU supports, because a cmake test had failed quietly. On the same device, model and four threads, prefill read 22.4 tok/s before the fix and 92.6 after. A 4.1x difference from a compiler flag.

Missing the reasoning tokens. Models stream their thinking tokens in different fields, Qwen in one, gpt-oss in another. Watch only the normal text field and the first token looks late while decode looks impossible. During development I saw runs reporting tens of thousands of tokens per second, which is a buffered stream rather than a measurement. The published results come from the corrected code, and those broken runs are not in the repo.

Computing a decode rate from a handful of tokens. Below 24 generated tokens that figure is noise. That is why the classify row has no decode number in this post. Those runs produced five or six tokens.

Run provenance

Where every number in this post comes from. Everything you need to repeat a row is here.

The spread is in the repo as well. On the Pixel at six threads decode reads 15.66 ± 2.17 and at four threads 17.43 ± 0.74, so the difference between those two sits inside the deviations and I claim no firm ordering there.

Back to the panel

First the limit. Every measurement in this post uses text input. I did not measure image input, so the numbers below are not the panel job itself, they are the closest text job to it.

That job is a short prompt and a very short output, so prefill decides the wait. On the 256 token prompt I measured, the first answer came in 0.18 seconds on the TPU path and 1.30 seconds on the CPU path.

Do not carry that ratio into your own work as it stands. Around 50 ms of each figure is fixed overhead. For a genuinely short prompt, say 51 tokens, the same numbers work out to roughly 0.30 seconds on the CPU and 0.08 on the TPU. That is close to four times, not ten. The arithmetic is simple: subtract the fixed overhead from the 256 token run and scale what remains per token.

And if the model is reloaded every time, the three second warm-up on the TPU path is paid again for every photo, and the gain disappears. The architectural decision is simple. Keep the model warm and the session open.

With an image the mechanism stays the same but even more weight moves to prefill, because a photo encodes into hundreds of tokens. I will not put a number on that before I measure it. That is the next post. How many tokens is a photo of an electrical panel.

The one thing to remember

Look at two numbers when you pick a model.

If you make it read long text, prefill is your waiting time, and on the phone I measured the choice of accelerator changed that by an order of magnitude. If you make it write long answers, decode is your user experience, and on the same phone the accelerator changed nothing there. What changed it was lowering the bytes read per token.

I measured this on one phone with one model. Your numbers will be different. What will not be different is that there are two numbers to look at.

That was the question I could not answer twelve years ago. Today the answer can be measured.

If you would rather watch this than read it, I can run the whole thing live. Setting up the benchmark, pushing it to a real phone, reading the numbers as they come out, and wiring the result into an app. Tell me and I will record it.

Tools, raw output and method notes are in the repo: github.com/Burak-Arslan/prefill-vs-decode