How fast will a local AI answer?
Memory speed sets the pace. We measured it on our own machines.
A model writes one token at a time. A token is a piece of a word: in English, 100 tokens is about 75 words. For every token, the machine reads the model's numbers from memory once. So the top speed is roughly memory bandwidth divided by the gigabytes read for each token.
We measured it on the Mac mini in Prototype 1.2, which reads its memory at 307 GB/s. A dense 27 billion parameter model at 4 bits is about 17.5 GB, so the rule predicts about 17 tokens a second at most. We measured 15. A dense 9 billion parameter model measured 48.
Mixture of experts models change the maths. They keep many parameters in memory but use only a few for each token. Qwen3.6-35B-A3B holds 35 billion parameters and reads about 3 billion per token. On the same Mac it measured 66 tokens a second, faster than the 9 billion model. The 80 billion parameter Qwen3-Coder-Next, squeezed to 3 bits, measured 60.
Other machines, as reported by their owners for that same coder: a Mac Studio with an M5 Max (614 GB/s) does about 80 tokens a second at 8 bits. The AMD Ryzen AI Max+ 395 that DAK-1 is planned around (256 GB/s) does about 40 at 4 bits, with room for much larger models in its 128 GB.
Reading your prompt is a different job. It depends more on raw computing power than on memory, and runs much faster: our Mac mini reads about 1,000 tokens a second with that coder. That's why a long document takes a moment to start, and then the answer streams at a steady pace.
So pick by job. For one person coding all day, speed matters most. For a team with many jobs and private files, memory size matters more. For drafts and scans, a smaller box is plenty.
Quick answers
How do I estimate tokens per second for a local model?
Divide the machine's memory bandwidth by the gigabytes the model reads for each token. Real speeds land at about 85 to 90 percent of that.
Why can a 35B model be faster than a 9B model?
A mixture of experts model like Qwen3.6-35B-A3B reads only about 3 billion of its parameters for each token, so it reads less memory per token than a dense 9 billion model.
Sources
- Our measurements: llama.cpp llama-bench on a Mac mini M5 Pro, 48 GB (18 Nov 2026)
- Apple: Mac mini M5 Pro, 307 GB/s memory bandwidth
- Apple: Mac Studio M5 Max, 614 GB/s memory bandwidth
- AMD: Ryzen AI Max+ 395, 256-bit LPDDR5X-8000
- oMLX community benchmarks: Qwen3-Coder-Next on M5 Max
- strix-halo-llm-perf: Qwen3-Coder-Next on Ryzen AI Max+ 395
- OpenAI: what tokens are