What the community has measured.

How to read this

• MoE vs dense is the whole ballgame at 24 GB+. On one used RTX 3090, Gemma 4’s 26B MoE generates 64 tok/s while holding 256K of context; Gemma 4’s 31B dense manages 31 tok/s at 32K. Same family, same vendor, same quant, same card — the architecture is the entire difference. This is why our picks lean MoE.
• A MacBook out-decodes a $4,699 DGX Spark. Identical model, identical quant (gpt-oss-20b MXFP4): M4 Max 118 tok/s, DGX Spark 86. The Spark wins prompt processing and wins capacity outright — it runs gpt-oss-120b at 42 tok/s, which no consumer card in this table can load at all. But on generation, Apple’s memory bandwidth simply wins. Buy the Spark for what fits, not for what’s fast.
• Context is the tax nobody quotes you. The 3090 runs gpt-oss-20b at 148 tok/s at 4K and 62 tok/s at 128K. Same card, same model, less than half the speed. Any benchmark quoted without a context depth is quoting you its best case.
• A runner build can beat a hardware upgrade. The DGX Spark row is the cautionary one: prompt processing went from 2,009 to 3,798 tok/s on the same box and model, purely by moving llama.cpp from build 6771 to 7067. That is an 89% gain from a software bump. It is also why every row here pins a runner — a throughput number without a build is a rumour.
• The gaps are part of the data. There is no Mac Studio M3 Ultra row, no Mac mini row, no Arc B580 row, and no AMD row, because we could not verify a current-model benchmark for them to the standard above. Those numbers are easy to find elsewhere and mostly should not be trusted — at least one popular source generates them from a formula and admits it in its footer.

The table

Caveats

• This is an audit log, not original research. We cross-check published community figures and date when we cross-checked. Where ≥2 sources agreed within ~5% we adopt the figure; where they disagreed we publish a band. The source list per category is in the References section below.
• Single-stream throughput only. Batched inference wins meaningfully at 4–8 concurrent requests; the numbers here are for one user, one prompt at a time — matching how community sources typically report.
• Llama-bench convention for PP and TG (pp512 and tg128 equivalents). MLX runs use the analogous mlx_lm metrics. Published source figures are typically the median of 3+ warmed-up runs in their original reports; FA setting stated per row.
• Context-length matters more than people think. 4K context numbers are not 32K context numbers. Where the context-length effect is the headline (the RTX 5090 30B-A3B long-context row), we publish both.
• Each row is dated independently. When a published figure goes stale (new runner, new quant, new model version), we cross-verify the new number and update that row with a new date, leaving the others alone.

References — community benchmark sources

Every row above links its own primary source directly, so you can check any single number without trusting this list. These are the sources we consider credible enough to draw from in the first place — they publish a methodology, state their runner and build, and report results that survive arithmetic.

What we refuse to cite, and why you should care. A large share of the pages ranking for “[model] on [GPU] tok/s” are not measurements. Some are calculators — WillItRunAI states in its own footer that “all estimates are approximations based on mathematical models and public specifications”, and it returns the same tok/s for a 96 GB and a 256 GB machine because a formula cannot tell them apart. Others are simply broken: we found sites reporting prompt processing an order of magnitude below token generation, which is not physically possible on these runners, and one claiming an RTX 3090 beats an RTX 4090 by 15× on the same workload. A third group reprints Hardware Corner’s figures verbatim and reads like independent corroboration when it is nothing of the sort. We cited an estimate engine ourselves in planner copy until July 2026 — this list exists so that does not happen twice.

• LocalScore — community-submitted llama.cpp benchmark database with per-accelerator pages. Primary source for RTX 5090 / 5060 Ti / 5070 Ti / RTX 3090 figures.
• Hardware Corner — independent local-LLM hardware benchmarks. Primary source for Mac Apple Silicon (M3/M4/M5) figures across the Mac product line.
• llama.cpp GitHub Discussions + Issues — runner-author + maintainer benchmarks and the canonical place to find AMD ROCm vs Vulkan back-and-forth.
• NVIDIA Developer Forums — vendor-published DGX Spark throughput figures + community reproductions on driver versions (e.g. v2.1 patches for the 122B-A10B benchmark).
• Ollama blog — runner-author throughput claims for new backends (Metal, MLX, ROCm) at release.
• HuggingFace model card discussions on the canonical model repos (e.g. Qwen3.5-35B-A3B discussions, ubergarm GGUF quants for community llama-bench results).
• r/LocalLLaMA benchmark threads — community lived experience, especially useful for catching regressions in newer driver/runner builds.
• Databasemart benchmarks — server-hosted multi-GPU figures (dual 5090, dual A6000).

Want a row updated, added, or corrected? Send a reproducible benchmark — model, quant, runner, hardware, prompt, measured PP and TG — and we’ll cross-verify against the existing sources and either update the row or add a new one.

Back to the methodology