What the community has measured.

How to read this

• MoE vs dense at the same total weight is the headline story for the 24 GB+ tier. Compare the RTX 5090 Qwen3 32B dense row to the Qwen3moe 30B-A3B row above it — same memory, ~3× the TG.
• Mac prefill is the real friction. The M5 Max Qwen 3.5 27B row shows PP ~10× lower than the 5090 with comparable TG. Long-prompt latency is what Mac users actually feel.
• AMD with the right backend is competitive. The RDNA3 Vulkan row on Qwen3 30B-A3B sits within ~10% of NVIDIA. The friction is software, not silicon.
• Capacity-first hardware (DGX Spark, M5 Max 128 GB) wins what NVIDIA consumer cards can’t do at all — running a 122B-class model on a single device. 38–60 tok/s is genuinely usable for the right workload.
• Backend choice can swing 2–3×. The Intel Arc B580 row reports a band because Vulkan delivers ~62 tok/s and the (now-archived) IPEX-LLM SYCL path delivers ~25–30. Same card, different software story.

The table

Caveats

• This is an audit log, not original research. We cross-check published community figures and date when we cross-checked. Where ≥2 sources agreed within ~5% we adopt the figure; where they disagreed we publish a band. The source list per category is in the References section below.
• Single-stream throughput only. Batched inference wins meaningfully at 4–8 concurrent requests; the numbers here are for one user, one prompt at a time — matching how community sources typically report.
• Llama-bench convention for PP and TG (pp512 and tg128 equivalents). MLX runs use the analogous mlx_lm metrics. Published source figures are typically the median of 3+ warmed-up runs in their original reports; FA setting stated per row.
• Context-length matters more than people think. 4K context numbers are not 32K context numbers. Where the context-length effect is the headline (the RTX 5090 30B-A3B long-context row), we publish both.
• Each row is dated independently. When a published figure goes stale (new runner, new quant, new model version), we cross-verify the new number and update that row with a new date, leaving the others alone.

References — community benchmark sources

These are the sources we cross-check against. Per-row entries typically draw from 2–3 of these; vendor-specific rows (DGX Spark, M5 Max, Strix Halo) lean harder on the vendor + community follow-up combination.

• LocalScore — community-submitted llama.cpp benchmark database with per-accelerator pages. Primary source for RTX 5090 / 5060 Ti / 5070 Ti / RTX 3090 figures.
• Hardware Corner — independent local-LLM hardware benchmarks. Primary source for Mac Apple Silicon (M3/M4/M5) figures across the Mac product line.
• llama.cpp GitHub Discussions + Issues — runner-author + maintainer benchmarks and the canonical place to find AMD ROCm vs Vulkan back-and-forth.
• NVIDIA Developer Forums — vendor-published DGX Spark throughput figures + community reproductions on driver versions (e.g. v2.1 patches for the 122B-A10B benchmark).
• Ollama blog — runner-author throughput claims for new backends (Metal, MLX, ROCm) at release.
• HuggingFace model card discussions on the canonical model repos (e.g. Qwen3.5-35B-A3B discussions, ubergarm GGUF quants for community llama-bench results).
• r/LocalLLaMA benchmark threads — community lived experience, especially useful for catching regressions in newer driver/runner builds.
• Databasemart benchmarks — server-hosted multi-GPU figures (dual 5090, dual A6000).

Want a row updated, added, or corrected? Send a reproducible benchmark — model, quant, runner, hardware, prompt, measured PP and TG — and we’ll cross-verify against the existing sources and either update the row or add a new one.

Back to the methodology