63 measured runs
Measured speeds
Real runs, with the source for each one. We also use them to check our own speed estimate — in public.
49/52
short-prompt runs fall inside our estimated speed range
9%
typical gap between the middle of our range and the measured speed
Our estimate is simple physics: to write one word, the computer reads the whole active model from memory. We tuned two numbers on these runs, so treat this as a fit, not a blind test. New runs get added here and checked against the same formula. How it works.
Our own runs
Measured by us on a MacBook Pro (M5 Pro, 64 GB) on 10 October 2026, with llama.cpp’s own benchmark tool. These runs were not used to tune the estimate, so they are a fair test of it.
| Model | Conversation | Measured | Our estimate | Notes |
|---|---|---|---|---|
| Qwen3.8 27B (unsloth UD-Q4_K_XL) | 512 | 13.4 tok/s | 9.9–16.8 | Machine was busy with other work (load average about 470); decode varied only ±0.04 tok/s across 3 runs. llama.cpp d81235049 (11429) |
| Qwen3.8 27B (unsloth UD-Q4_K_XL) | 32K | 10.28 tok/s | 8.9–15.1 | Same machine and session. Shows how a long conversation slows writing. llama.cpp d81235049 (11429) |
We also tried the MLX version through Ollama, but results swung from 0.9 to 6.4 tok/s while the machine was busy, so we did not publish them. We will re-run on a quiet machine.
All measurements
| Hardware | Model | Version | Prompt | Speed | Runner | Date |
|---|---|---|---|---|---|---|
| MacBook Pro (M5 Pro, 64 GB) | Qwen3.8 27B | UD-Q4_K_XL | 512 | 13.4 tok/s | llama.cpp llama-bench (Metal), tg128, flash attention on, 3 runs | Ours |
| MacBook Pro (M5 Pro, 64 GB) | Qwen3.8 27B | UD-Q4_K_XL | 32K | 10.28 tok/s | llama.cpp llama-bench (Metal), tg128 after 32,768 tokens, flash attention on | Ours |
| Apple M1 (4P+4E CPU, 8-core GPU) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 14.15 tok/s | llama.cpp llama-bench (Metal), tg128 | 2023-12-02 |
| Apple M3 (4P+4E CPU, 10-core GPU, MacBook Pro 14) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 21.34 tok/s | llama.cpp llama-bench (Metal), tg128 | 2023-11-27 |
| Apple M4 (4P+6E CPU, 10-core GPU, Mac mini 2024) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 24.11 tok/s | llama.cpp llama-bench (Metal), tg128 | 2024-11-10 |
| Apple M5 (10-core GPU, MacBook Air 16 GB, macOS 26.6.1) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 31.88 tok/s | llama.cpp llama-bench (Metal), tg128 | 2026-08-30 |
| Apple M4 Pro (10P+4E CPU, 20-core GPU) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 50.74 tok/s | llama.cpp llama-bench (Metal), tg128 | 2024-11-08 |
| Apple M5 Pro (20-core GPU, MacBook Pro) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 66.33 tok/s | llama.cpp llama-bench (Metal), tg128 | 2026-08-31 |
| Apple M3 Max (12P+4E CPU, 40-core GPU, MacBook Pro 16) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 66.31 tok/s | llama.cpp llama-bench (Metal), tg128 | 2023-11-24 |
| Apple M4 Max (12P+4E CPU, 40-core GPU, MacBook Pro 16 2024) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 83.06 tok/s | llama.cpp llama-bench (Metal), tg128 | 2024-11-13 |
| Apple M4 Max (12P+4E CPU, 40-core GPU, MacBook Pro 16 2024) | Llama 2 7B (Q8_0) | Q8_0 | 512 | 54.05 tok/s | llama.cpp llama-bench (Metal), tg128 | 2024-11-13 |
| Apple M4 Max (12P+4E CPU, 40-core GPU, MacBook Pro 16 2024) | Llama 2 7B (F16) | F16 | 512 | 31.64 tok/s | llama.cpp llama-bench (Metal), tg128 | 2024-11-13 |
| Apple M5 Max (6S+12P CPU, 40-core GPU, 128 GB) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 119.92 tok/s | llama.cpp llama-bench (Metal), tg128 | 2026-08-25 |
| Apple M5 Max (6S+12P CPU, 40-core GPU, 128 GB) | Llama 2 7B (Q8_0) | Q8_0 | 512 | 72.42 tok/s | llama.cpp llama-bench (Metal), tg128 | 2026-08-25 |
| Apple M5 Max (6S+12P CPU, 40-core GPU, 128 GB) | Llama 2 7B (F16) | F16 | 512 | 37.11 tok/s | llama.cpp llama-bench (Metal), tg128 | 2026-08-25 |
| Apple M3 Ultra (24P+8E CPU, 80-core GPU, Mac Studio) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 92.14 tok/s | llama.cpp llama-bench (Metal), tg128 | 2025-03-23 |
| Apple M5 Ultra (36-core CPU, 80-core GPU, Mac Studio, 256 GB, macOS 27.0.1) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 179.1 tok/s | llama.cpp llama-bench (Metal), tg128 | 2026-09-30 |
| Apple M2 Ultra (16P+8E CPU, 76-core GPU, Mac Studio) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 94.27 tok/s | llama.cpp llama-bench (Metal), tg128 | 2023-11-22 |
| Apple M2 Ultra (16P+8E CPU, 76-core GPU, Mac Studio) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 125.21 tok/s | llama.cpp llama-bench (Metal) with flash attention, tg128 | 2026-08-25 |
| NVIDIA GeForce RTX 3060 12 GB (Ryzen 7 5700X, Linux, CUDA 13.0) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 76.92 tok/s | llama.cpp llama-bench (CUDA), -fa 1, tg128 | 2025-08-21 |
| NVIDIA GeForce RTX 5060 Ti 16 GB (single card of a 4-GPU rig, -sm none) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 93.46 tok/s | llama.cpp llama-bench (CUDA), -fa 1, tg128 | 2025-08-02 |
| NVIDIA GeForce RTX 5070 Ti 16 GB (300 W default, driver 580.105.08) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 182.43 tok/s | llama.cpp llama-bench (CUDA), -fa 1, tg128 | 2025-12-07 |
| NVIDIA GeForce RTX 5080 16 GB (driver 575.64.03, CUDA 12.9) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 184.68 tok/s | llama.cpp llama-bench (CUDA), -fa 1, tg128 | 2025-08-28 |
| NVIDIA GeForce RTX 3090 24 GB | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 161.89 tok/s | llama.cpp llama-bench (CUDA), -fa 1, tg128 | 2025-08-01 |
| NVIDIA GeForce RTX 4090 24 GB (nvidia-dkms 575.64.03, CUDA 12.9) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 188.96 tok/s | llama.cpp llama-bench (CUDA), -fa 1, tg128 | 2025-08-06 |
| NVIDIA GeForce RTX 5090 32 GB (Zotac Arctic Storm, 450 W) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 300.4 tok/s | llama.cpp llama-bench (CUDA), -fa 1, tg128 | 2025-10-23 |
| NVIDIA DGX Spark (GB10, 128 GB LPDDR5x) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 56.74 tok/s | llama.cpp llama-bench (CUDA 13.0, driver 580.95.05), -fa 1, tg128 | 2025-10-15 |
| NVIDIA GeForce RTX 4090 24 GB | Qwen3 30B-A3B-Instruct-2507 | Q4_K (assumed Q4_K_M) | 4K | 207.3 tok/s | llama.cpp (Hardware Corner; CUDA; build not stated on page) | 2026-03 |
| NVIDIA GeForce RTX 4090 24 GB | Qwen3 32B | Q4_K (assumed Q4_K_M) | 4K | 39.6 tok/s | llama.cpp (Hardware Corner; CUDA; build not stated on page) | 2026-03 |
| NVIDIA GeForce RTX 4070 12 GB | Qwen3 14B | Q4_K (assumed Q4_K_M) | 4K | 42.5 tok/s | llama.cpp (Hardware Corner; CUDA; build not stated on page) | 2026-03 |
| NVIDIA GeForce RTX 4070 Ti 12 GB | Qwen3 8B | Q4_K (assumed Q4_K_M) | 4K | 75.8 tok/s | llama.cpp (Hardware Corner; CUDA; build not stated on page) | 2026-03 |
| NVIDIA GeForce RTX 4060 Ti 16 GB | Qwen3 14B | Q4_K (assumed Q4_K_M) | 4K | 27.4 tok/s | llama.cpp (Hardware Corner; CUDA; build not stated on page) | 2026-03 |
| NVIDIA DGX Spark (GB10, 128 GB LPDDR5x) | Qwen2.5-Coder-7B-Instruct (Q8_0) | Q8_0 | 512 | 29.43 tok/s | llama.cpp llama-bench (CUDA), -fa 1 -ub 2048 -mmp 0, pp2048/tg32 | 2026-02-05 |
| NVIDIA DGX Spark (GB10, 128 GB LPDDR5x) | Qwen3-Coder 30B-A3B | Q8_0 | 512 | 61.06 tok/s | llama.cpp llama-bench (CUDA), -fa 1 -ub 2048 -mmp 0, pp2048/tg32 | 2026-02-05 |
| NVIDIA DGX Spark (GB10, 128 GB LPDDR5x) | gpt-oss 120B | MXFP4 | 512 | 58.72 tok/s | llama.cpp llama-bench (CUDA), -fa 1 -ub 2048 -mmp 0, pp2048/tg32 | 2026-02-05 |
| NVIDIA DGX Spark (GB10, 128 GB LPDDR5x) | Qwen3.5 35B-A3B | Q8_0 | 512 | 51.33 tok/s | llama.cpp llama-bench (CUDA), -fa 1 -ub 2048 -mmp 0, pp2048/tg32 | 2026-03-10 |
| AMD Radeon RX 7900 XTX 24 GB (Powercolor Red Devil, Windows 11, Adrenalin 26.8.1) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 190.92 tok/s | llama.cpp llama-bench (Vulkan, AMD proprietary driver), -fa 1, tg128 | 2026-09-10 |
| AMD Radeon RX 9070 XT 16 GB (Mesa RADV GFX1201, Linux) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 137.11 tok/s | llama.cpp llama-bench (Vulkan, Mesa RADV), -fa 1, tg128 | 2026-01-24 |
| Intel Arc B580 12 GB (Linux 6.17-rc6, Mesa 25.2.3) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 72 tok/s | llama.cpp llama-bench (Vulkan, Mesa open-source driver), -fa 1 -sm none, tg128 | 2025-09-20 |
| Intel Arc B580 Limited Edition 12 GB (Core Ultra 5 250K Plus, CachyOS 7.0.11, oneAPI 2026.0) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 88.13 tok/s | llama.cpp llama-bench (SYCL, GGML_SYCL_F16=ON, min core clock pinned 2850 MHz), -fa 1, tg128 | 2026-06-13 |
| Intel Arc B580 Limited Edition 12 GB (Core Ultra 5 250K Plus, CachyOS 7.0.11, oneAPI 2026.0) | Gemma 4 12B | Q4_K_M | 512 | 41.34 tok/s | llama.cpp llama-bench (SYCL, GGML_SYCL_F16=ON), -fa 0, tg128 | 2026-06-13 |
| AMD Ryzen AI Max+ 395 "Strix Halo" / Radeon 8060S (Framework Desktop, 128 GB LPDDR5x-8000) | Llama 2 7B (llama-2-7b.Q4_0.gguf) | Q4_0 | 512 | 51.88 tok/s | llama.cpp llama-bench (Vulkan, Mesa RADV), -fa 1 --mmap 0, build 7ad67ba9 (6111) | 2025-08-27 |
| AMD Ryzen AI Max+ 395 "Strix Halo" / Radeon 8060S (Framework Desktop, 128 GB LPDDR5x-8000) | Qwen3 30B-A3B-Instruct-2507 | UD-Q4_K_XL | 512 | 78 tok/s | llama.cpp llama-bench (Vulkan AMDVLK, b=256) - best backend per model, lhl summary | 2025-08 |
| AMD Ryzen AI Max+ 395 "Strix Halo" / Radeon 8060S (Framework Desktop, 128 GB LPDDR5x-8000) | gpt-oss 120B | MXFP4 | 512 | 33.7 tok/s | llama.cpp llama-bench (Vulkan AMDVLK, b=256) - best backend per model, lhl summary | 2025-08 |
| AMD Ryzen AI Max+ 395 "Strix Halo" / Radeon 8060S (Framework Desktop, 128 GB LPDDR5x-8000) | Qwen3 32B | Q8_0 | 512 | 6.4 tok/s | llama.cpp llama-bench (Vulkan AMDVLK, fa=1) - best backend per model, lhl summary | 2025-08 |
| Apple M4 Pro (20-core GPU, 24 GB) | Llama 3.1 8B | MLX 4-bit | 4K | 47.6 tok/s | oMLX v0.7.0 (MLX), macOS 27.0.1 | 2026-10-06 |
| Apple M5 Max (40-core GPU, 128 GB) | Qwen3 30B-A3B-Instruct-2507 | MLX 4-bit | 4K | 119.1 tok/s | oMLX v0.4.4 (MLX), macOS 26.5.1 | 2026-06-20 |
| Apple M3 Ultra (80-core GPU, 512 GB) | gpt-oss 120B | MLX 8-bit | 1K | 67.8 tok/s | oMLX v0.5.7 (MLX), macOS 26.6 | 2026-08-06 |
| Apple M5 Max (18-core CPU, 40-core GPU, 14-inch MacBook Pro, 128 GB) | Qwen3.5 122B-A10B | MLX 4-bit | 4K | 65.9 tok/s | mlx_lm (MLX), version not stated | 2026-03-11 |
| Apple M5 Max (18-core CPU, 40-core GPU, 14-inch MacBook Pro, 128 GB) | gpt-oss 120B | MXFP4 (MLX, 8-bit) | 4K | 87.9 tok/s | mlx_lm (MLX), version not stated | 2026-03-11 |
| Apple M4 Max (12P+4E CPU, 40-core GPU, 128 GB) | Llama 3.1 8B | Q4_K_M | 2K | 52.9 tok/s | LocalScore (Llamafile-based harness), average across its 9 prompt/generation tests | |
| NVIDIA GeForce RTX 5060 Ti (reported as 15 GB) | Llama 3.1 8B | Q4_K_M | 2K | 59 tok/s | LocalScore (Llamafile-based harness), average across its 9 prompt/generation tests | |
| NVIDIA GeForce RTX 5090 32 GB | Qwen3.5 35B-A3B | MXFP4 | 256K | 97.3 tok/s | llama.cpp CUDA, FA on (per calibration.js) | 2026-03 |
| NVIDIA GeForce RTX 5090 32 GB | Gemma 4 31B | Q4_K | 128K | 43.4 tok/s | llama.cpp CUDA, FA on (per calibration.js) | 2026-03 |
| NVIDIA GeForce RTX 3090 24 GB (used) | gpt-oss 20B | MXFP4 | 4K | 147.5 tok/s | llama.cpp CUDA, -fa 1 (HC row label) | 2026-03 |
| NVIDIA GeForce RTX 3090 24 GB (used) | Qwen3.5 35B-A3B | MXFP4 | 128K | 79.4 tok/s | llama.cpp CUDA, FA on (per calibration.js) | 2026-03 |
| NVIDIA GeForce RTX 3090 24 GB (used) | Gemma 4 26B-A4B | Q4_K | 256K | 64.4 tok/s | llama.cpp CUDA, FA on (per calibration.js) | 2026-03 |
| NVIDIA GeForce RTX 3090 24 GB (used) | Gemma 4 31B | Q4_K | 32K | 31.4 tok/s | llama.cpp CUDA, FA on (per calibration.js) | 2026-03 |
| NVIDIA GeForce RTX 5060 Ti 16 GB | gpt-oss 20B | MXFP4 | 128K | 43.8 tok/s | llama.cpp CUDA, FA on (per calibration.js) | 2026-03 |
| NVIDIA GeForce RTX 5060 Ti 16 GB | Qwen3 14B | Q4_K | 32K | 25.9 tok/s | llama.cpp CUDA, FA on (per calibration.js) | 2026-03 |
| NVIDIA DGX Spark (GB10, 128 GB LPDDR5x) | Qwen3.5 35B-A3B | Q4_K_M | 512 | 59.98 tok/s | llama.cpp llama-bench (CUDA), pp2048/tg32 | 2026-03-30 |
| NVIDIA DGX Spark (GB10, 128 GB LPDDR5x) | gpt-oss 20B | MXFP4 | 512 | 86.23 tok/s | llama.cpp llama-bench (CUDA), -fa 1 -ub 2048 | 2025-11-16 |
| Apple M4 Max (40-core GPU, MacBook Pro, 64 GB) | gpt-oss 20B | MXFP4 | 512 | 117.83 tok/s | llama.cpp llama-bench (Metal), -fa 1 -ub 2048, 1 thread | 2025-11-16 |
Speed = text generation (decode) in tokens per second. “Prompt” is how much text was already in the conversation. Click a date for the source.
Corrections
10 October 2026 — two old rows were the wrong machine
Our previous benchmark page listed two DGX Spark results (Qwen3-Coder-30B at 44 tok/s and gpt-oss-120b at 42 tok/s). The source comment was about NVIDIA’s Jetson AGX Thor, not the DGX Spark. The real DGX Spark numbers from the llama.cpp project are 61 and 59 tok/s. We removed the wrong rows.