Resources

production · current Beats DeepSeek V3.2 on sequential tool-use, loses on multi-hop reasoning. ~130 tok/s peak on GGUF (new hardware record), ~101 tok/s warm on the NVFP4A16 production build. Head-to-head vs DeepSeek V3.2: 84.8% vs 86.4% overall, but Ornith goes 20/20 vs DeepSeek's 11/20 on single-tool sequential typewriter tasks — DeepSeek loops to the turn cap not knowing when to stop. DeepSeek wins multi-hop math and relational lookups. Route by task shape, not model religion.

tested · standby backend 117.6 tok/s — a hardware record that held for a month. Requires the cuda128-clean llama.cpp build target; cuda13 crashes with a CUDA MMQ kernel error on SM_120, every time. Quality: 4.84/5.00 on a 25-probe suite, perfect scores on reasoning and factuality. NVFP4 via TensorRT-LLM is hardware-limited — Mamba layers can't tensor-parallelize, and the sharding path comes up 3MB short of VRAM headroom with no fix available.

standby backend The model that proved vLLM was mandatory. In llama.cpp: 22 tok/s — no CUDA kernel for GDN hybrid layers. In vLLM with Genesis patches + MTP n=3: 83–88 tok/s. The AutoRound INT4 quant from Lorbus is the specific one that preserves MTP projection weights; a 32B variant that drops them runs at 17.9 tok/s. Check what your quantization actually includes.

tested Fast on llama.cpp, dead on vLLM. Single GPU ceiling 71.1 tok/s; dual GPU 107.23 tok/s. SWA architecture makes context essentially free — 25 of 30 layers use a fixed 5,120-token ring buffer regardless of window size. NVFP4 (30.6GB) won't fit on 2×16GB with KV headroom. AWQ: 32.77 tok/s, no efficient Blackwell kernels.

flat wall · ecosystem gap New model, no native engine support yet. 30.9 tok/s, completely flat across 19 tuning experiments — pure VRAM-bandwidth bound. Neither vLLM 0.21.0 nor SGLang 0.5.12 had native support at release; both fall back to a Transformers wrapper whose overhead pushes fp8 loading over the limit by 114 MiB. Lesson: test llama.cpp first on any new model release, it always works; revisit the faster backends after their next release ships support.

new architecture Not autoregressive — a genuinely different generation model. Iterative canvas denoising instead of token-by-token generation. 41 tok/s baseline, 121 tok/s at longer output lengths (per-step cost is nearly constant regardless of canvas size). Requires an unmerged llama.cpp PR. Best flag: --diffusion-kv-cache on — disabled by default on 2-GPU splits, forcing it on gives +17% with fewer steps.

17.9 tok/s Bigger, slower, wrong. The kaitchup quant doesn't include MTP heads — one token per pass with 32B parameters instead of two with 27B. 4.5x slower than Genesis for a "bigger" model.

tested · ceiling found 100.24 tok/s, 22 experiments, zero improvement. MoE expert routing over PCIe is fundamentally sequential — this is a PCIe-vs-NVLink architectural boundary, not a tuning gap. Long-context variant degrades hard with depth: 32K at 71.6 tok/s down to 39.7 at 131K.

VRAM wall A hardware problem, not a config problem. 55.6 tok/s baseline, unoptimizable — sits right at the 32GB VRAM ceiling. All 10 tuning experiments that touched context or KV dtype triggered OOM. Needs more VRAM or a quantization below Q4.

tested Fast, one atypical win. 95.9 → 97.07 tok/s. q5_0 KV cache beat both q4_0 and f16 — an unusual result confirming KV dtype tuning is architecture-dependent, not universally monotonic.

universal Works everywhere. Has limits. The default for consumer hardware — excellent for MoE and Mamba/SSM models (Nemotron hit 117.6 tok/s here). Hard limit: no optimized CUDA kernel for GDN/DeltaNet hybrid layers. On SM_120, always build cuda128-clean, never cuda13 — the latter crashes on the first generation call after prompt eval, every time.

required for GDN + NVFP4 Fast when it works, fragile when it doesn't. The only backend with kernel support for GDN hybrid layers and NVFP4A16. With the right patches: Genesis at 83–88 tok/s, Ornith NVFP4 at ~101–124 tok/s. Pin your version — 0.20.x lacks SM_120 TP=2 support and crashes with an NCCL error at multi-GPU init.

safe default Works everywhere. Architecture match matters more than bit count. Q4_K_M is the GPU sweet spot — fits more model in VRAM. Only q4_0 and f16 KV cache have fast CUDA paths on SM_120; q8_0, q5_0, iq4_nl all degrade 15–30%.

winner · check what it includes Same format, wildly different outcomes. The Lorbus quant of Qwen3.6-27B preserves MTP projection weights (280MB). The kaitchup quant of Qwen3-32B doesn't. 83 tok/s vs 17.9 tok/s on the same format name.

production (as NVFP4A16) Went from a dead end to the production format. Standard NVFP4 only quantizes MLP layers, attention stays BF16 — the loaded model is bigger than expected (Gemma 4 NVFP4: 30.6GB, won't fit). But NVFP4A16 (FP4 experts, BF16 attention) with the SM_120 Marlin patch is exactly what's running production today at ~101–124 tok/s with a 131K context window.

no Blackwell kernels Dead on this hardware. No efficient kernel paths on RTX 5060 Ti. Gemma 4 AWQ: 32.77 tok/s vs 100+ for the winning stacks. The format itself is the bottleneck — not necessarily a bad format, just the wrong one for Blackwell.

public Every experiment behind this site — raw data, per-run tables, and the autoresearch loop scripts themselves.

source of truth The complete benchmark record this site draws from — every campaign, every delta, every quality eval, in full table form.