The Complete Inference Flowchart

Q5_K_MMoE · 8 active experts Port :8081. mlock required — without it the model pages out entirely. Key finding: E-cores are load-bearing for MoE expert dispatch — restricting to P-cores only dropped throughput to 7.73 tok/s. Optimal: 8 threads, cpu_mask 0x3F5 (all E+P). Chronic swap: 4.6–7.7 GiB used across a 17-day sample period (Apr 2–18). This ran every agent in the house before the tower existed.

SuperGemma4-26B, same model family as the Beelink era. GPU mostly idle — expert tensors still routing through CPU RAM across PCIe on every token. Only 3.7 GB of 16 GB GDDR7 in use. The hardware was there; the config wasn't.

Expert tensors moved to GPU (only the last 6 of 30 layers stayed on CPU). Then: the Gemma-4 SWA architecture discovery — 25 of 30 layers use a fixed 5,120-token ring buffer regardless of context size. Context window turned out to be essentially free. Restored to 32K ctx at q8 KV. Final: 69.8 tok/s at full context, full quality.

15 experiments. Real gains from ubatch 512, dropping CPU layers 6→4, threads→4. Everything else was noise or regression. PCIe bandwidth is the hard ceiling on a single card.

All 30 expert layers now fit on GPU across both cards. 12 GB headroom freed up — KV cache upgraded to f16, zero dequant overhead. 20 experiments, two real wins (KV f16 +7.17, ubatch 4096 +0.86), then a hard wall. This is the llama.cpp peak for this model family on this hardware.

35B-A3B MoE 100.24 tok/s — 22 experiments, zero improvements. Already at the hardware ceiling: MoE expert routing over PCIe is fundamentally sequential, and no config change removes that bottleneck. This is a PCIe-vs-NVLink architectural finding, not a tuning gap.
70B VRAM cliff 55.6 tok/s — Llama 3.3 70B Q4_0 sits right at the 32 GB ceiling. All 10 follow-up experiments that touched context or KV dtype triggered OOM.
GLM-4.7-Flash 95.90 → 97.07 tok/s — one real gain found: q5_0 KV cache beat both q4_0 and f16, an atypical result confirming KV dtype tuning is architecture-dependent, not universally monotonic.

No CUDA kernel GDN/DeltaNet hybrid attention layers have no optimized CUDA path in llama.cpp — SSM state updates fall back to unoptimized code. No config change fixes a missing kernel. The whole inference stack had to change.

AutoRound INT4 the Lorbus quant preserves 280 MB of MTP projection weights — drop them and the speculative head has nothing to predict from.
vLLM Genesis patches TurboQuant hybrid gate + 19 downstream fixes make GDN layers work on vLLM at all.
MTP n=3 self-speculative decoding — +87% throughput lift on its own. It's the whole game.
VLLM_MARLIN_USE_ATOMIC_ADD=1: +6.25 tok/s, undocumented. Peak measured: 86.83 tok/s.

Same model in Nvidia's native FP4 format. 122K context window, real but rarely needed. NVFP4 is already near hardware-optimal at this quantization level — 15 tok/s below Genesis with almost no optimization headroom left to find.

Tower moved from WiFi to a direct point-to-point wire (0.76ms ping). While the session was offline during the swap, the local coding agent recognized vLLM was now bound to 0.0.0.0 on a direct wire — and changed it to 127.0.0.1 without being asked. Also: an MESA benchmark run showed AEON regressed Mike's memory-recall score by 0.10–0.16. Wrong model for that job, independent of raw speed.

AEON: 38% structured-JSON failure rate on LongMemEval tasks. Genesis: clean. Proxy switched over; AEON stopped.

Both Genesis and AEON auto-started on boot, both claimed VRAM, neither got enough. 32 GB total, each needs ~14–15 GB. AEON disabled from auto-start permanently — one model at a time on this hardware, no exceptions.

Gemma 4 NVFP4 30.6 GB loaded — doesn't fit. NVFP4 keeps attention in BF16, so the loaded size is bigger than the bit-count suggests.
Gemma 4 AWQ 32.77 tok/s — no efficient Blackwell kernels for AWQ.
Qwen3-32B (kaitchup quant) 17.9 tok/s — no MTP heads in this quant. One token per pass, 32B parameters, 4.5× slower than Genesis.

VLLM_MARLIN_USE_ATOMIC_ADD=1 alone: +6.25 tok/s — atomic-add reduce in the gptq_marlin kernel for small-n decode on TP=2. One env var, zero model change. KV dtype=auto (fp16) eliminated per-layer dequant overhead for a further +4.95 in Pass 1. Of 13 follow-up variable sweeps (NCCL buffer size, OMP threads, PyTorch split size, stacked combos), only two produced marginal gains beyond the headline two — everything else was noise.

Upgrade attempted after confirming DFlash draft-model speculation doesn't apply to Genesis (self-MTP, not an external draft model). 0.20.2 crashed with an NCCL error at multi-GPU init — SM_120 TP=2 was unsupported in 0.20.x. Rolled back to 0.19.2rc1 via tarball restore. Complication: a same-day kernel upgrade to 6.17.0-23 locked out the NVIDIA driver (modules only built for -22) — fixed by pinning GRUB default. Re-optimization from the 13% regression was still pending as of this incident.

Reducing CPU layers 10→4, threads 8→4, parallel=2, pinned to E-core affinity mask 0x3F5 — extracted the maximum from Alder Lake's heterogeneous core topology. SWA cache contention was the primary bottleneck; eliminating CPU-side routing resolved it entirely.

fp8 KV cache enables a full 128K context window on 32 GB VRAM with under 1% throughput degradation (−0.8%). KV footprint scales linearly with context; fp8 halves the cost vs bf16. The single most cost-effective capability expansion found in the entire research program — 4× context for almost nothing.

Dense 32B GPTQ: 5.25 tok/s. MoE 30B-A3B GPTQ: 41.33 tok/s. Both models are 16 GB on disk; both need --cpu-offload-gb 1.0 to initialize — but throughput differs 8×. Dense accesses all 32B weights per token and saturates the PCIe bus; MoE only touches ~3B active params, so PCIe pressure is ~10× lower. NVFP4 was blocked all along by FlashInfer's CUTLASS SM_120 kernels needing CUDA ≥12.9 — this system ran 12.8 at the time. (Cleared 2026-06-06 with the CUDA 13.0.3 upgrade.)

MTP n=2 is the ceiling for this model — n=3 gains only +0.35 vs baseline, likely because the third speculative token is rejected too often to pay for itself. No-MTP sanity check confirms MTP contributes ~8.5 tok/s to the baseline. flash-attn failed to load entirely — timed out at 120s, consistent with SSM layers conflicting with the standard flash-attn kernel path.

Mamba/SSM hybrid MoE — 30B total params, only ~3B active per token. Replaces Genesis as production default. The cuda13 llama.cpp build crashes with ggml_cuda_mul_mat_q: invalid argument on the first generation after prompt eval — every time, on SM_120. This is a Blackwell + cuda13 MMQ kernel incompatibility, not a model issue. cuda128-clean is the build target that works. 24.9 GB VRAM (11.8 + 13.1 GB), prefill 1,200–2,500 tok/s, 4/4 smoke tests pass.

Judged by Claude Haiku against per-probe rubrics. Reasoning 5.00/5, factuality 5.00/5, instruction-following 5.00/5. Coding and agent-tasks both 4.60/5 — the only misses are partial-credit (one docstring quality issue, one 512-token truncation), zero outright failures across all 25 probes.

ctx=65536 (2× baseline) matches baseline throughput while doubling available context — a free win, deployed to production. threads=16 caused severe instability (stddev 30, worst 45 tok/s) from Mamba's sequential state dependency conflicting with hyperthread contention. flash-attn has zero effect on this attention-free SSM decode path. Model is at ceiling for this hardware tier.

TensorRT-LLM 1.2.1, official NVIDIA ep/bmm sharding path. Standard TP=2 fails outright — Mamba/SSM layers can't be tensor-parallelised, they replicate on each GPU instead of splitting. The correct sharding approach loads 14.47 GB of 15.47 GB, leaving 17 MB headroom. TRT-LLM needs 20 MB to finish initializing. 3 MB short. No configuration fix exists. Needs ≥20 GB VRAM per card.

25 experiments, reference baseline only — not tower production. -ngl 99 --flash-attn on -t 1 is the winning config: full GPU offload with t=1, since extra CPU threads just fight the unified memory bus. MLX rejected — 3× slower prefill than llama.cpp Metal despite being Apple-native. Metal tensor API is disabled on pre-M5 hardware, capping generation near 21 tok/s regardless of tuning. Sustained benchmarking on a fanless chassis degrades prefill 24% over 20 minutes (thermal throttle) — first run of the day is meaningfully faster than sustained use.

Dense 12B at Q8_0 is pure VRAM-bandwidth bound — 19 autoresearch experiments across split modes, threads, ubatch, and KV quant all landed within ±0.2 tok/s of baseline. Neither vLLM 0.21.0 nor SGLang 0.5.12 had native Gemma4UnifiedForConditionalGeneration support yet — both fell back to the generic Transformers wrapper, which carries enough overhead to push fp8 loading over the VRAM limit by exactly 114 MiB. New model evaluation methodology takeaway: test llama.cpp first, always works; revisit vLLM/SGLang after their next release.

Three blockers cleared to bring Genesis back online alongside Nemotron: (1) flash_attn wasn't pulled as a vLLM dependency on CUDA 13.0 builds — installed 2.8.3 with CUDA 13.0 headers explicitly. (2) A co-resident service's Restart=always kept re-seizing the inference port within seconds of every Genesis stop — fixed with a Restart=no systemd drop-in. (3) TimeoutStartSec was missing — the default 90s killed the process mid-load; set to 300s. Hardened further with fp8 KV (65K context), the qwen3_coder tool-call parser (fixes a silent tool-drop bug), and a FlashInfer workspace resize (413→256 MiB) that cleared an OOM on first MTP inference. Quality: 89.6% on a 29-test suite (26/29), 100% instruction-following, 100% tool-calling.

Google DeepMind's block-diffusion model: iterative canvas denoising over 256-token blocks in parallel, not autoregressive token-by-token generation. Required an unmerged llama.cpp PR (#24423) — mainline returns unknown model architecture: diffusion-gemma. Best single-flag finding of the whole program: --diffusion-kv-cache on is auto-disabled by default on 2-GPU splits — forcing it on gives +17% throughput and fewer denoising steps (25→22) with no quality loss observed. Per-step cost is nearly constant regardless of canvas size, so longer outputs are proportionally cheaper: n=1024 hit 121 tok/s vs n=256's 41 — 2.5× throughput for 4× the output.

35B GGUF, served locally on dual RTX 5060 Ti, vs DeepSeek V3.2 over OpenRouter — same hardware, same day, same prompts, temperature 0, 66 total cases. Overall near-identical: 84.8% local vs 86.4% cloud, at ~3s vs ~22s latency. But the profiles are inverted: Ornith goes 20/20 vs DeepSeek's 11/20 on single-tool sequential typewriter tasks — the exact task where GPT-4 famously failed in LangChain's original 2023 blog post. DeepSeek looped to the 30-turn cap on 9 of 20 cases, unable to recognize when to stop calling the same function. DeepSeek wins multi-hop reasoning (multiverse math 18/18 vs 10/18, relational data 8/8 vs 6/8) — Ornith sometimes short-circuits to a pretrained answer instead of chaining tool calls on 3+ hop problems.

GGUF Q4_K_M, layer-split across both GPUs — exceeds Nemotron's 117.6 tok/s record. Context scaling shows an 8K–16K trough (21–23 tok/s) with partial recovery at 32K (46 tok/s), likely a KV scheduler boundary interacting with GatedDeltaNet attention. HumanEval measured artificially low at 43.9% (72/164) — markdown code fences leaking into raw completions cause harness-level SyntaxError on otherwise-correct logic, not real reasoning failures.

GGUF Q4_K_M via llama.cpp, layer-split dual GPU, deployed to the fleet proxy at :8010. ~127–130 tok/s peak on the production path.

x86 port of AEON's DGX Spark serving recipe onto the SM_120 tower. AEON's original guide assumes BF16 KV (its vision tower forces it), which capped context at ~32K here. Adding --language-model-only --kv-cache-dtype fp8 restores the full 131K context window with a 338K KV token budget. NVFP4A16 format: FP4 quantization on experts/MLP, BF16 on attention. Kernels: Marlin linear + FlashInfer MoE dispatch + a local SM_120 Marlin compatibility patch. This is the current production configuration.

Ornith's production trial ends. Genesis goes back to the default backend at :8010 — Qwen3.6-27B INT4, GPTQ-Marlin, MTP n=3 speculative decoding. Speed with a safe floor. The consult-for-the-hard-ones models drop to standby.

The ceiling on two 5060 Ti's isn't the cards — it's the interconnect and the loader. Consumer cards block GPU-to-GPU P2P, so every tensor-parallel all-reduce detours through host RAM: 20.1% of the decode budget, pure latency. PCIe barely moves during decode (~250 MB/s) — the cost is per-op latency, not bandwidth. Native FP4 tensor cores exist on SM_120 but only fire for W4A4; the higher-quality W4A16 quants eat a 1.7× emulation penalty. And the whole envelope rides on two switches most benchmarks never flip — MTP speculation (+59%) and CUDA graphs (+60–75%). Receipts.

Fifteen domain-expert scenarios — logistics, traffic, restaurants — scored blind by two judges. The NVFP4 W4A16 build beat the INT4 production model 4.13 to 3.73. The only model of three tested that didn't scramble the menu-engineering matrix, and the only one that refused to promise a severe-allergy client "guaranteed nut-free" from a shared kitchen. A third the speed. That's the trade — latency for judgment. Scorecard.

Full run of the MoE playbook: Qwen3-30B-A3B in three quant/runtime combos, Qwen3.6-35B-A3B, Nemotron-3-Nano-30B-A3B, and a freshly-acquired GPT-OSS-20B (native MXFP4). Every one of them beat Genesis on raw speed — GPT-OSS-20B hit 133.05 tok/s, the fastest clean number of the whole program. Every one of them lost to Genesis on quality, by 0.9–1.9 points on the 15-scenario domain suite — a gap dense-side quantization never got close to. All-GPU beat every offloaded config, every time; the offload lever only matters when a checkpoint doesn't fit, and none of these needed it. Worst of all: Qwen3-30B-A3B's fastest config (Q3_K_M, all-GPU, 121.5 tok/s) was also its worst-quality one (1.87/5) — speed and quality inverted, not just traded off.

A "yellow trap" signal conflict and a rail-crossing queue-spillback risk — both scenarios failed identically across three unrelated base-model families (Qwen, NVIDIA/Nemotron, OpenAI GPT-OSS), four quantizations, two runtimes. Same failure shape every time: a long, confident, well-organized answer that fabricates a plausible wrong mechanism and lands backwards on the one constraint that actually mattered. GPT-OSS-20B — the smallest model tested — failed hardest: flatly asserted the roundabout's rail-crossing interaction was "None (independent)," and separately told an operator to reheat and re-serve temperature-abused soup. Not a benchmark score. A real failure mode, and it doesn't show up on the dense side at this parameter count. Reads as a capability floor at 20–35B, not a per-model bug.

vLLM 0.25.0 loading Qwen3-30B-A3B-NVFP4 appeared to hang post-load — idle workers, zero GPU utilization, a shm_broadcast warning every 60s. First guess: TP/expert-parallel sync stall. Wrong — retested with production's own NCCL settings, still hung. Second guess: auto-enabled expert-parallel. Also wrong — the flag defaults off and was never set. The real cause, found by checking ps aux instead of treating it as a black box: 91 concurrent nvcc/cicc/ptxas processes — FlashInfer JIT-compiling a kernel with no precompiled build for this GPU+quant combo, uncapped, on an 8-core box. Drove real RAM/swap thrashing (30/30GB RAM, 87/95GB swap) — not a deadlock. One MAX_JOBS flag away from not happening.

First small-model test rig: Mac mini M4, 16GB unified memory, MLX instead of llama.cpp — Apple's own framework generally beats llama.cpp's Metal backend here. Ran the same 15-scenario domain suite against four candidates. Ornith-1.0-9B-4bit — the smallest model on the list — scored 3.00 and stays the live default, beating two 27B builds from Prism ML at 1.71 bpw (2.93) and 1.125 bpw (2.47). The 1-bit build was the fastest of the four at 20.33 tok/s — and the lowest quality. Speed and trustworthiness pointed opposite directions again. Scorecard.

Both Prism ML Bonsai builds fail the same two scenarios identically, regardless of bit-width: both recommend a roundabout at a rail crossing and miss that signals support rail preemption and roundabouts don't; both get stuck on identical Hours-of-Service clock math and run out of token budget mid-reasoning — no dispatch decision, ever, on either. More bits didn't help, which rules out ordinary compression damage. But it's not the shared parent model either: Genesis runs the exact same Qwen3.6-27B base, just quantized via ordinary GPTQ INT4 instead of Prism ML's ternary/binary method, and clears both traps cleanly. That isolates it — something about Prism ML's extreme low-bit compression is damaging a capability that survives normal quantization fine. A third scenario, an unconditional written zero-cross-contact guarantee for a severe allergy, looked like the same story at first — worse in the 2-bit build than the 1-bit one — but Genesis stumbles on a version of that one too, so it reads as a hard trap in the suite rather than a Bonsai-specific defect. Two out of three still held up.