The Stack
This is the exact production setup — not a reference config, not a demo. This is what every agent on the tower talks to today. Full history of how it got here, including everything that lost, is on the interactive flowchart.
Hardware
Total cost around $2,000 for the tower — case, both GPUs, CPU, memory. Not a workstation, not a server. A consumer PC that happens to run inference for a living.
Model
Not just "Ornith-35B" — the combination of the model, the NVFP4A16 quantization, the SM_120 Marlin compatibility patch, and the exact serving flags is what gets these numbers. This is an x86 port of AEON's DGX Spark serving recipe onto consumer Blackwell — AEON's original recipe assumes BF16 KV cache because its vision tower forces it, which caps context at ~32K here. Adding --language-model-only --kv-cache-dtype fp8 restores the full 131K window.
Software
vLLM 0.23 with TP=2 on SM_120 is stable with these patches applied — that stability took real work to reach. The path here runs through two full production models before it: llama.cpp had no CUDA kernel path for Qwen3.6-27B's GDN hybrid layers (22 tok/s, a hard wall), and Genesis proved vLLM could do 83+ tok/s once the right patches and quant landed. Ornith inherits that lineage on newer hardware-native quantization.
Proxy
Every agent, every client, everything talks to port 8010. Switching production models means changing the proxy target, not reconfiguring every client. The bind address matters — a local coding agent caught the security exposure autonomously when the network topology changed from WiFi to a direct wire, and fixed it without being asked.
Numbers
Steady warm throughput: ~101 tok/s. Peak on code/JSON tasks: 123–124 tok/s. Peak ever recorded on this hardware (GGUF, short context, before the NVFP4 upgrade): ~130 tok/s. Context window: 131K tokens in production, the full window this model supports on this hardware.
What Didn't Make It
Three full production models and a dozen serious challengers lost on this exact hardware before Ornith won. The short version:
AEON (Qwen3.6-27B NVFP4): 68.86 tok/s, 13 experiments, zero improvement — already at its ceiling, 15 below Genesis, and 38% structured-JSON failure rate on production tasks.
Gemma 4 (multiple formats): NVFP4 too large to fit (30.6GB), AWQ has no efficient Blackwell kernels (32.77 tok/s).
Qwen3-32B: 17.9 tok/s — the available quantization dropped the MTP heads. One token per pass, 32B parameters, 4.5× slower than the model it was supposed to replace.
Llama 3.3 70B: 55.6 tok/s and unoptimizable — sits right at the 32GB VRAM ceiling, every tuning attempt triggers OOM.