Getting Started

Not a tutorial. The decisions you need to make.

This isn't a step-by-step guide — you can find those everywhere. This is the decision guide: what you need to figure out before you start clicking, and what actually matters. Every claim below is backed by a dated experiment on the flowchart, with raw logs in inference-research on GitHub.

Decision 1: Hardware

The only thing that matters is VRAM. Period. Everything else is secondary.

Under 12GB VRAM: You're running small models — 7B–8B dense, quantized, llama.cpp backend. We don't have GPU benchmarks at this tier; our CPU-only baseline ran at 10.44 tok/s, but a GPU changes the picture significantly.

16GB VRAM: You can fit a 27–35B model at INT4/NVFP4 — this is our production tier. Dual 16GB cards (32GB total, no NVLink) run Qwen3.6-27B INT4 (Genesis) in production at ~97 tok/s warm; Ornith-1.0-35B NVFP4A16 is the high-context alternative at ~101–124 tok/s with a 131K window.

16–24GB VRAM: MoE models with small active parameter counts are the best value — 30B+ total, only 3B active per token (Nemotron 3 Nano hit 117.6 tok/s here).

24GB+ VRAM: More choices, higher ceiling — the rules stay the same. 70B dense models need real headroom: ours OOM'd at the 32GB ceiling every time.

RAM matters too, but mainly for CPU offloading. If a model doesn't fit in VRAM, layers move to CPU RAM and pay a PCIe penalty on every token — that's exactly what happened in our first tower experiment: 16GB of VRAM, only 3.7GB in use, because expert tensors were routing through CPU RAM.

CPU matters most for llama.cpp; vLLM leaves the CPU mostly idle during decode. For llama.cpp, get E-cores — they're load-bearing for MoE expert dispatch on Alder Lake and newer. Restricting to P-cores only dropped our CPU baseline from 10.44 to 7.73 tok/s.

Don't buy a GPU for local inference without checking kernel support first. Our RTX 5060 Ti (Blackwell SM_120) needs the cuda128-clean llama.cpp build target — the cuda13 build crashes with a CUDA MMQ kernel error on the very first generation, on every model we tried. This is a build-target issue, not a CUDA toolkit version issue — the system runs CUDA 13.0.3 fine day to day. No flag fixes a missing or mismatched kernel; check before you buy.

Decision 2: Backend

Two real options. Not ten. Two.

llama.cpp — The universal backend. CPU, GPU, anything with GGUF support. No optimized CUDA kernels for some architectures: GDN/DeltaNet hybrid layers ran at 22 tok/s here because the SSM state updates had no fast path. No flag fixes a missing kernel. Good for: CPU inference, MoE and Mamba/SSM models, experimentation with brand-new model releases (engine support always lands here first).

vLLM — The performance backend. GPU-only. Has optimized kernels llama.cpp doesn't, plus native NVFP4 and speculative decoding support. The same GDN model that ran 22 tok/s in llama.cpp hit 83+ tok/s in vLLM with the right patches. Good for: anything with CUDA kernel support, anything where speed or long context matters.

The rule: if the model has optimized CUDA kernels in vLLM, use vLLM. If it doesn't, or you're on CPU, use llama.cpp. That's the whole decision.

Decision 3: Model

This is the hardest decision. The model that's fastest on one hardware setup might be slowest on another. The model that works for chat might fail for tool calling. The model that tops benchmarks might produce malformed JSON a third of the time.

The real constraint is VRAM. Calculate the model size at your chosen quantization, add KV cache overhead, and check if it fits. If it doesn't fit, nothing else about it matters.

For 32GB total (2×16GB) VRAM: Qwen3.6-27B AutoRound INT4 (Genesis) is what we run in production — ~97 tok/s warm, ~28GB, MTP speculative decoding, a safe quality floor. Ornith-1.0-35B NVFP4A16 is the alternative when you want a 131K context window and can take the variance — ~23GB, ~101–124 tok/s. The specific quantization matters as much as the model: one AutoRound quant of Qwen3.6-27B preserves the MTP speculative-decoding weights; another drops them and runs 4.5× slower.

For 12GB or less: 7B–8B models, GGUF format, llama.cpp backend. Untested at this tier here — pick one that fits your VRAM and start there.

For 16–24GB: MoE models become practical. Qwen3-35B-A3B and GLM-4.7-Flash are both 30B+ total with only ~3B active per token — we measured 100.24 and 97.07 tok/s respectively on 32GB total (2×16GB cards). A single 24GB card is untested territory here.

Decision 4: Quantization

Not all quantizations are equal. The format matters more than the bit count.

GGUF (Q4_K_M, Q5_K_M, IQ4_XS) — llama.cpp native, works everywhere. On Blackwell SM_120, only q4_0 and f16 KV cache have fast CUDA paths — others degrade 15–30%.

AutoRound INT4 — Needs vLLM with Marlin kernel support. Check what the specific quant preserves — MTP speculative-decoding heads matter more than the bit count. One quant that keeps them beats one that doesn't by 4.5×.

NVFP4 / NVFP4A16 — Nvidia's native Blackwell format. Standard NVFP4 only quantizes MLP layers and keeps attention in BF16, so loaded size is bigger than expected (a Gemma 4 NVFP4 build didn't fit on 2×16GB). NVFP4A16 (FP4 experts, BF16 attention, with an SM_120 Marlin patch) ran our production trial through July 3; Genesis (Qwen3.6-27B GPTQ INT4) is the current production backend.

AWQ — No efficient kernel paths on Blackwell. 32.77 tok/s here vs 100+ for the winning stacks. The format itself is the bottleneck on this hardware.

Decision 5: What You Actually Need It For

Speed isn't the only metric. It's the easiest one to measure, but it's not the only one that matters.

Running agents that need tool calling? Head-to-head testing here found a local 35B model beating a much larger cloud model 20/20 vs 11/20 on sequential single-tool tasks — but losing on multi-hop reasoning chains. Route by task shape, not model size.

Running an agent that needs to remember things? Check memory-recall performance specifically, not just raw speed — one fast model here actively regressed a memory-benchmark score compared to a slower one. Fast doesn't mean right for your workload.

Just want it to run? Pick the model that fits your VRAM, use the backend that has kernels for it, and don't overthink it. The experiments will tell you what actually works.

The Shortest Path

If you want the fastest path from zero to something useful: get a GPU with 16GB+ VRAM, install llama.cpp, download a GGUF model that fits, and run it. See how fast it goes. If it's slow, figure out why — is the model mostly on CPU? Are the kernels missing? Is the architecture unsupported? Answer those questions and you'll know what to do next.

The alternative — buying expensive hardware and spending weeks tuning blind — is what most people do. It's not what you need to do.