Lesson 8: Choosing for Your Machine

Lesson 8: Choosing for Your Machine

Time to make it personal. Here's the exact procedure I'd run to pick a local LLM — using Qwen3.8-27B as the running example.

Step 1 — Inventory your hardware

• GPU VRAM (the king): 8, 12, 16, 24, 32, 48 GB…
• System RAM: matters for CPU offload and for CPU-only inference.
• Apple Silicon unified memory: Macs share one pool — treat it like VRAM.
• CPU/GPU vendor: NVIDIA (everything works), AMD (ROCm — GGUF fine, some formats less), Apple (MLX/Metal), Intel (GGUF, slower).

Step 2 — Budget memory: file + context + overhead

VRAM needed ≈ model file + KV cache + ~1-2 GB overhead. Rule of thumb: multiply the file size by 1.15-1.35 for a safe context budget. Qwen3.8-27B's hybrid architecture (Gated DeltaNet linear attention) keeps its KV cache unusually small, but long contexts and thinking-mode outputs still eat memory. Bigger context = more KV, always.

Context window: the silent VRAM eater

The context window is the maximum number of tokens the model can see at once — Qwen3.8-27B ships with 262,144 tokens natively, extensible to ~1M. But context is not free: every token of history must be stored in the KV cache (the key/value activations attention needs), and that cache lives in VRAM for the whole session.

The math: KV ≈ 2 (K and V) × attention layers × KV heads × head dim × 2 bytes, per token. Qwen3.8-27B is a hybrid — only 16 of its 64 layers use classic attention (the rest are linear-attention DeltaNet layers with a fixed-size state, no growing cache). That works out to ~64 KB per token, about 4× cheaper than a same-size pure transformer:

What this means when choosing:

• Match context to your longest real task — not the model's maximum. Paying KV for 128K you never use is wasted VRAM that could have bought a higher quant.
• Cap it in your runtime: set num_ctx in Ollama or -c in llama.cpp to what you actually need — many tools default to 4K-8K context even when the model supports more, which is often exactly right.
• Context is a dial you trade against quant: same GPU, same model — want 128K context? Drop Q4_K_M to Q4_K_S or an IQ quant to free the 6-8 GB. Want max quality? Trim context instead.
• Thinking mode and agents burn context fast: reasoning tokens count against the same budget; Qwen recommends a 262K reasoning allowance inside the 1M window for agentic work. Long-horizon tasks are the #1 reason to buy more context.
• Native beats extended: models can stretch beyond native context via RoPE scaling (YaRN/NTK — the -1M variants), but quality degrades as you stretch. Prefer a model whose native context covers the task.
• KV can be quantized too: llama.cpp can store the cache at 8-bit or 4-bit (-ctk q8_0 -ctv q4_0), roughly halving or quartering the numbers above for a small quality cost — handy when context is the bottleneck.

Step 3 — Pick the biggest model that fits, then the quant

Step 4 — Pick the ecosystem (this decides the format)

Step 5 — Test before you commit

Run the same short prompt suite on the candidate: one math question, one coding task, one instruction-following task, one creative task. Watch for: gibberish or repetition (too-low quant), refusal loops (wrong fine-tune), slow token rates (too big), and context truncation (too small). Then ask: would a comfortably-fitting smaller model have felt better? — that's the question that ends most upgrade loops.

Further Reading

• Ollama
• LM Studio
• llama.cpp GitHub
• vLLM GitHub