Model Selection: Choosing the Right Brain on Evidence

Lesson 2: Model Selection — Choosing the Right Brain on Evidence

Every week a new model drops with a chart showing it winning some leaderboard, and teams rewrite their stack in a panic. Model selection should not be a vibe — it should be a measured decision driven by the same eval loop from Lesson 1: run candidates against your data, score them with your rubric, and compare on your cost and latency budget. This lesson gives you the framework to do that.

Benchmarks: What They Measure and Where They Lie

Public benchmarks are useful for one thing: a rough, up-to-date sense of model capability order. They are useless for telling you which model is best for your application, because your application is not MMLU. Know what each benchmark actually measures:

Benchmark What it measures Blind spot
MMLU / MMLU-Pro Broad knowledge + reasoning across 57 subjects (multiple choice) Multiple-choice format favors models tuned for it; doesn't test free-form generation, tool use, or instruction following under ambiguity
GPQA Graduate-level science questions (PhD-expert level) Narrow; not representative of most business workloads
HumanEval / MBPP Code generation from docstrings Short, self-contained functions; misses real-world multi-file engineering and debugging
SWE-bench Real GitHub issues → patch generation (agentic coding) Tests the whole agent pipeline, not the raw model; setup-heavy
LMArena (Chatbot Arena) Human preference via blind pairwise votes, Elo-ranked Measures general chat preference, not task correctness; favors style and length
MT-Bench Multi-turn instruction-following quality, judged by GPT-4 Judge-model bias (see Lesson 4); generic tasks
Rule of thumb: Use leaderboards to build a shortlist (top 3–5 candidates for your task class), then let your own golden dataset pick the winner. Never let a leaderboard pick the winner directly.

The Selection Triangle: Quality, Latency, Cost

Quality Latency Cost Bigger models sit near Quality; smaller models trade down to win Cost/Latency.

Frontier models (GPT-5-class, Claude-opus-class, Gemini-class) win on quality but cost more per token and respond slower. Compact models (fast variants, 7–70B open models, distilled models) are dramatically cheaper and faster, and with the right prompting, retrieval, and guardrails they often clear the quality bar for narrow tasks. The job of your eval harness is to find out where the bar sits and which candidate clears it at the best price.

Other Axes That Matter as Much as Score

  • Structured output reliability — how often the model returns valid JSON/schema you asked for. For agentic systems this can matter more than raw reasoning (malformed tool calls kill automation).
  • Tool calling / function calling — correctness of argument selection, not just the model's prose.
  • Context window and long-context behavior — advertised size vs. actual retrieval of facts buried at position 90k ("lost in the middle").
  • Instruction following under system prompts — obedience to formatting rules and guardrail instructions, and resistance to prompt injection (a small, locked-down model can beat a frontier model on injection resistance).
  • Portability / vendor strategy — can you swap providers without rewriting? A multi-provider wrapper (LangChain, OpenRouter, Azure OpenAI) makes the model a config value, not a dependency.
  • Multilingual and domain coverage — benchmarks are English-heavy; if your users aren't, you need your own data to know.

The Evaluate-Before-You-Commit Workflow

Concretely, model selection is just your eval loop applied to candidates:

# Pseudocode — the real thing in Lesson 6 (LangSmith)
for model in shortlist:
    score = run_evals(dataset=golden_set, evaluator=rubric, model=model)
    print(model, score, latency_p50, cost_per_1k_queries)
# Winner = best score that fits your budget
# Runner-up = your fallback for failover routing
  1. Build a small golden set first (Lesson 3) — even 30–50 representative examples beats zero.
  2. Run every candidate through the same dataset with the same rubric and temperature.
  3. Record the full matrix: score, p50/p95 latency, cost per query, structured-output failure rate.
  4. Pick a primary and a fallback — the fallback matters for failover and for A/B testing later.
  5. Re-run the comparison when the shortlist changes — model releases are a recurring event, so make the comparison script a standing artifact (this is literally a cron job or a CI job).
Cheap trick: run your hardest 10 examples first. If a model can't clear your adversarial set, it doesn't matter how good it looks on MMLU — save the tokens and drop it before the full run.
Pitfall: benchmarking at temperature 0 with clean inputs tells you nothing about real-world variance. Sample a few runs per example (e.g. 3–5) so you can measure stability, not just average quality. A model that's great on average but flips between great and terrible is a production hazard.

🧠 Knowledge Check

1. What is the correct role of public leaderboards in model selection?

2. Which capability can matter MORE than raw reasoning for agentic applications?

3. Why should you sample multiple runs per eval example instead of a single run?

Further Reading