The Evaluator Toolbox: Scoring Techniques

Lesson 4: The Evaluator Toolbox — Scoring Techniques

You have the data (Lesson 3). Now you need the scoring instruments. An evaluator is any function that takes an output (and optionally the input, reference answer, and context) and returns a score. The toolbox ranges from dead-simple deterministic checks to LLM judges — and the skill is knowing which instrument fits which question. Use the cheapest instrument that answers the question reliably.

The Instrument Hierarchy (Cheapest First)

Instrument How it works Best for Weakness
Exact / fuzzy match Output equals (or contains / matches regex of) the expected string Classification, entity extraction, IDs, one-word answers, formatting compliance Brittle for anything generative
Structured checks Validate JSON schema, required fields, enum values, tool-call arguments, code compiles/executes Agent tool calls, structured output, pipelines Only validates shape, not meaning
Reference similarity Embedding or token-overlap similarity between output and a golden answer Summarization, paraphrase tasks with a reference Similarity ≠ correctness; misses subtle factual errors
Rubric / LLM-as-judge (pointwise) Judge model scores output 1–5 against a written rubric with criteria and examples Open-ended quality, helpfulness, tone, safety, faithfulness to given context Judge bias; needs a good rubric; costs tokens
Pairwise comparison Judge model (or human) picks the better of two outputs (A/B) Model selection, prompt A/B tests, tuning Gives ordering, not absolute quality; position bias
Custom programmatic Assertions about behavior: did the agent call the right tool? Did the retriever return the right doc? Is the number within tolerance? Anything measurable in code — the most underrated evaluator You have to write it
Rule: If a regex can check it, don't pay for a judge. Reserve LLM judges for the semantics that only language models can grade — and always pair a deterministic gate with a judge, not a judge alone.

LLM-as-Judge: Powerful, Biased, and Controllable

The "LLM-as-a-Judge" paradigm (popularized by the MT-Bench and Chatbot Arena work at UC Berkeley and LMSYS) uses a strong model to grade outputs. It scales to hundreds of examples, is surprisingly correlated with human judgment, and is the workhorse of modern eval frameworks. But it has documented biases you must engineer around:

  • Position bias — judges favor the first (or last) answer shown in comparisons. Mitigation: swap order and average, or randomize.
  • Verbosity bias — longer, fancier answers score higher even when worse. Mitigation: rubrics that penalize fluff, length caps in the judge prompt.
  • Self-preference bias — judges favor their own family of models. Mitigation: use a judge from a different model family than the candidate, or use the strongest model you can afford as judge.
  • Reasoning limitation — weak judges make reasoning errors. Mitigation: use a strong judge; some frameworks let the judge "think" (chain-of-thought) before scoring, which improves agreement with humans.
# Rubric quality is the #1 lever on judge reliability.
# Bad:  "Is this answer good? Rate 1-5."
# Good: "Score 1-5 on: (1) factual accuracy vs the reference,
#        (2) does it answer the user's exact question,
#        (3) is it grounded in the provided context.
#        A 5 requires ALL criteria met. Penalize unsupported claims.
#        Example of a 5: ...  Example of a 1: ..."

The rubric turns the judge from a vibes meter into an instrument. Concrete criteria plus anchor examples (what a 5 looks like, what a 1 looks like) are worth more than any judge-model upgrade.

RAG-Specific Metrics: Score the Pipeline, Not Just the Answer

If you're building retrieval-augmented generation, answer quality is downstream of retrieval quality. The RAGAS framework popularized a metric family that scores each link in the chain:

  • Faithfulness — is every claim in the answer supported by the retrieved context? (Hallucination detector.)
  • Answer relevancy — does the answer actually address the question?
  • Context precision — of the retrieved chunks, how many were relevant? (Noise in = bad answers out.)
  • Context recall — did retrieval bring back everything needed to answer? (Misses here are silent quality killers.)

Debugging pattern: if faithfulness is high but answer relevancy is low, the generator is fine and the retriever/query is the problem. If context recall is low, your chunks or embeddings are missing content. Score the pipeline, and you'll know where to fix it.

Aggregating: From Per-Example Scores to a Gate

Per-example scores become decisions through aggregation. The standard toolkit:

  • Pass rate — % of examples above a per-example threshold (e.g. judge score ≥ 4). Robust and interpretable.
  • Mean / median score — sensitive but hides tail failures; always report with pass rate.
  • Worst-k / p5 — the worst 5% of examples. In safety-critical apps this matters more than the mean.
  • Category breakdown — score per taxonomy bucket (happy path vs adversarial vs edge). A flat mean can hide an adversarial-bucket collapse.
  • Delta vs baseline — the regression signal. Lesson 9 turns this into a gate; Lesson 10 makes it a trend.
Warning: An aggregate mean is a lie when the tail is dangerous. Always gate on the buckets that matter (zero failures on adversarial, pass rate on happy path) and only then look at the mean.

🧠 Knowledge Check

1. Which evaluator should you prefer when a regex can check the property?

2. Which is a documented bias of LLM-as-judge?

3. In RAGAS-style metrics, which metric specifically detects hallucination (claims not supported by retrieved context)?

Further Reading