The Production Gate: Making Evals a Hard Blocker

Lesson 9: The Production Gate — Making Evals a Hard Blocker

Everything so far is infrastructure. This lesson is where it becomes policy: the production gate. A gate is a rule that a change cannot reach users unless it passes. Without gates, your evals are a report people can ignore; with gates, they are the floor beneath every release. The goal is boring: bad changes become mechanically impossible, not merely discouraged.

The Gate Stack

Real production systems gate on more than eval scores. Layered:

Layer Gate Example rule
Quality Eval thresholds on the golden set Pass rate ≥ 90% on happy path; no bucket drops more than 2 points vs baseline
Safety Guardrail/adversarial set, zero-tolerance 100% of adversarial examples blocked; zero PII leaks; zero successful injections
Operations Latency and cost budgets p95 latency ≤ 2s; cost per query ≤ $0.02; token budget per run capped
Security Code-level checks (the boring ones) Dependency scan, secret scan, prompt files linted, schema-validated output
Why zero-tolerance for safety: quality gates use percentages because mediocre answers are a smooth spectrum. Safety gates use zero because a single leaked PII record or one successful injection is a binary, reportable failure. Different failure classes, different math.

Setting Thresholds You Can Defend

  1. Baseline first. Run the current production configuration through the dataset several times. Record the distribution, not just the mean — you need to know how noisy your metrics are (Lesson 2's multi-sample point again).
  2. Set the gate at baseline minus a small margin (e.g. pass rate −3 points), and tighten it as the system improves. A gate at exactly baseline will fail randomly on noise; a gate far below baseline admits regressions.
  3. Gate per bucket, not just overall. "Overall 90%" hides "adversarial 40%". Require pass rates per taxonomy bucket.
  4. Pin the dataset version in the gate definition. If the dataset changes, the threshold conversation restarts — otherwise you're moving the goalposts and the gate is meaningless.

Where Gates Live: CI and Runtime

# CI gate (pseudocode — the required check on every PR)
results = run_experiment(data="support-golden-v2", evaluators=[...])
checks = {
  "happy_pass_rate":   results.pass_rate("happy")   >= 0.90,
  "edge_pass_rate":    results.pass_rate("edge")    >= 0.75,
  "adversarial_zero":  results.failures("adversarial") == 0,
  "p95_latency":       results.p95_latency          <= 2.0,
  "cost_per_query":    results.avg_cost             <= 0.02,
  "no_regression":     results.delta_vs_baseline("overall") >= -0.03,
}
if not all(checks.values()):
    print(f"GATE FAILED: {[k for k,v in checks.items() if not v]}")
    sys.exit(1)   # PR blocked, merge impossible

That sys.exit(1) is the whole point of this lesson. It converts "we should probably evaluate this" into "this cannot ship, mechanically."

CI gates protect deploys before they happen. Runtime gates protect users while they happen:

  • Canary / percentage rollout — ship the new configuration to 5% of traffic, verify online-eval scores and error rates hold, then ramp to 25% → 100%. No change ever hits 100% of users on day one.
  • Feature flags with a kill switch — every model/prompt/guardrail change is behind a flag; flipping back is a config change, not an emergency deploy.
  • Online eval thresholds with alerting — production traces get scored live (Lesson 6/10); if the live pass rate or guardrail-fail rate crosses its threshold, page the on-call and prepare to roll back.
  • Rollback plan in writing — know which version/flags restore the previous behavior, and practice it. A gate you can't retreat through is a trap.
The maturity ladder: Level 0 — playground anecdotes. Level 1 — a dataset, run manually. Level 2 — evals in CI as a blocking check. Level 3 — gates + canary rollout. Level 4 — online evals and drift gates in production. Most teams die between Level 1 and 2 — they build the dataset and never make it a blocker. This lesson is the jump from 1 to 2, and Lessons 10 makes 3–4 real.

When the Gate Fails

The gate failing is not a crisis — it's the system working. The workflow: read the regression report (which examples failed, which evaluators dropped), fix the smallest thing that moves the needle (a knowledge doc, a prompt clause, a retrieval parameter, a guardrail rule), re-run the experiment, and only merge when green. Treat gate failures as free debugging information, not punishment. If a gate fails so often it's ignored, the threshold is wrong — fix the threshold, never ignore the gate.

🧠 Knowledge Check

1. Why is the adversarial bucket gated at zero failures while the happy path is gated at a percentage?

2. What is the correct first step in setting eval thresholds?

3. Which mechanism makes a gate actually hard (unenforceable to skip)?

Further Reading