Measuring Change: From One-Off Evals to a Living Loop

Lesson 10: Measuring Change — From One-Off Evals to a Living Loop

A single eval run is a photograph; a system needs a movie. Models get updated under you, providers deprecate endpoints, your knowledge base drifts, and user behavior changes — all of which silently degrade quality while your one-time eval set collects dust. This lesson is the final layer: measuring change over time, so quality is a trend you watch, not a certificate you once earned.

The Three Time Scales of Measurement

Scale Question Tooling
Per-change (minutes) Did THIS change regress anything? Experiments + baseline diff in CI (Lesson 6/9)
Per-release (days) Is the shipped version still scoring where it did? Re-run the golden set on the published configuration; canary cohort comparisons
Continuous (weeks) Is production drifting? Online evals on live traces, thresholds, alerting, dashboards

Online Evals: Scoring Every Live Conversation

In LangSmith, attach evaluators to production traces so every conversation gets a live score (correctness, groundedness, guardrail-pass, tool-appropriateness). In Copilot Studio, the Analytics tab gives you resolution rate, escalation rate, and CSAT as continuous signals. Either way the pattern is the same: score live traffic, aggregate, alert on the aggregate.

# Monitoring pseudocode — run on a schedule, alert on regression
today   = online_eval_scores(window="24h")
weekly  = online_eval_scores(window="7d")

if today.pass_rate < weekly.pass_rate - 0.05:
    alert("LIVE PASS RATE DIPPED 5+ pts vs weekly baseline")
    suggest_rollback()   # flip the feature flag back

Feedback Loops That Grow the Dataset

Your dataset is a living artifact. Three loops keep it fed:

  • Thumbs up/down and CSAT — the cheapest signal. A thumbs-down is a candidate test case; sample and confirm it's a real failure before adding it.
  • Annotation queues (LangSmith) / transcript mining (Copilot Studio) — humans review traced conversations; confirmed failures become new dataset examples. This is how your golden set learns about problems you didn't anticipate.
  • Escalations and tickets — every time a user gives up and talks to a human, that's a failure mode report. Route it into the taxonomy.
Dataset refresh rhythm: add new examples continuously (from the loops above), but validate the gates quarterly: refresh ~20% of the set with recent real data, re-check your thresholds against the baseline distribution, and re-run model selection (Lesson 2) on the refreshed set. Models improve; the model that won three months ago may no longer be the winner.

Drift: The Silent Quality Killer

Four kinds of drift to monitor, each with a different detector:

  • Input drift — users start asking different things (new product, new slang). Detect with embedding-distribution comparison of live inputs vs training/eval inputs; respond by adding coverage to the dataset.
  • Output drift — same inputs, different behavior. Detect by re-running a fixed sample of the golden set regularly and watching scores/length/style shift; often caused by silent model or provider updates.
  • Score drift — online eval pass rate trending down. Your alerting threshold fires; investigate (it's usually input or knowledge drift underneath).
  • Cost/latency drift — tokens per conversation creeping up, p95 latency climbing. Watch the operational side; unbounded consumption (LLM10) often shows up here first.

The Dashboard You Actually Need

Keep it small enough to read in ten seconds:

  • Golden-set pass rate over time, by bucket (happy / edge / adversarial), with the baseline band
  • Latest experiment delta vs baseline (the "did the last PR help?" number)
  • Online eval pass rate (24h vs 7d) and guardrail-fail rate
  • Cost per query and p95 latency, trended
  • Dataset size and coverage by bucket (is it still representative?)
The loop, complete: production feedback grows the dataset → refreshed datasets re-validate thresholds → gates catch regressions → online evals catch drift → alerts trigger fixes → new changes go through the gates. That's not a process you maintain; it's a system that maintains quality. This is what "eval-driven AI" means in practice — and it's now fully implemented in your stack, whether your stack is LangSmith/LangGraph or Copilot Studio.

🧠 Knowledge Check

1. What does "output drift" mean and what detects it?

2. Which loops keep the eval dataset fed with real-world failures?

3. Why re-run model selection quarterly on a refreshed dataset?

Further Reading