LangSmith: Datasets, Experiments & Regression Testing

Lesson 6: LangSmith — Datasets, Experiments & Regression Testing

LangSmith is the closest thing to "CI for LLM apps" that exists today: it traces every run of your application, stores datasets, runs experiments (your app against a dataset, scored by evaluators), lets you compare versions side-by-side, and extends into production with online evals and monitoring. This lesson walks the full implementation: from dataset to gated CI job. Everything here works for plain LLM chains, LangChain, and LangGraph apps (Lesson 7) alike.

The Pieces and How They Fit

Step 1 — Create the Dataset (SDK or UI)

from langsmith import Client

client = Client()  # uses LANGCHAIN_API_KEY env var

ds = client.create_dataset(
    dataset_name="support-golden-v1",
    description="Support copilot: 40 happy/edge/failure + 10 adversarial"
)
client.create_examples(
    dataset_id=ds.id,
    inputs=[{"question": "How do I reset my password?"}],
    outputs=[{"answer": "Go to Settings > Security > Reset password."}],
    # ... one dict per example; adversarial examples can omit outputs
)

You can equally create datasets and paste examples in the UI — the SDK version is what makes it scriptable and CI-friendly.

Step 2 — Define Evaluators

from langsmith.evaluation import evaluate
from langsmith.schemas import Example, Run

# (a) Deterministic: exact match on a field
def exact_match(run: Run, example: Example) -> dict:
    pred = run.outputs.get("answer", "")
    return {"key": "exact_match", "score": int(pred == example.outputs.get("answer", ""))}

# (b) LLM-as-judge: built-in criteria scorer ("correctness")
from langsmith.evaluation.evaluators import criteria
correctness = criteria("correctness")

# (c) Custom judge with your own rubric prompt
def rubric_judge(run: Run, example: Example) -> dict:
    # call your judge model with a detailed rubric; return {"key","score","comment"}
    ...

# (d) Programmatic pipeline check: did the right tool get called?
def tool_check(run: Run, example: Example) -> dict:
    called = [c.get("name") for c in run.outputs.get("calls", [])]
    return {"key": "tool_check", "score": int(example.outputs.get("expected_tool") in called)}

Step 3 — Run the Experiment

results = evaluate(
    target=my_app,               # any callable: chain.invoke, graph.invoke, plain function
    data="support-golden-v1",    # dataset name
    evaluators=[exact_match, correctness, rubric_judge, tool_check],
    experiment_prefix="prompt-v2",
)
summary = results._summary  # aggregate scores

Each example runs once; every evaluator scores every output; results land in the UI with per-example breakdowns, failure traces, and token/latency stats. Name experiments by change (prompt-v2, model-gpt5-mini) so the history reads like a changelog.

Step 4 — Regression Testing: Diff Against Baseline

In the UI, select two experiments on the same dataset and LangSmith shows a per-example score diff — green/red per row, with the failing traces one click away. Programmatically, fetch both experiments and compute the deltas yourself for CI:

from langsmith import Client
client = Client()

def pass_rate(experiment_id: str) -> float:
    ex = client.read_experiment(experiment_id)
    scores = [f.get("score") for f in ex.feedback_stats.get("correctness", [])]
    return sum(1 for s in scores if (s or 0) >= 0.8) / len(scores)

new_rate = pass_rate(new_exp_id)
base_rate = pass_rate(baseline_exp_id)
print(f"new={new_rate:.2f} baseline={base_rate:.2f}")
# CI gate: fail on regression or sub-threshold

Step 5 — Wire It Into CI

# ci_eval.yml (GitHub Actions sketch)
# 1. checkout, pip install langsmith langchain
# 2. LANGCHAIN_API_KEY=${{ secrets.LANGCHAIN_API_KEY }}
# 3. python run_evals.py --experiment-prefix "ci-${{ github.sha }}"
#    (runs steps 2-4, then exits non-zero if gate fails)

The eval job becomes a required check on every PR touching prompts, models, retrieval config, or guardrails. Slow? Yes — and that's the point: the eval job is your test suite; a 10-minute eval job on a 50-example dataset is cheap insurance against shipping a regression.

Production: Online Evals, Annotation, Playground

• Online evals — attach evaluators to production traces so every live conversation gets scored (e.g. guardrail-pass, topic-relevance). Use LangSmith monitoring dashboards and thresholds to alert when the live pass rate dips.
• Annotation queues — human reviewers mark real conversations; the flagged ones become your next dataset version. This closes the loop: production failures become regression tests.
• Playground + prompt versioning — iterate on prompts in the Playground against dataset examples, then promote a winning version and validate it with a full experiment before shipping. The prompt, not just the code, is a versioned artifact.

Further Reading

• LangSmith documentation (home)
• LangSmith evaluation docs
• LangSmith datasets guide