LangSmith — Observability & Evaluation

Lesson 5: LangSmith — Observability & Evaluation

Once your app runs, you need to see it. LangSmith is LangChain Inc.'s LLMOps platform: a proprietary, cloud-hosted service for tracing every LLM call, evaluating outputs, managing prompts, and monitoring production. It's the closest thing the Lang ecosystem has to "the official dashboard."

Key idea: LLM apps fail differently than regular software — outputs are probabilistic, costs accumulate per token, and "it works" is a spectrum. LangSmith turns those invisible failure modes into inspectable traces, scored evals, and live dashboards.

What LangSmith gives you

FeatureWhat it does
TracingRecords every step of a run — model calls, retriever hits, tool invocations — with tokens, latency, and costs
Datasets & evalsCurate test inputs, run them through your app, score outputs (LLM-as-judge, custom scorers, or human)
Prompt HubVersioned, shareable prompt management — pull prompts into code, roll back bad edits
MonitoringDashboards, alerts, and feedback collection for production traffic
PlaygroundInteractive UI to experiment with prompts and models before changing code
AutomationsRule-based actions: run an eval when a new trace arrives, annotate anomalous runs

Hooking it up

With LangChain, tracing is nearly free — set an API key and runs appear automatically:

# Works automatically with LangChain/LangGraph — no code changes needed
export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY=lsv2_...
export LANGCHAIN_PROJECT=my-app

# Or set it in code:
from langchain_core.callbacks import config
from langchain_core.runnables import RunnableConfig

config = RunnableConfig(tags=["production"], metadata={"user_id": "u-42"})
result = chain.invoke({"topic": "RAG"}, config=config)

Not using LangChain? The langsmith SDK traces any framework via decorators, and OpenTelemetry support lets non-LangChain apps send spans in too. The tracing layer is deliberately framework-agnostic — LangSmith doesn't require you to use LangChain.

TRACE FLOW your app (any framework) LangSmith SDK / OTel traces · evals dashboards prompts · datasets · feedback · alerts

The eval loop

The workflow that makes LangSmith (or any LLMOps tool) valuable is the eval loop: collect a dataset of real inputs → run them through your app → score outputs automatically → fix the worst failures → re-run. With LLM-as-judge scorers you can grade hundreds of outputs in minutes. This is how "good enough" becomes "measurably better."

Pitfall: Tracing every single call is the default, but production traffic is noisy and costs storage/quota. Use projects to separate dev/staging/prod, and filter or sample high-volume traces. Also: LangSmith is a paid SaaS with no self-hosted option (free tier available) — it is not open source, which is exactly why Langfuse exists.
Rule of thumb: Tracing answers "what happened?"; evals answer "was it good?"; prompt versioning answers "what changed?" — you need all three for any app that will outlive a demo.

🧠 Knowledge Check

1. What is LangSmith, in one line?

2. How do you enable LangChain tracing?

3. What is the "eval loop"?

Further Reading