Guardrails: The Runtime Safety Layer

Lesson 5: Guardrails — The Runtime Safety Layer

Evals catch problems before you ship. Guardrails stop problems at runtime. They are the enforcement layer that sits between users, your model, and the world — filtering inputs before they reach the model, constraining what the model can do, and screening outputs before they reach the user. A production AI system without guardrails is a production system with an unauthenticated admin endpoint. This lesson gives you the threat model and the guardrail taxonomy; the tool-specific implementations come in Lessons 6–8.

The Threat Model: OWASP Top 10 for LLM Applications

The reference taxonomy of LLM application risks is the OWASP Top 10 for LLM Applications (2025 edition). It is the starting point for deciding what your guardrails must defend against:

# Risk What it means
LLM01 Prompt Injection Attacker instructions smuggled into user input override the system prompt
LLM02 Sensitive Information Disclosure Model leaks PII, secrets, or other users' data from context or training
LLM03 Supply Chain Compromised models, plugins, or datasets enter your stack
LLM04 Data and Model Poisoning Tampered training/fine-tuning data or retrieval sources corrupt behavior
LLM05 Improper Output Handling Unvalidated model output flows into SQL, HTML, or shell — classic injection
LLM06 Excessive Agency Model has too much tool/action permission and does damage autonomously
LLM07 System Prompt Leakage Users extract the system prompt via "repeat your instructions" attacks
LLM08 Vector and Embedding Weaknesses Poisoned or manipulable RAG content (LLM04 applied to your knowledge base)
LLM09 Misinformation Confidently wrong outputs (hallucination) as a systemic failure
LLM10 Unbounded Consumption Runaway token/API costs from malicious or pathological inputs
Design implication: You don't build one guardrail; you build a stack, because the risks live at different layers. Input filtering handles LLM01/02/10; output validation handles LLM05/09; permission design handles LLM06; content-safety filtering handles harmful output; rate limits and budgets handle LLM10.

Guardrails by Position in the Pipeline

Input guard injection, PII, topic Orchestr. scope, agency Model Output guard safety, PII, schema Action gate
  • Input guardrails — prompt-injection/jailbreak detection (e.g. Azure AI Content Safety Prompt Shields), PII detection before data reaches the model, topic/scope allowlists ("this copilot only answers HR questions — everything else routes to a human"), length and rate limits (LLM10).
  • Orchestration guardrails — least-privilege tool access (the model can only call what the user could), scoped retrieval (row/tenant-level filtering so one user can't pull another user's data), no "run arbitrary SQL" for agents, human approval for high-impact actions.
  • Output guardrails — content-safety classifiers on generated text (hate, violence, sexual, self-harm categories), PII/sensitive-data redaction, schema validation of structured output, policy compliance checks ("refuse anything not grounded in the knowledge base" — the LLM09 defense).
  • Action gate — the final control on anything with side effects: confirmations, budgets, sandboxes, audit logs (LLM06 defense).

Guardrail Technology: The Landscape

Technology What it does Notes
Azure AI Content Safety Harm-category moderation + Prompt Shields (injection/jailbreak detection) + PII detection Deeply integrated with Copilot Studio (Lesson 8); also available as an API for any stack
NVIDIA NeMo Guardrails Programmable rails (topical, safety, fact-checking, jailbreak) defined in Colang Open-source, framework-agnostic; adds a dialogue-management layer around your app
Guardrails AI Validators that run on input/output (regex, PII, toxicity, groundedness) Python library with a validator registry; good for custom pipelines
Llama Guard (Meta) Instruction-tuned classifier model for content safety Runs locally; common in self-hosted stacks
Moderation APIs (OpenAI, etc.) Classification of harmful content categories Simple to call, cloud-hosted
Your own code Schema validation, allowlists, rate limits, permission checks Often the most important layer and the cheapest

Guardrails vs Evals: Prevention and Detection

The two-layer rule: Guardrails are prevention (runtime, blocking); evals are detection (development-time, regression). You need both, and you need to test the guardrails themselves: your adversarial dataset must pass the guardrail stack (does it catch the injections?), AND you must track false positives (does it block legitimate users?). A guardrail that blocks 10% of real traffic is a product bug you're shipping in production.

In practice this means guardrails are subject to the same eval loop as the model: they get examples, thresholds, and regression tests. When you add a new jailbreak technique to your adversarial set, you're upgrading both the guardrail's training signal and your test coverage.

🧠 Knowledge Check

1. Which OWASP LLM risk is mitigated by limiting tool permissions and requiring human approval for high-impact actions?

2. What is the relationship between guardrails and evals?

3. Why is false-positive rate a critical metric for any input guardrail?

Further Reading