How AI Systems Fail

Lesson 2: How AI Systems Fail

Before you can red team anything, you need a map of the territory. AI systems fail in four fundamentally different ways, and each requires different probing. The classic mental model separates failures by where they live: the model itself, the data it was trained on, the application built around it, or the infrastructure underneath. Understanding the full stack of failure is what separates a red team that finds one bug from a red team that finds classes of bugs.

The Four Layers of Failure

Attacks rarely respect these boundaries — an attacker will chain failures across layers. But the taxonomy gives you a starting point: what am I trying to make fail, and where would that failure live?

Failure Family #1: Harms & Policy Violations

The most visible class of AI failure. The model does something its operator explicitly doesn't want it to do. Red teams typically organize these into a harms taxonomy — a structured list of prohibited output categories. Microsoft's red teaming guidance, for example, organizes generative AI harms into families including harmful content (hate, harassment, sexual content, self-harm, violence), misinformation, and socioeconomic and civil liberties harms (e.g., providing unauthorized financial, legal, or medical advice; facilitating disenfranchisement). These categories are not academic: they map directly to the content policies of every major AI provider, and to what a regulator will hold you accountable for.

Failure Family #2: Capability Failures

Sometimes the model does exactly what it was asked — and that's the problem. A red team probes for dangerous capabilities: things the model can technically do that, at scale or in the wrong hands, cause harm. Examples:

• Weapons & biosecurity knowledge: synthesizing instructions for weapons, toxins, or biological agents.
• Cyber offense: writing working exploit code, evasion techniques, or malware that passes.
• Mass manipulation: producing persuasive disinformation, spear-phishing, or propaganda at scale.
• Unauthorized expertise: giving confident, authoritative answers in regulated domains (medical, legal, financial) without guardrails.

Capability failures are subtle because the model's benchmark score might be excellent — the model is working as trained. The risk is contextual: capability + deployment = harm.

Failure Family #3: Security Vulnerabilities

These are the failures that look most like traditional security bugs, and they're the fastest-growing area of AI red teaming:

• Prompt injection — hijacking the model's instructions (Lesson 4).
• Jailbreaks — bypassing safety training (Lesson 5).
• Data & model attacks — poisoning, extraction, membership inference (Lesson 6).
• Excessive agency — an agent with tools, permissions, or API access doing more than it should when manipulated.
• Data exfiltration — indirect injection used to siphon confidential data out of an enterprise system.

Failure Family #4: Reliability & Drift

Less dramatic, often forgotten, frequently catastrophic: the system degrades over time. Models drift as the world changes, RAG corpora go stale, evaluation pipelines rot, and a system that passed every red team test in January fails in July because of a changed dependency, a shifted data distribution, or a model update. A mature red teaming program isn't a one-shot engagement — it's a cadence that re-tests after every significant change.

A Working Harms & Failures Checklist

Here's a starter checklist to structure your own red teaming — we'll deepen each item in later lessons:

Further Reading

• Microsoft AI Red Team — Building a Future of Safer AI
• Microsoft Responsible AI: content categories and harms
• On the Opportunities and Risks of Foundation Models (Bommasani et al.)