The Red Team Process

Lesson 3: The Red Team Process

Red teaming looks like chaos from the outside — people trying to break a chatbot with increasingly weird prompts. Underneath, the best engagements are ruthlessly organized. Microsoft, which has red teamed more generative AI products than anyone (over 100 as of 2023), treats it as a formal engineering process with a charter, a taxonomy, a playbook, and a report. This lesson gives you the full engagement lifecycle — the skeleton every serious red team runs, whether it's a three-person startup or a frontier lab.

The Engagement Lifecycle (7 Steps)

Step 1: Scope

Define what's in and what's out — the system, the models, the features, the data flows, and the limits (no production data destruction, no real customer harm, no external targets). Scope is where legal and engineering meet: a red team without boundaries is a liability; a red team with boundaries is a controlled experiment.

Step 2: Threat Model

Who are the adversaries, what do they want, and what can they touch? The threat model drives everything downstream. Typical questions: is the adversary a random internet user, a sophisticated state actor, a malicious insider, a supply-chain compromise? Can they only send prompts, or do they control training data? Do they have API access or physical access? Microsoft's guidance stresses involving domain experts here — a banking AI needs people who understand financial fraud, not just ML engineers.

Step 3: Charter & Rules of Engagement

The charter is the contract that authorizes the work: who's on the team, what they may test, what success looks like, how findings are handled, and escalation paths for dangerous discoveries (e.g., a real exploitable vulnerability, a child-safety issue, an active attack). A well-written charter means nobody has to ask permission mid-engagement — and it protects the red team when they find something uncomfortable.

Step 4: Build the Playbook

Turn the threat model into concrete test cases organized by your harms taxonomy (Lesson 2): a matrix of probe categories × attack techniques. Include both standard probes (known jailbreaks, injection payloads) and domain-specific scenarios ("an attacker tries to make the assistant reveal another customer's balance"). This is also where you decide the tooling (Lesson 9): which automated scanners, which manual techniques, who probes what.

Step 5: Execute — Attack, Document, Repeat

Now the actual adversarial work. Two rules make execution productive:

• Document as you go. Every finding gets: the exact prompt/payload, the output, the harm category, severity, and reproducibility. A jailbreak that worked once and can't be reproduced is a data point, not a finding.
• Iterate on the model's behavior. Red teamers follow threads: a refusal on the direct ask gets re-asked through roleplay, then encoding, then multi-turn escalation. The best findings are chains, not one-offs.

Step 6: Report

The report is the deliverable that matters — the thing management, engineers, and (increasingly) regulators will read. Structure it around your taxonomy with severity ratings, evidence, and reproduction steps. Microsoft's guidance emphasizes that findings must be actionable: each one points to a fix or an accepted residual risk, with an owner. A red team that finds 500 issues but can't say which 5 matter most has failed at the last step.

Step 7: Remediate & Retest

Findings feed back to product teams: stronger system prompts, guardrails, input/output filters, tool permissions, human review gates. Then retest — the same playbook, plus new probes targeting the fixes (fixes often create new bypasses). Red teaming is a loop, not a milestone; the engagement closes only when residual risk is explicitly accepted by a decision-maker.

Further Reading

• Microsoft AI Red Team — Building a Future of Safer AI (2023)
• Lessons from Red Teaming 100 Generative AI Products (Microsoft, 2023)
• Red Teaming Language Models to Reduce Harms (Ganguli et al., 2022)