Standards & Governance: NIST, EU AI Act & ISO

Lesson 8: Standards & Governance — NIST, EU AI Act & ISO

Red teaming stopped being optional. Around the world, regulators and standards bodies have turned adversarial testing of AI from a "nice to have" engineering practice into an explicitly expected obligation. If you're building or deploying AI systems, this lesson is the part of the course where red teaming becomes a compliance requirement — and knowing the landscape early is a genuine competitive advantage.

NIST AI Risk Management Framework (AI RMF 1.0)

The US National Institute of Standards and Technology published its AI RMF in January 2023 — the closest thing the field has to a consensus baseline for AI risk governance. It's voluntary (not a regulation), but it is the reference framework that regulators, courts, and customers will expect you to have considered. The RMF is built around four core functions:

Function What it means Red team link
Govern Set risk culture, roles, and policies. Charter, scope, and approval for red teaming.
Map Understand the system, context, and stakeholders. Threat modeling, harms taxonomy, attack surface.
Measure Assess and test risks with metrics. The red team engagement itself — probing, evals, scoring.
Manage Prioritize, respond to, and communicate risks. Findings triage, remediation, residual risk sign-off.

In 2024, NIST published the Generative AI Profile (AI 600-1), which tailors the RMF to generative AI and — importantly for this course — explicitly names red teaming and adversarial testing as a primary risk management action, along with content provenance, watermarking, and information disclosure controls. If a US regulator asks "what did you do to manage AI risk?", "we ran NIST-aligned red teaming" is the answer they expect to hear.

US Executive Order 14110 (Oct 2023)

President Biden's October 2023 Executive Order on Safe, Secure, and Trustworthy AI made red teaming a federal requirement: developers of the most powerful foundation models (models that could pose serious risks to national security, public safety, or public health) must perform red team testing and report results to the government before release. It directed NIST to develop the GenAI Profile and mandated standards for AI safety, watermarking, and testing. The policy direction continued into 2024–2025 with implementation guidance. The exact legal details matter less than the trend: frontier model red teaming is now a precondition for release in the US federal orbit.

Key idea: The US approach is largely voluntary-but-expected — standards-backed and procurement-driven. The EU approach, by contrast, is binding law. Both converge on the same practical demand: document your adversarial testing.

The EU AI Act

The EU AI Act — the world's first comprehensive AI law, adopted in 2024 — creates a risk-tiered regulatory regime. For this course, the key obligations:

  • General-purpose AI (GPAI) models with systemic risk: the most powerful foundation models must undergo adversarial testing ("red teaming") under the Act's systemic-risk provisions — evaluations, incident reporting, and mitigation obligations. This is a hard legal requirement with fines attached.
  • High-risk AI systems: subject to conformity assessments, risk management, and documentation — which effectively requires evidence of testing that covers adversarial scenarios.
  • Practical implication: if your product is built on a systemic-risk model or classifies as high-risk, your red team reports are legal evidence. Run them like you'd run a penetration test for PCI-DSS — documented, repeatable, and defensible.

ISO/IEC 42001: The Management-System Angle

ISO/IEC 42001 is the first international standard for AI management systems (published 2023). Think of it as "ISO 27001 for AI": it gives organizations a certifiable framework for governing AI throughout its lifecycle — risk assessment, policies, controls, continual improvement. Red teaming slots in as a control under its risk-management and testing expectations. Certification is voluntary, but enterprises and governments increasingly ask for it from AI suppliers — and an auditable red teaming program is one of the strongest pieces of evidence you can produce.

What This Means for Your Red Team

  1. Document everything. Scope, charter, playbook, findings, remediation, retest results — in a form an auditor can pick up and understand. "We attacked it and it seemed fine" does not satisfy any of these frameworks.
  2. Align your taxonomy. Map your harms categories and findings to NIST RMF functions and OWASP/ATLAS IDs so external reviewers can navigate your work instantly.
  3. Test at the right cadence. Pre-release for releases, post-change for every significant model or feature update, and on an ongoing schedule for live systems — the frameworks expect continual monitoring, not a single engagement.
  4. Know your tier. The obligations scale with risk: a customer-support chatbot and a frontier foundation model live in different regulatory universes. Scope your program to your actual exposure.
Takeaway: Governance frameworks don't replace red teaming — they demand it and give it a structure to live in. NIST gives you the risk lifecycle, ATLAS/OWASP give you the attack language, and the EU AI Act and EO 14110 give you the legal reason to care. A well-run red team program is simultaneously great engineering, great risk management, and great compliance evidence.

🧠 Knowledge Check

1. Which NIST AI RMF function covers "assess and test risks with metrics" — where red team engagements live?

2. Which regulation makes adversarial testing (red teaming) a binding legal obligation for the most powerful general-purpose AI models?

3. Why should red team reports be written to be auditor-friendly?

Further Reading