Tools of the Trade

Lesson 9: Tools of the Trade

No serious red team attacks a modern LLM system by hand alone. An open-source tooling ecosystem has grown up around AI security, and a good red teamer knows which tool does what — and, just as importantly, what tools can't do (no scanner replaces human adversarial creativity). This lesson is your tool map.

The Big Three Frameworks

Tool Maker What it does Best for
PyRIT (Python Risk Identification Tool) Microsoft Automates red teaming pipelines: generates attack prompts (jailbreaks, injection, leakage probes), fires them at targets, and scores the results with configurable scoring engines (self-ask, LLM-based graders, Azure content safety). Continuous, scripted red teaming of chat models and agents; integrating into CI.
garak NVIDIA An LLM vulnerability scanner: plug in a model endpoint, and garak runs hundreds of probes across categories — prompt injection, jailbreaks, data leakage, hallucination, encoding attacks. Quick, broad vulnerability sweeps and regression testing after model updates.
ART (Adversarial Robustness Toolbox) IBM The oldest and broadest ML-security library: adversarial examples (FGSM, projected gradient descent), poisoning, evasion, extraction for classifiers and models of all kinds, not just LLMs. Traditional ML models, image classifiers, and the non-prompt attack surface (Lesson 6).

Specialists & Adjacent Tools

  • TextAttack — academic framework for adversarial attacks on NLP models (word swaps, paraphrases); great for research-grade robustness testing of classifiers.
  • LLM-Attacks (GCG) — the reference implementation of gradient-based adversarial suffix attacks (the "universal jailbreak" you met in Lesson 5).
  • promptfoo — an LLM evaluation and testing framework with a red-team mode: run prompt variations, compare outputs, and detect injection/jailbreak failures in CI. Hugely popular for app teams.
  • Giskard — ML/LLM testing platform covering security, robustness, and performance; useful for keeping red team findings in the same place as other model quality gates.
  • CyberSecEval (Meta) — a benchmark suite that measures LLM security: how readily models produce unsafe code, comply with cyberattacks, or leak secrets. More of an evaluation than an attack tool, but the de facto standard for comparing models' security posture.
Key idea: Framework vs. scanner vs. benchmark: PyRIT is a harness (you drive it, it scores), garak is a scanner (point it and get a report), CyberSecEval is a benchmark (measures models against fixed tests). Most mature programs use all three types: benchmarks for model selection, scanners for regression sweeps, and PyRIT-style harnesses for deep targeted campaigns.

Automated & LLM-vs-LLM Red Teaming

Can AI red team AI? Yes — and it scales. The landmark paper Red Teaming Language Models with Language Models (Perez et al., 2022) showed that one LLM can automatically generate thousands of attack prompts against another, discovering harmful behaviors at a scale human teams can't match — with the discovery rate scaling with the red team model's size (bigger attacker = more findings). This insight powers modern automated pipelines:

  • LLM-as-attacker: generate jailbreak variants, roleplay frames, and injection payloads programmatically; run them through PyRIT/garak; score with LLM graders.
  • Adversarial collaboration (BIG-Bench, Anthropic): human+AI teams deliberately try to break other models' tasks or safety — the "attacker" mindset institutionalized.
  • RL from red-team data (Ganguli et al., 2022): findings from red teaming become training data — the model is fine-tuned to refuse the discovered attacks, then re-tested (the loop from Lesson 3, automated).
Where automation fails: scanners miss novel attacks. Automated tools are great at re-finding known techniques at scale; they are weak at the creative leaps — cross-domain attacks, subtle multi-turn manipulations, domain-specific abuse — that human experts (especially domain experts, per Microsoft) catch. The right ratio is roughly: automation for breadth and regression, humans for depth and novelty.

A Practical Toolchain (Startup-to-Enterprise)

  1. Model selection: run CyberSecEval (and safety benchmarks) to shortlist models.
  2. Continuous sweep: wire garak into CI; run it on every model or prompt change. Fail the build on new high-severity probe hits.
  3. Targeted campaigns: PyRIT for injection/jailbreak/leakage campaigns against your specific system and harms taxonomy, with your own scoring.
  4. Non-LLM surface: ART/TextAttack for any classifiers, embeddings, or vision components in your stack.
  5. Humans on top: quarterly expert red team engagements focused on domain abuse and novel techniques; feed findings back into the automated tooling.
Takeaway: The tooling is mature enough that "we don't know how to start" is no longer an excuse — garak gives you a baseline report in an afternoon. But tools are force multipliers, not substitutes: the framework is your harness, the taxonomy is your map, and humans are still the ones who find what nobody thought to probe.

🧠 Knowledge Check

1. Which Microsoft tool automates red teaming pipelines with attack generators and configurable scoring engines?

2. NVIDIA's garak is best described as:

3. What did Perez et al. (2022) demonstrate about LLM-vs-LLM red teaming?

Further Reading