Beyond Prompts: Data & Model Attacks

Lesson 6: Beyond Prompts — Data & Model Attacks

Prompt attacks target the model at inference time. But an AI system is also a machine learning artifact with training data, weights, and a supply chain — and each of those can be attacked. This lesson covers the non-prompt attack surface: adversarial examples, data poisoning, backdoors, model extraction, membership inference, and model inversion. These techniques predate the LLM boom by years and remain central to red teaming any ML system.

Adversarial Examples: Fooling the Model

In 2014, Ian Goodfellow and colleagues showed something shocking: you can add an imperceptible perturbation to an image — noise invisible to the human eye — and a well-trained classifier will confidently misclassify it. A panda becomes a gibbon; a stop sign becomes a speed limit sign. The technique, FGSM (Fast Gradient Sign Method), computes a perturbation using the model's own gradients: move the input slightly in the direction that maximizes loss. Adversarial examples are the reason we now take image-classifier robustness seriously — and they extend to LLMs (adversarial suffixes, which you met in Lesson 5) and to physical objects (adversarial stickers on road signs fooling autonomous driving perception systems).

Key idea: Neural networks are smooth, continuous functions, and small input changes can push outputs over decision boundaries. Adversarial examples exploit this geometry — the model isn't "wrong," it's manipulable in ways humans can't perceive.

Data Poisoning & Backdoors

If you can influence what a model is trained on, you can influence what it does. Data poisoning is the injection of malicious samples into the training data; a backdoor is a poisoned model that behaves normally until a trigger appears — then does the attacker's bidding.

  • BadNets (Gu et al., 2017): the canonical backdoor paper — poisoned a face-recognition model with samples stamped with a particular pattern, so any face with that pattern was misidentified as the attacker's chosen target. Clean accuracy stayed high; the trigger was invisible to normal testing.
  • LLM supply-chain risk: foundation models are trained on billions of tokens scraped from the open web — a surface attackers can seed. Researchers have shown both "poisoning the web" (planting text that steers later-trained models) and poisoned fine-tuning datasets.
  • RAG poisoning: with retrieval-augmented generation, the "training data" is a live corpus — and if attackers can inject documents into it (a wiki, a knowledge base, an email store), they get many of the same effects with much less effort.
Red team implication: backdoors are the hardest attack to detect because the model passes every normal test. Finding them requires trigger hunting (feeding unusual patterns and looking for anomalous outputs), provenance auditing of training data, and treating any third-party model or dataset as untrusted until proven otherwise.

Extraction, Inference, and Inversion: Stealing the Model's Secrets

Attack What the attacker gets Classic reference
Model extraction A functional copy of a proprietary model by querying it with carefully chosen inputs and training a replica on the outputs. Tramèr et al., 2016
Membership inference Whether a specific record was in the training data — a privacy leak (e.g., "was this patient in the training set?"). Shokri et al., 2017
Model inversion Reconstructing training data (faces, text) from the model's outputs — the most dramatic privacy breach. Fredrikson et al., 2015

All three are query-based: the attacker only needs API access. For LLMs, extraction is a real economic threat (stealing a fine-tuned model's behavior), and membership inference raises hard questions about models trained on sensitive data. Defenses: rate limiting and query monitoring (extraction), differential privacy (membership), and output perturbation plus access control (inversion).

Non-LLM AI Attacks Worth Knowing

Red teams in the real world attack more than chatbots:

  • Evasion of image/video classifiers: adversarial patches and perturbations defeat content filters, face recognition, and fraud detectors.
  • Physical-world attacks: adversarial stickers on stop signs caused a real (test-track) Tesla to misread "STOP" as a speed-limit sign (Eykholt et al., 2017).
  • Deepfake fraud: voice and video clones used for CEO-fraud ("voice phishing" / vishing), which has cost companies millions. Red teaming here means testing whether your org's people and verification systems can be fooled — and probing detection systems with synthetic media.
  • Recommender and fraud-model gaming: adversaries learn to game the models that decide ad delivery, credit, or spam — the "model is the product" attack surface.
Takeaway: An AI system is attackable at every stage of its lifecycle — data collection, training, deployment, and inference. A complete red team engagement covers all of it: poisoning and backdoor audits for the data and model, adversarial examples and extraction tests at the API, and prompt attacks at the application layer. Most teams start with prompts; mature teams cover the whole pipeline.

🧠 Knowledge Check

1. FGSM (2014) demonstrated that you can fool a classifier by:

2. What distinguishes a backdoored model from a simply broken one?

3. Which attack lets an attacker determine whether a specific record was in a model's training data?

Further Reading