AI Red Teaming in 2026: How to Red-Team an LLM (Methods, Attacks & Tools)

How do you red team an LLM in 2026?

You red team an LLM by running structured adversarial attacks against it — prompt injections, jailbreaks, data-exfiltration attempts, and tool-misuse probes — then scoring how often each attack category succeeds and feeding the failures back into fixes and a CI regression gate. The critical mindset shift: treat attack success as a statistical rate, not a binary pass/fail. The scale of the problem is well documented. The "Do Anything Now" study collected 1,405 in-the-wild jailbreak prompts across 131 online communities from December 2022 to December 2023, and found five prompts that achieved a 0.95 attack success rate against both GPT-3.5 and GPT-4 across 107,250 test samples (Shen et al., 2023, arXiv:2308.03825). If five public prompts can break a frontier model 95% of the time, a single manual pass tells you nothing — you need automated breadth, targeted depth, and a scoreboard.

This is an operator's runbook, not an explainer. Below is the attack taxonomy mapped to the OWASP LLM Top 10 and MITRE ATLAS, the open tools that probe each one, a phased methodology with commands, a statistical scoring rubric, and a checklist.

Key facts

What is LLM red teaming, and how is it different from a pentest?

LLM red teaming is adversarial testing of a generative-AI system to find inputs that make it violate its safety, security, or policy constraints — leaking secrets, generating disallowed content, or taking unauthorized actions. Unlike a traditional penetration test, the target is probabilistic: the same prompt can succeed one run and fail the next, and the attack surface is natural language plus any tools, retrieval sources, and agents wired to the model. That is why red teaming an LLM is measured in attack-success rates over many trials, not exploited-or-not.

There are two structural targets, and conflating them is the most common mistake:

• Model-level (scanner) testing — you attack the model or endpoint directly: jailbreaks, prompt injection, system-prompt leakage, toxicity, encoding tricks. Tools like Garak run breadth-first here.
• System-level (agentic) testing — you attack the model as it is deployed: indirect prompt injection through retrieved documents, tool/function misuse, excessive agency, and MCP-connected actions. This surface is where real-world impact lives, and generic "what is red teaming" pages skip it.

The attack taxonomy: a unified crosswalk

Most guides give you either an attack list or a framework mapping. Here is the artifact that ties all four together — attack technique to OWASP LLM category to the closest MITRE ATLAS tactic to the open tool that probes it to how you score it.

No single competitor page ties technique, OWASP, ATLAS, tool, and scoring into one table. Print this and use it as your test-plan skeleton.

The three frameworks you build the exercise around

• OWASP Top 10 for LLM Applications (2025) — the canonical vulnerability taxonomy. Map every attack you run to a category so findings are portable and comparable. The 2025 list: LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Supply Chain, LLM04 Data and Model Poisoning, LLM05 Improper Output Handling, LLM06 Excessive Agency, LLM07 System Prompt Leakage, LLM08 Vector and Embedding Weaknesses, LLM09 Misinformation, LLM10 Unbounded Consumption (OWASP GenAI Security Project).
• MITRE ATLAS — an ATT&CK-modeled, living knowledge base of real-world adversary tactics and techniques against AI-enabled systems, spanning the AI lifecycle from Reconnaissance and Resource Development through ML Attack Staging, Exfiltration, and Impact, with case studies and mitigations (MITRE ATLAS, atlas.mitre.org). Use it to structure attack chains and to give findings a threat-intel spine. (ATLAS is versioned and its technique counts change with each release — pull current tactics/techniques directly from atlas.mitre.org rather than quoting a fixed count.)
• NIST AI Risk Management Framework (AI RMF 1.0) — the governance wrapper. It is a voluntary framework organized around four core functions — Govern, Map, Measure, Manage — released January 26, 2023, with its Generative AI Profile (NIST AI 600-1) released July 26, 2024 (NIST). Red teaming is your Measure function; the rubric and remediation loop below is how you close Manage.

Manual vs. automated red teaming

You need both, in sequence.

Automated red teaming runs large probe libraries and adversarial-search algorithms at scale. It is how you get statistical coverage across every OWASP category cheaply and reproducibly — the right first pass. Frontier labs still run large human red teams for depth and novelty, so automation is a floor, not a ceiling.

Manual red teaming is targeted human creativity against your application's specific logic — its system prompt, its tools, its business rules. Automated tools rarely discover the domain-specific jailbreak that abuses your refund flow or your agent's file access. Manual probing is where the highest-severity findings usually come from.

The efficient pattern: automate breadth first, then spend human hours only where automation flags weakness or where your app has unique surface.

Open-source red-teaming tools (2026)

Two tool notes worth pinning:

• Garak is the default breadth-first scanner — an open-source LLM vulnerability scanner maintained by NVIDIA under Apache-2.0 that probes for prompt injection, jailbreaks, data leakage, encoding attacks, toxicity, malware generation, misinformation, and glitch tokens (NVIDIA/garak GitHub).
• PyRIT (Python Risk Identification Tool for generative AI) is Microsoft's MIT-licensed red-teaming framework. Its canonical repo is now github.com/microsoft/PyRIT — the older Azure/PyRIT repo was archived read-only on March 27, 2026, so pin your dependency to the new location (microsoft/PyRIT GitHub).

The end-to-end exercise: a phased runbook

Run these phases in order. Commands are illustrative starting points — adapt endpoints and probe sets to your target.

Phase 1 — Scope and threat-model

Define the target (model, endpoint, agent, or full app), the assets at risk (secrets, PII, tools, spend), and the in-scope OWASP categories. Decide model-level vs. system-level split up front. Write down the harm scenarios you actually care about — this becomes your private, domain-specific test set.

Phase 2 — Automated breadth scan (Garak)

Get a statistical baseline across probe families before any human effort.

python -m pip install garak
python -m garak --model_type openai --model_name gpt-4o \
  --probes dan,promptinject,leakreplay,encoding,malwaregen

Record attack-success rate per probe family. This is your starting scoreboard.

Phase 3 — Targeted manual probing

Attack your app's specific logic by hand: system-prompt extraction, refusal-boundary probing, roleplay jailbreaks tuned to your domain, and abuse of your unique tools. Log every successful prompt verbatim.

Phase 4 — Multi-turn escalation (PyRIT / Crescendo)

Single-turn refusals often collapse over several turns. Use PyRIT's orchestrators to run patient, multi-turn escalation (Crescendo) toward each harm goal and score attack success by turn depth.

pip install pyrit
# Configure a Crescendo orchestrator against your target endpoint,
# set the objective, and run multi-turn escalation trials.

Phase 5 — Agent and tool-misuse / indirect-injection tests

For agentic systems, plant malicious instructions in the places your agent reads — retrieved documents, tool outputs, web pages, MCP resources — and measure whether the agent executes unauthorized actions. This is LLM06 (Excessive Agency) territory and the surface most explainer pages miss.

npx promptfoo@latest redteam init
# Use presets: owasp:llm, owasp:agentic, mitre:atlas, nist:ai:measure
npx promptfoo@latest redteam run

Phase 6 — Statistical scoring

Aggregate results as attack-success rate by OWASP category, not pass/fail. Run enough trials per attack that the rate is stable. A category that fails 8% of the time is a different risk than one that fails 80% — binary reporting erases that.

Phase 7 — Severity triage and remediation

Rank findings by severity (impact × exploitability × attack-success rate), fix the top tier (input/output filtering, system-prompt hardening, tool-permission scoping, retrieval sanitization), and re-test to confirm the rate dropped.

Phase 8 — Regression gate in CI

Convert your worst findings into a Promptfoo test suite and wire it into CI so a merge that regresses a fixed vulnerability fails the build. Red teaming becomes continuous, not a one-off.

The scoring rubric: results are statistical, not binary

Anchor the "why statistical" point in the data: the "Do Anything Now" study needed 107,250 test samples across 13 forbidden scenarios to characterize behavior, and even the strongest prompts landed at a 0.95 rate — not 1.0 (Shen et al., 2023). Automated adversarial search tells the same story: TAP jailbreaks GPT-4-Turbo and GPT-4o on more than 80% of prompts (Mehrotra et al., NeurIPS 2024) — a rate, reported as a rate.

The red-team exercise checklist

• Scope defined: target, assets, in-scope OWASP categories, model-level vs. system-level split
• Private, domain-specific harm scenarios written down
• Threat model mapped to MITRE ATLAS tactics
• Automated breadth scan run (Garak); ASR per probe family recorded
• Targeted manual probing done against app-specific logic
• Multi-turn escalation run (PyRIT / Crescendo); ASR by turn depth
• Agentic tests run: tool misuse, indirect injection, MCP resources (Promptfoo)
• Results scored as ASR by OWASP category, not pass/fail
• Findings triaged by severity (impact × exploitability × ASR)
• Top-tier findings remediated and re-tested
• Worst findings converted to a CI regression gate
• Results reported through the NIST AI RMF Measure/Manage functions for risk owners

Frequently asked questions

What is the difference between LLM red teaming and jailbreaking?

Jailbreaking is one attack technique — crafting a prompt that bypasses a model's safety guardrails. Red teaming is the full discipline: it uses jailbreaks alongside prompt injection, data exfiltration, tool misuse, and multi-turn escalation, then scores, triages, and remediates across the whole system.

Which OWASP category covers prompt injection?

Prompt injection is LLM01 in the OWASP Top 10 for LLM Applications (2025), the first and most prominent category. Indirect prompt injection through retrieval also touches LLM08 Vector and Embedding Weaknesses, and when it drives an agent to act it becomes LLM06 Excessive Agency (OWASP GenAI Security Project).

Is Garak or PyRIT the better tool to start with?

Start with Garak for a fast, broad, reproducible baseline scan of the model — it is NVIDIA's Apache-2.0 scanner with dozens of probe families. Move to PyRIT (Microsoft, MIT) once you need multi-turn escalation like Crescendo or multimodal orchestration. They are complementary, not competing.

How do you red team an AI agent, not just a model?

Attack the agent's inputs and actions, not only its prompt. Plant malicious instructions in retrieved documents, tool outputs, web pages, and MCP resources (indirect prompt injection), and measure whether the agent takes unauthorized actions — that is LLM06 Excessive Agency. Promptfoo's agentic presets and PyRIT orchestrators are built for this system-level layer.

Why is attack-success rate reported as a percentage instead of pass/fail?

Because LLMs are probabilistic — the same prompt can succeed one run and fail the next. A category that fails 8% of the time carries very different risk than one failing 80%, and binary reporting erases that. The research reflects this: even the strongest documented jailbreak prompts landed at a 0.95 attack success rate, not a certain break (Shen et al., 2023).

Does red teaming satisfy AI regulation and governance requirements?

Red teaming maps to the Measure function of the NIST AI Risk Management Framework (AI RMF 1.0), whose four functions are Govern, Map, Measure, Manage (released 2023-01-26), with a Generative AI Profile (AI 600-1) released 2024-07-26. It is a core input to governance, but you still need the Govern and Manage functions — documented ownership, remediation, and continuous monitoring — around it (NIST).

Related reading

• Security roles in AI: the 2026 career path
• AI governance in 2026
• What is an AI engineer in 2026?

Want to run a real red-team exercise end to end instead of reading about one? Build the full workflow — scope, automated scan, manual and multi-turn probing, scoring, and a CI regression gate — live in one session in the AI Red Teaming sprint.

Sources: OWASP GenAI Security Project — Top 10 for LLM Applications (genai.owasp.org/llm-top-10); NIST — AI Risk Management Framework (nist.gov/itl/ai-risk-management-framework); MITRE ATLAS (atlas.mitre.org); NVIDIA/garak GitHub (github.com/NVIDIA/garak); microsoft/PyRIT GitHub (github.com/microsoft/PyRIT); Shen, Chen, Backes, Shen & Zhang (2023), "Do Anything Now," arXiv:2308.03825; Mehrotra et al., "Tree of Attacks with Pruning," NeurIPS 2024, arXiv:2312.02119.