AI Red Teaming: How to Test LLMs, RAG Pipelines and AI Agents

Key takeaways

Test the whole system, not just the model: the system prompt, retrieval data, tools and agent permissions are where most exploitable risk lives.
Prompt injection, data leakage, jailbreaks, RAG poisoning, tool abuse and runaway cost are the core categories, as reflected in the OWASP Top 10 for LLM Applications.
Automated scanners replay known payloads; adaptive, agent-driven testing changes approach when the target refuses, which finds what replay misses.
Every finding should carry the full exchange and a clear verdict. Unverified hits from scanners are often false positives.
Run it on every model, prompt, data or tool change, not once a year.

What AI red teaming is

AI red teaming borrows its name from military and security red teams: a group that plays the adversary to expose weaknesses. Applied to AI, it means deliberately trying to make a system misbehave: leak data it shouldn't, follow instructions planted by an attacker, take actions its owner never intended, or produce harmful output.

It differs from a traditional penetration test in what it targets. A pen test looks for flaws in code and infrastructure. AI red teaming looks for flaws in behavior, which emerge from the combination of a model, its instructions, the data it retrieves and the tools it can use. It also differs from model safety evaluations run by model providers: those test the base model in general, while your exploitable surface is your system prompt, your data and your tool permissions, which the provider never sees.

Red teaming is only legitimate against systems you own or have written permission to test. Agree scope, targets and budget in writing before any traffic is sent.

What to test: the attack surface

Surface	What goes wrong	OWASP LLM (2025)
System prompt and instructions	Instructions overridden, hidden rules and secrets disclosed	LLM01, LLM07
Conversation and memory	Another user's data leaks; refusals erode over many turns	LLM02, LLM01
RAG pipeline	Poisoned documents change answers or carry hidden instructions	LLM08, LLM04
Tools and agents	An agent calls tools it shouldn't, with arguments an attacker chose	LLM06
MCP servers and agent skills	Poisoned tool definitions, command injection, leaked credentials	LLM03, LLM06
Output handling	Markdown links or images exfiltrate data; output executed downstream	LLM05
Multimodal input	Instructions hidden in images or documents	LLM01
Cost and capacity	Token flooding and amplification run up cost or deny service	LLM10

The main attack techniques

Direct prompt injection: the user tells the model to ignore its instructions, often wrapped in a role-play, a hypothetical or an official-sounding request.
Indirect prompt injection: the instruction arrives inside content the model reads, such as a retrieved document, a web page, an email or a tool response. This is the most dangerous form for agents, because the attacker never talks to the system directly.
Encoding and obfuscation: instructions disguised with base64, character substitution or other encodings to slip past filters.
Multi-turn escalation: a request is built up gradually across a conversation so no single turn looks harmful. Crescendo and refusal-erosion attacks work this way.
Data extraction: coaxing out system prompts, other users' data or training data.
Retrieval poisoning: planting content in a knowledge base so the model repeats it as fact or follows it as an instruction.
Tool abuse and confused deputy: getting an agent to use its legitimate permissions for the attacker's purpose, such as issuing a refund or sending data to an external address.
Resource abuse: prompts that make the model generate unbounded output or loop, driving cost up.

Manual, automated and agentic red teaming

Approach	Strength	Limitation
Manual (human experts)	Creative, finds novel attack chains	Slow, expensive, a point-in-time snapshot
Automated scanners	Fast, repeatable, broad coverage of known payloads	Replays a fixed probe list, can't adapt to how the target refuses, many false positives
Agentic (adaptive AI attackers)	Changes the next payload based on the last response; runs multi-turn conversations; drills into what works	Needs a budget, a careful judge and evidence for every finding

Open-source frameworks such as NVIDIA's garak and Microsoft's PyRIT are good starting points for automated testing. The practical problems at scale are running several of them consistently, normalizing their results, and separating real findings from noise.

Judging results without drowning in false positives

A red-team run produces thousands of exchanges. Deciding which ones are real vulnerabilities is the hard part. Keyword matching flags refusals that happen to repeat the attack text; a single LLM judge can be confidently wrong.

01
Prefer deterministic checks where possible
A canary token that appears in the output, or a tool call that was actually made, is proof. Use these before asking a model to judge.
02
Use more than one judge for the rest
When two independent judges agree, treat it as confirmed. When they disagree, mark it partial and send it to a person.
03
Keep the full exchange
Every finding should include the exact prompt, response and context, so anyone can reproduce and verify it.
04
Call out likely false positives
Report confirmed, partial and refused results separately. A report that counts every scanner hit as a vulnerability loses the reader's trust.

Frameworks and requirements

OWASP Top 10 for LLM Applications: the shared vocabulary for LLM risks. A 2026 edition was published in August 2026.
OWASP Top 10 for Agentic Applications: risks specific to autonomous agents, published for 2026.
NIST AI 600-1, the Generative AI Profile of the NIST AI Risk Management Framework: recommends regular adversarial testing of generative AI systems.
MITRE ATLAS: a knowledge base of adversary tactics and techniques against AI systems, modeled on MITRE ATT&CK.
EU AI Act, Article 55: requires providers of general-purpose AI models with systemic risk to conduct and document adversarial testing.

How often to red team

A once-a-year engagement tests a system that no longer exists by the time the report lands. Model versions, system prompts, retrieval data and tools all change, and any of them can reopen a fixed hole. Run a consistent suite on every change, compare against a baseline so regressions stand out, and gate releases in CI. Keep human-led engagements for depth on the highest-risk systems; they start from a better baseline when automated testing runs underneath.

From findings to controls

A finding is only useful if something changes. Fix what can be fixed at the source: remove secrets from system prompts, narrow tool permissions, require approval for state-changing actions, sanitize output rendering, cap tokens. For what can't be fully fixed, prompt injection above all, turn the confirmed attack pattern into a runtime detection, run it in monitor mode to learn its false-positive rate, and enforce it once that rate is acceptable.

How Mongoose runs AI red teaming

Mongoose uses a team of agents: Recon fingerprints the target, a Planner picks attack categories, a Generator applies mutators to seed attacks, an Executor fires them, a Judge scores every exchange, an Adaptive Strategist drills into what worked, and a MultiTurn agent runs long conversations such as crescendo and refusal erosion. Open-source frameworks including garak and PyRIT run behind the same runner alongside Mongoose's own Malagasy modules for RAG, agent, multimodal, multi-turn and resource attacks. Findings map to the OWASP LLM Top 10 (2025) and your own controls, and confirmed findings become Vigilant runtime rules.

Frequently asked questions

What is the difference between AI red teaming and a penetration test?

A penetration test looks for flaws in code and infrastructure. AI red teaming looks for flaws in behavior: what a model, its instructions, its data and its tools can be made to do. The two complement each other.

Can AI red teaming be automated?

Largely, yes. Automated scanners cover known payloads quickly, and adaptive agent-driven testing adds multi-turn and context-aware attacks. Human experts are still valuable for novel attack chains on the highest-risk systems.

What is indirect prompt injection?

An attack where malicious instructions arrive inside content the model reads, such as a retrieved document, web page, email or tool response, rather than from the user. It's especially dangerous for AI agents.

How do you avoid false positives in AI red teaming?

Use deterministic checks such as canary tokens and observed tool calls first, require agreement between independent judges for the rest, keep the full exchange for every finding, and report confirmed, partial and refused results separately.

Is AI red teaming legally required?

For providers of general-purpose AI models with systemic risk, the EU AI Act requires documented adversarial testing. For most other organizations it isn't mandated by law, but frameworks such as the NIST AI Risk Management Framework recommend it, and auditors increasingly ask for it.

AI red teaming: how to test LLMs, RAG pipelines and AI agents