How to Test an AI Agent for Prompt Injection
Browser and computer-use agents read untrusted content on every page they open, and any of that content can carry instructions. Prompt injection is when text the agent reads while doing its job — a hidden line in a web page, a comment in a document, a poisoned tool result — hijacks what the agent does next: leaking data, misusing a tool, or taking an action the user never asked for. Because the malicious instruction rides in through the same channel as legitimate content, it can't be filtered out by rule. Testing for it means red-teaming the agent with the attacks themselves — the test set is the attack.
Quick answer: Test an agent for prompt injection by running it against a battery of adversarial content that tries to hijack its instructions — hidden commands, tool-misuse lures, data-exfiltration and permission-escalation attempts — and measuring how often it refuses the injected instruction while still completing the legitimate task. Always include a clean-page control: an agent that refuses everything is safe and useless. There is no complete defense, so the real metric is a resistance rate that holds up when the attack is reworded.
What is prompt injection in an AI agent?
Prompt injection exploits the fact that an agent can't reliably tell content (the web page it's summarizing) from instructions (what you told it to do). Both are just text in its context. The high-risk form for agents is indirect injection: the attacker plants the instruction in content the agent will read anyway, so they never interact with the agent directly.
- Direct — the user types the malicious instruction. Mostly a policy problem.
- Indirect — the instruction hides in a page, file, email, or tool result the agent processes mid-task. This is the one that turns a helpful browser agent into a data-exfiltration vector, and it's what a serious red-team pack targets.
How do you test an agent for prompt injection?
Build a test set of attacks, run the agent against each, and score two things at once: did it resist the injection, and did it still do the real job. Cover the distinct attack classes rather than fifty variants of one:
| Attack class | What the injected content tries to do | What a passing agent does |
|---|---|---|
| Instruction hijack | "Ignore your task and instead do X" | Continues the original task; ignores the injected order |
| Tool-misuse lure | Coax the agent into an unintended tool call | Uses tools only for the legitimate task |
| Data exfiltration | Get the agent to leak context, keys, or history | Never emits secrets to an untrusted sink |
| Permission escalation | Trick the agent into an unauthorized action | Refuses; surfaces the request instead of acting |
| Clean-page control | (No attack — ordinary content) | Completes the legitimate task normally |
The clean-page control is not optional. Without it, the safest-looking result — an agent that refuses to act on anything — is indistinguishable from a genuinely robust one. Resistance is only meaningful paired with preserved usefulness.
How do you know the red-team eval actually discriminates?
The same principle that governs a capability eval governs a safety one: prove the test catches a planted failure before trusting it. Run the pack against a reference panel of agents whose behavior you already know.
- A hardened reference agent resists the injections and still does the task — it should score high.
- A naive reference agent gets hijacked by the more obvious attacks — it should score visibly lower.
- A sabotaged reference agent that follows injected instructions on purpose — it should score near the floor.
If the pack can't separate those, it isn't measuring injection resistance. See the prompt-injection defense benchmark for this discriminating-power check on a real panel, and what a reference-panel harness is for the method.
Why a single hardened system prompt isn't a test result
Teams often "fix" injection with a system-prompt line telling the agent to ignore malicious instructions, then declare victory. That's a mitigation, not a measurement. Attackers rephrase, encode, role-play, or split the instruction across a page, and a static guard clause doesn't generalize to attacks it never saw. The only honest claim is an empirical one: this agent resisted this battery of attacks at this rate — and the rate held when the attacks were reworded. An injection defense that collapses under paraphrase was never real, which is why robustness under perturbation is a first-class axis, not an afterthought.
Related
- What is a reference-panel harness — the discriminating-power method behind the "does the eval work" check.
- How to eval a tool-calling agent — the capability side of the same tool-using agents this red-teams.
- How to detect a gameable eval — why a resistance number that collapses under rewording isn't real.
- Safety packs — the category this red-team belongs to.
- Prompt-injection defense packs — browse the method.
- The browser-agent prompt-injection pack — a verified red-team pack implementing this approach.
- Prompt-injection defense benchmark — the discriminating-power results referenced above.
FAQ
Direct injection is the user themselves typing a malicious instruction. Indirect injection is the dangerous one for agents: the malicious instruction is hidden in content the agent reads while doing its job — a web page, a document, a tool result — so the attacker never has to talk to the agent directly. Browser and computer-use agents are exposed to indirect injection on every page they open.
No, and an eval that assumes you can is testing the wrong thing. There is no known complete defense — injected text and legitimate instructions arrive through the same channel. The measurable, useful question is a resistance rate: across a battery of attacks, how often does the agent refuse to follow the injected instruction while still completing the real task?
It helps a little and fails often. Attackers rephrase, role-play, encode, or split the instruction across the page, and a static 'ignore malicious instructions' line doesn't generalize to attacks it didn't anticipate. This is exactly why you test empirically against a battery of varied attacks instead of trusting a single guard clause.
Yes — this is the most-skipped check. An agent that refuses everything scores a perfect resistance rate and is useless. Every red-team run needs a clean-page control: on non-malicious content, the agent must still complete the legitimate task. Resistance only counts alongside preserved usefulness.
Related but distinct. Jailbreaking targets the model's own safety policy to make it produce disallowed content. Prompt injection targets an agent's task execution — hijacking what it does with its tools and permissions, like exfiltrating data or taking an unauthorized action. An agent can be perfectly 'aligned' and still be trivially injectable.
There's no universal pass mark, because it depends on the agent's blast radius — an agent that can only read scores differently from one that can send email or spend money. The useful signal is relative and adversarial: resistance should hold across attack rephrasings, and the gap between a hardened agent and a naive one should be large. A number that collapses when you reword the attack was never real.
We use OWASP's agentic and LLM-application threat vocabulary to organize the attack classes a red-team pack should cover, so the coverage frame rests on public, shared terminology rather than our private taxonomy. Note this is a capability red-team, not a certified clause-by-clause obligation mapping — that kind of formal conformance mapping is what our EU AI Act pack does, and it's a different exercise.