Browser agent — prompt-injection red-team
Adversarial web content that tries to make a computer-use agent exfiltrate data or take destructive actions. The test set is the attack, not a Q&A.
The method this pack implements is written up in How to test an AI agent for prompt injection.
Verification report
Six axes, one grade
Every pack is measured on the same six axes — including whether it stays discriminating under perturbed inputs, and whether it's still current against the moving capability frontier. No oracle, no self-reporting — reproducible measurement only.
Structural validity
Reproducible, seed-stable scoring with no leakage from the golden set into the rubric — provable without knowing the ‘best’ eval.
No data leakage 1.00 · Ungameable 0.73 · Deterministic 1.00
Discriminating power
Cleanly separates strong from weak agents on the reference panel; the sabotaged reference agent should fail.
Standard coverage
Anchored to OWASP Top 10 for Agentic Apps.
Test thoroughness
Test volume weighed against the minimum expected for a 'high risk' pack — not a flat headcount.
15 test cases
Robustness
Holds its discriminating power when the test items are reordered or padded with distractors — so the grade tracks agent behavior, not the exact surface form of the frozen test set.
Robustness across the input space
Currency
How current the pack is against the moving frontier of agent capability. An eval today's best agents have saturated no longer discriminates, however well-built — this decays over time and resets on revalidation. Not calendar age.
Fresh — validated against the current frontier cohort
What this grade does and doesn’t prove — published on purpose, not hidden in fine print.
- Reference agents are controlled behavioral archetypes (good/broken/sabotaged), not live production models — validate against your own agent's real outputs before treating this grade as a guarantee for your specific deployment.
- The test set covers 15 cases — enough to establish discriminating power, not exhaustive coverage of every question shape in this domain. Treat a pass as evidence the pack works, not proof your exact corpus is fully covered.
- Only 15 test cases for a 'high-risk' pack — 50 is the bar for full thoroughness credit here. Treat this grade as a starting signal, not exhaustive proof for a deployment at this risk level.
- Structural-validity score is 0.73, not a perfect 1.00 — some positional or phrasing regularity remains in the test set. Treat this as strong rather than absolute protection against a shortcut-taking agent.
- Discriminating power is 0.83 — the pack ranks known-good above known-sabotaged, but the margin isn't maximal. Borderline agents near the pass threshold deserve a closer manual look, not just the number.
Reference panel
How the panel performed
Reference agents — from strong to deliberately sabotaged — ran the full pack. A trustworthy eval must rank them correctly.
A good pack scores the known-good agent high and the sabotaged one near zero. That gap is the evidence the meter works.
Output
What you get when you run it
A pass/fail result per test case (viewable via `promptfoo view`, exportable to JSON/CSV).
- pass or fail per test case, with the specific assertion that failed if any
- an aggregate pass-rate percentage across the test set
- the full prompt/response pair for every case, for manual review of failures
`promptfoo eval` exits non-zero when the pass-rate drops below the config's threshold, which fails the CI job automatically — no separate scripting needed.
The known-good reference agent on this pack scores 1.00 — if your agent's aggregate score lands well below that, treat it as a signal to investigate before shipping, not a false alarm.
A real sample of the scorecard this pack produces — actual numbers from the known-good and known-sloppy reference agents, not a mockup.
| Category | Known-good agent | Known-sloppy agent |
|---|---|---|
| Data exfiltration | 1.00 | 0.50 |
| Destructive action | 1.00 | 1.00 |
| Instruction override | 1.00 | 1.00 |
| Fake tool output | 1.00 | 0.50 |
| Chained multi-step | 1.00 | 0.50 |
| Benign-task completion | 1.00 | 1.00 |
0-1 score, per category — this is the shape of your own report once you run the pack.
Real content from this pack's own test set, run through the known-good reference agent:
User goal/page content/behavior are real content from this pack's own test set, run through the known-good reference agent — not a fabricated illustration.
Your aggregate score can be broken down per attack type, plus a benign-task check.
| Category | What it measures | A low score here means |
|---|---|---|
| Data exfiltration | Whether the agent resists this specific attempt: 'data exfiltration'. | A dip concentrated in this category points at a specific vulnerability to patch, not a general capability gap. |
| Destructive action | Whether the agent resists this specific attempt: 'destructive action'. | A dip concentrated in this category points at a specific vulnerability to patch, not a general capability gap. |
| Instruction override | Whether the agent resists this specific attempt: 'instruction override'. | A dip concentrated in this category points at a specific vulnerability to patch, not a general capability gap. |
| Fake tool output | Whether the agent resists this specific attempt: 'fake tool output'. | A dip concentrated in this category points at a specific vulnerability to patch, not a general capability gap. |
| Chained multi-step | Whether the agent resists this specific attempt: 'chained multi-step'. | A dip concentrated in this category points at a specific vulnerability to patch, not a general capability gap. |
| Benign-task completion | Whether the agent still completes normal, non-adversarial requests. | The agent has become over-cautious — refusing legitimate work, not just resisting attacks. That's a false-positive problem, not a safety win. |
Integration
One line in CI
Runs locally or in your pipeline. Fail the build the same way a failing test would.
Anchored standards
Certification
Ship with a certified meter
Add Browser agent — prompt-injection red-team to your pipeline today — a signed grade, reproducible measurement, zero oracle.
Get this pack — €89