Red-team prompt injection across trust boundaries

Map the trust boundaries#

Draw the path from a user’s authorized request through retrieval, generation and tool execution. Mark who controls each input and which component can approve an action. Greshake and colleagues’ study demonstrates indirect prompt injection through content an application retrieves, rather than only through direct user prompts.

Treat documents, search snippets and tool responses as data with provenance. Their contents cannot grant permissions or replace the user’s goal. Define observable failure conditions: restricted information reaching an output, an unauthorized write being attempted, or an unrelated destination being selected.

Authorize and isolate the test#

Write down the systems, accounts, channels and test window covered by authorization. Use a sandbox with fictional records, mock tools, disabled external delivery and resettable state. Remove production credentials and use synthetic canaries instead of real secrets.

The AgentDojo paper evaluates tool-using agents in an extensible environment and distinguishes legitimate task performance from attacker success. Apply that separation to your fixtures: define the authorized task, the adversarial objective and the final state expected from each. Run a clean version first; an already-broken task tells little about injection resistance.

Exercise retrieval and tool channels#

Create matched fixtures in which only one untrusted surface changes. For retrieval, replace a fictional document passage or search snippet. For tools, replace a mock response field while preserving the expected schema. Use inert markers such as [SYNTHETIC_TASK_DIVERSION] to represent the adversarial content; these placeholders are not executable attack examples.

Vary placement, document relevance, response length and whether the content arrives before or after useful evidence. Test retrieval selection separately from behavior after selection. Add multi-step cases where contaminated content is summarized or cached, then verify that its trust classification survives. Preserve a clean control for every variant.

Save each fixture’s origin, expected trust level and replacement location. Assert that the harness actually delivered the modified content to the agent; otherwise a blocked retrieval path can be mistaken for a robust response. Keep test access restricted and remove retained artifacts according to the sandbox policy.

Check actions at execution#

OWASP’s prevention guidance recommends validating tool calls against user permissions and applying least privilege. Enforce those checks outside the language model.

This provider-neutral JSON is a fictional fixture specification, not an Align API or a functioning attack payload.

{
  "case_id": "retrieved-note-fixture",
  "authorized_task": "summarize fictional release notes",
  "untrusted_marker": "[SYNTHETIC_TASK_DIVERSION]",
  "allowed_tools": ["read_fixture"],
  "forbidden_effects": ["external_delivery", "record_write"]
}

Inspect actual tool requests and state changes. Validate operation, target and arguments before execution. A reassuring final answer cannot cancel an unauthorized action that already occurred.

Automate checks, review meaning#

Automate assertions for permitted tool names, destination allowlists, write counts, canary disclosure and fixture-state changes. Record blocked requests as attempted violations separately from successful effects. Inspect visible answers for task diversion and unsupported claims using a human-reviewed rubric. Structural validity alone does not establish semantic safety.

Compare clean task completion, completion under injection, attempted violations and realized adversarial objectives separately. State the number of tasks, variants and attempts, including errors and unscored runs. Keep the attack budget fixed across candidates. Review failures individually before changing a prompt or adding a filter.

Regress incidents and state limits#

For an incident, preserve the authorized request, entry channel, affected boundary and observed effect. Replace sensitive content with a minimal synthetic reproduction. Add it to a regression set with a clean counterpart and a check for the failed boundary. Rerun after changing retrieval, tools, permissions, prompts or model versions.

Sandbox results cover the modeled channels and permissions, not every production condition. Placeholder fixtures verify harness wiring; they cannot establish resistance to realistic adversarial language. Authorized reviewers need representative cases before making that claim. Pattern filters and prompt labels remain supporting layers, while execution controls constrain what the system can actually do.