Threat-model tool misuse and data exfiltration paths

An agent connects untrusted language to capabilities. Threat modelling must follow the complete path from user and retrieved content through model output, tool arguments, credentials, network destinations and stored results.

Define the measurable control

Draw trust boundaries and assets. Enumerate direct and indirect prompt injection, excessive agency, SSRF, confused deputy behavior, cross-tenant access and covert exfiltration. For each path, reduce capability, validate typed arguments, authorize at execution, constrain egress, require approval and write an adversarial test.

The AI reliability lab makes the decision reproducible with authored data:

from reliability_cases import threat_model_case

result = threat_model_case()
assert result["tested"] == 4
assert result["model_as_trust_boundary"] is False
assert len(result["threats"]) == 4
assert result["residual"] == ["authorized user may export permitted data"]

The synthetic model covers four paths and explicitly refuses to treat the model as a security boundary. All four controls have tests, while one residual risk remains: an authorized user can export data they are legitimately permitted to access. That risk needs policy and monitoring, not a fictional perfect control.

Challenge the result

A prompt that tells the model to ignore instructions is not an enforcement mechanism. Put controls outside the model. Also test encoded destinations, redirects, DNS changes, tool chaining and data fragments spread across several calls.

Keep a reviewable evidence pack:

• data-flow and trust-boundary diagram
• asset and abuse-case register
• control mapped to an executable adversarial test
• residual-risk owner and review date

This work aligns with the evaluation, security, cloud operations, reliability and FinOps sequence in the FDE for Professionals course. The linked course describes the learning pathway; this article’s numbers are synthetic and do not report a model, client, audit or production result.

Practice task

Choose one read tool and one write tool. Trace how a malicious document could influence each argument and block the path without relying on model obedience.

Continue learning

This article is part of the Advanced AI reliability and assurance sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Test cross-tenant cache isolation.
• Continue with Build an outbound network allowlist for AI tools.

Reference: OWASP Top 10 for LLM Applications 2025.