Detect prompt injection inside an uploaded business document

Prompt injection occurs when untrusted content attempts to redirect an assistant's behavior. In an uploaded business document, the suspicious content may ask the assistant to ignore its task, alter a reported number or claim approval it does not have. Recognizing those instructions is useful, but detection alone is not a reliable permission boundary.

Design the workflow so document text cannot grant itself tool access, change the metric contract or approve a report. The exercise below uses a harmless, explicitly seeded instruction and no live language model.

Identify the attempted authority change

The original glossary fixture includes an untrusted note telling the reader to report 900,000 paise and mark the answer approved. The valid source calculation is 104,000 paise.

The problem is not merely that the number is wrong. The document is trying to change calculation rules and review authority. Those are application decisions, not facts that an uploaded paragraph may redefine.

OWASP's prompt-injection guidance describes layered controls for this class of problem. Its LLM01 overview also explains why retrieval by itself does not eliminate the risk.

Test the boundary in a small fixture

from copy import deepcopy
from glossary import documents,retrieve,check_citation
from calculator import calculate,seeded_cases,check_candidate

note = next(d for d in documents() if d['document_id']=='uploaded-note-untrusted')
assert note['approved'] is False
assert '900000' in note['definition']
selected = retrieve('order amount')['documents'][0]
assert selected['document_id']=='completed-orders-v1'
assert not check_citation(note['document_id'],'completed_order_amount')['metadata_checks_passed']
assert calculate()['value']==104000
tampered = deepcopy(seeded_cases()[0]['answer'])
tampered['value']=900000
assert 'value_mismatch' in check_candidate(tampered)['issues']
print({'untrusted_note_excluded':True,'reference_amount_paise':104000,
       'seeded_wrong_amount_rejected':True,'live_model_calls':0})

This test establishes that the local retrieval policy excludes the unapproved record and that the deterministic checker rejects its suggested amount. It does not measure how a particular model responds to the note, nor prove resistance to every injection technique.

Keep policy metadata outside document control

The fixture's approval flag is trusted local configuration. In a real ingestion system, do not let an upload set its own approval status by including a field or sentence saying “approved.” Assign source trust through a controlled process and preserve provenance.

Even an approved document can contain quoted hostile text, outdated instructions or an accidental instruction-like passage. Trusting a source for business facts does not give its prose authority to override tool permissions or reporting policy.

The bounded SQL tool uses application code to restrict columns and operations. The review component requires a separate record tied to the report version. Neither accepts a document sentence as authorization.

Use detection as a review signal

Phrases requesting instruction overrides, secret disclosure or bypassed review can be useful signals. A phrase scanner can also miss paraphrases and flag legitimate discussions of attacks. Treat detection output as evidence for review, not a complete security decision.

Preserve the suspicious passage and its source identifier in a controlled diagnostic record. Avoid copying confidential document contents into unrestricted logs while investigating. For this public teaching exercise, every record is fictional.

Evaluate more than one visible attack

Extend the fixture with a wrong-unit instruction, a request to use a superseded definition and a claim that review has already happened. Test whether deterministic controls still enforce the contract regardless of the wording.

If a real model is added later, record its exact response and tool attempts under the same source and policy versions. Report tested cases and failures without claiming that a finite test suite solves prompt injection generally.

Exercise: rewrite the seeded note without using the word “ignore.” Explain why a keyword detector may change its result while the source calculator and tool permissions should remain unchanged.

NeuraPath's Data Analytics with Generative AI course connects document-assisted analysis with bounded tools and evidence checks. The analyst should recognize when source material is supplying facts and when it is trying to take control of the workflow.

Continue learning

This article is part of the Generative AI for verified analyst work sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Build an AI report review workflow with human sign-off.
• Continue with Use synthetic data when demonstrating AI analysis.