Prompt Injection and Instruction Handling

Prompt injection is the risk that untrusted input influences model behavior in an unintended way. Direct prompt injection comes from a user request. Indirect prompt injection can arrive through retrieved documents, uploaded files, tool outputs, emails, tickets, web pages, or other content that the application places into context. The defensive lesson is not to memorize payloads. The lesson is to recognize that natural language from untrusted sources can conflict with trusted instructions and application policy.

Instruction hierarchy is a design concept, not a magic shield. Trusted system and developer instructions can define the application's intended behavior, but retrieved or uploaded content should be labeled as data and handled according to its source and authority. Provenance and source labeling help the model and the user understand where content came from. More importantly, they help the application decide which content can affect answers and which content cannot affect permissions, tool access, or policy.

Wording alone is not a reliable security boundary. A prompt can say that documents are not allowed to change rules, but the Python application still needs to enforce authorization, data access, tool permissions, and approval gates outside the model. The model may help classify intent or summarize evidence, but code should decide whether a user can see a record, whether a tool call is allowed, whether an action is high impact, and whether human review is required.

Sensitive decisions should include review and constraint. For example, an AI assistant may draft an approval message, prepare a ticket, summarize evidence, or suggest a configuration change, but the application should require explicit confirmation for high-impact actions. Separating instructions from data, preserving provenance, limiting tool authority, validating arguments, and requiring human approval for sensitive operations reduces the chance that untrusted text becomes unreviewed control over the system.