LLM output security checklist
- Secure LLM outputs by treating them as untrusted input: validate them against strict schemas, apply context-aware encoding or sanitization before each downstream use, filter for harmful content and data leakage, restrict tools and privileges, sandbox execution, and require human approval for high-impact actions.
- For web output, contextually encode HTML, JavaScript, URLs, and other rendered content.
- For database use, use parameterized queries or prepared statements; never concatenate model text into SQL.
- For shell commands or code, prefer fixed allowlisted operations, validate arguments, and execute only in an isolated sandbox.
- For structured responses, enforce JSON Schema, Pydantic, or equivalent parsing and reject extra keys, malformed data, and unexpected commands.
Secure LLM outputs by treating them as untrusted input: validate them against strict schemas, apply context-aware encoding or sanitization before each downstream use, filter for harmful content and data leakage, restrict tools and privileges, sandbox execution, and require human approval for high-impact actions.
Every LLM output crosses a trust boundary before it is displayed, stored, executed, or passed to another system. No single filter, schema, or escaping step makes generated content safe for every use.
1. Treat every LLM output as untrusted data
Apply this rule to generated text, structured fields, code, queries, URLs, tool arguments, and summaries of retrieved content. A polished response can still contain harmful material, sensitive information, an unauthorized action, or instructions that affect a downstream interpreter.
Insecure or improper output handling occurs when model-generated content reaches a user or downstream component without the validation, destination-specific protection, authorization, review, and monitoring required for that use.
This applies to content that is:
- Rendered in a browser or sent to another person
- Passed to application logic as structured data
- Stored in a database or used in a query
- Used as a command, script, code fragment, or file operation
- Sent to an API, plugin, or agent tool
- Retrieved from an external source and echoed by the model
2. Map the output to its destination
Identify the receiving consumer before selecting a control. Ordinary text, HTML, SQL values, shell arguments, API parameters, and tool calls have different security boundaries. Apply protection at the point where the output will be interpreted or used. For testing AI-facing API boundaries, see AI API penetration testing.
| Destination | Validate | Protect and restrict | Approval and monitoring |
|---|---|---|---|
| Browser or rendered content | Check the expected type, fields, and permitted content. | Use context-aware encoding for HTML, JavaScript, URLs, and other rendered contexts. Do not directly render unvalidated output as active content. | Log transformations, blocks, and display decisions. Review sensitive or externally published content. |
| Database | Validate types, values, fields, and permitted operations. | Use parameterized queries or prepared statements. Never concatenate model text into SQL. | Use narrowly scoped database permissions and log rejected and executed operations. |
| Shell or generated code | Prefer fixed operations and validate every argument against explicit allowlists. | Do not execute generated commands or code directly. Use least privilege and an isolated sandbox when execution is required. | Require approval for consequential operations and record the proposed and actual action. |
| Structured response | Parse with JSON Schema, Pydantic, or an equivalent validator. Reject malformed data, extra keys, invalid values, and unexpected commands. | Pass only the validated representation to the next component. Schema validity does not prove final-use safety. | Track validation failures, retries, blocks, and downstream outcomes. |
| API, plugin, or agent tool | Validate the selected operation, target, identifiers, arguments, and side effects independently of the model’s explanation. | Expose only approved actions with narrowly scoped tokens, permissions, network access, and data access. | Require approval for high-impact calls and log proposed calls, decisions, results, and blocks. |
3. Validate structure before downstream use
Define the expected response before calling the model. Use a strict schema or structured parser to validate the type, required fields, field lengths, permitted values, operation names, identifiers, and argument types.
Use enums for finite choices, narrowly defined patterns where appropriate, and allowlists for permitted operations, resources, destinations, and tool names. For structured responses, enforce JSON Schema, Pydantic, or an equivalent validator. Reject extra keys unless they are explicitly allowed.
Fail closed when the response cannot be parsed, violates its schema, contains an unexpected command, requests an unapproved operation, or falls outside an allowlist. Block it, quarantine it for review, or request a new response; do not silently pass it onward.
Structural validation is only the first layer. A schema-valid string can still be unsafe when it reaches a browser, query engine, shell, API, or privileged tool. Validate it again at the final consumer boundary.
4. Encode or sanitize at the point of use
Use protection that matches the destination. Browser-facing output requires context-aware encoding for generated HTML, JavaScript, URLs, and other browser content so text is not interpreted as executable content.
Unsafe: inserting a chatbot response directly into an HTML page. Safer: treat the response as text, encode it for the specific context, and permit only the rendering behavior the application requires.
There is no universal sanitizer. Protection suitable for HTML is not automatically suitable for JavaScript, a URL, SQL, a shell, an API argument, or a plugin interface. Apply the appropriate encoding or sanitization immediately before each downstream use; a value protected for one consumer may not be safe for another.
5. Secure queries, commands, code, and tool calls
Database queries
Never concatenate model-generated text into SQL. Use parameterized queries or prepared statements, validate the intended operation and values, and limit the database account to the smallest necessary permissions.
Unsafe: allowing the model to construct a query string. Safer: select the permitted operation in application logic, validate each value, and bind those values through the database interface.
Shell commands and generated code
Do not execute generated commands or code directly. Prefer fixed, allowlisted operations selected by application logic, then validate every argument. If execution is necessary, use an isolated sandbox and limit filesystem, network, credentials, tools, and runtime permissions.
Unsafe: passing the model’s command string to a shell. Safer: map an approved operation to a predefined command, validate its arguments, run it with least privilege in a sandbox, and stop for approval when the action is consequential or irreversible.
Tool selection is not authorization
A model may propose a tool based on its available descriptions, but its selection and explanation are not authorization. Expose only permitted tools, constrain their operations, independently validate every argument and target, and check permissions before the call.
Block or escalate the call when validation, authorization, sandboxing, or approval fails. A model-selected tool must not obtain broader access merely because the request appears relevant. For a broader treatment of permissions and authorization, see identity and access management for AI applications.
6. Filter harmful content, secrets, and injection indicators
Use output filters, moderation, classifiers, or guardrails to identify harmful, inappropriate, biased, false, misleading, or sensitive content. Also screen for secrets, suspicious instructions, and signs that retrieved content or a prompt-injection attempt is being echoed into an output or action.
These checks can support decisions to block, rewrite, flag, or review a response before use, but filtering alone is insufficient. A classifier can miss unsafe content, and a benign-looking response can become dangerous when passed to an interpreter or privileged tool. Keep schema validation, context-specific protection, authorization, restricted execution, and human review in place.
Handle retrieved content and leakage separately
Documents, web content, files, and other external sources can contain instructions that influence a model’s response. Treat retrieved content as untrusted, keep data separate from instructions, validate every proposed action, and authorize it independently of the model’s reasoning.
Output encoding protects a destination from interpretation problems; it does not determine whether the response should disclose sensitive information. Leakage requires access controls, data-handling rules, filtering, review, and monitoring in addition to sanitization. For targeted prevention guidance, see our guide to sensitive information disclosure in LLM applications.
7. Require approval for consequential actions
Require human or domain-expert approval before an LLM-driven action is executed when it is high impact, irreversible, sensitive, or difficult to undo. This includes financial transactions, medical decisions, access changes, deletion, and actions affecting a person’s rights or resources.
Review content before it is published, sent externally, or used in a sensitive workflow. Approval must be a real control point: stop when it is missing, rejected, expired, or no longer matches the proposed action. Do not allow the model to approve its own output.
8. Log, monitor, and fail closed
Record the destination, validation result, transformations, filtering result, authorization decision, approval status, execution or delivery outcome, and reason for any block or rewrite. Protect these logs because they may contain sensitive output.
Monitor validation failures, blocked content, rejected tool calls, approval overrides, repeated retries, and unexpected destinations. Fail closed when a required validator, filter, authorization check, approval step, or sandbox is unavailable. A temporary inability to verify output should block the downstream action rather than create an implicit bypass. For a broader monitoring checklist, see AI security monitoring.
9. Test each output-handling path
Verify that browser output is encoded for its context, database values use parameterized handling, commands and code are restricted, structured responses reject unexpected fields, and tool calls cannot exceed their permissions.
Test malformed structures, unexpected commands, harmful content, sensitive information, unauthorized resources, retrieved instructions, prompt-injection attempts, and attempts to bypass approval. Confirm that parsing failures, unavailable filters, denied permissions, failed sandboxes, and rejected approvals cannot use a fallback path. For adversarial testing approaches, see AI red teaming versus penetration testing.
A warning is not enough. The workflow should block, safely rewrite, quarantine, or replace unsafe output with a controlled response—and log the event.
Practical pre-use checklist
- Treat the output as untrusted data. Apply this to text, structured values, code, queries, URLs, and tool arguments.
- Identify the destination. Determine whether the value will be rendered, stored, interpreted, executed, sent to an API, or shown to a person.
- Validate the structure. Use schemas, parsers, enums, patterns, and allowlists; reject malformed or unexpected results.
- Apply context-specific protection. Encode or sanitize at the point of use, and use parameterized database handling and safe command practices.
- Filter the content. Check for harmful material, secrets, leakage, suspicious instructions, and injection indicators, while treating filtering as supplementary.
- Restrict permissions and execution. Limit tools, APIs, databases, networks, shells, tokens, and actions; sandbox generated code or tool calls when execution is required.
- Obtain approval where needed. Stop for human or domain-expert review before high-impact, irreversible, sensitive, or externally disseminated actions.
- Log, monitor, and test. Verify that failures block or escalate rather than bypass controls.
Block or escalate whenever the destination is unclear, validation fails, the output contains an unexpected operation, filtering identifies a serious concern, authorization is insufficient, execution cannot be isolated, or required approval is absent. This layered process reduces the impact of unsafe output without claiming that any single control can make an LLM response safe.
