LLM Security: A Complete Guide to Securing AI Applications

Secure AI applications with layered defenses: map attack surfaces and data flows; protect prompts, context, models, and sensitive data; treat outputs as untrusted; constrain tools with least privilege and sandboxing; validate inputs and outputs; limit rate and cost; monitor continuously; and threat-model, red-team, and regression-test.

OWASP’s LLM recommendations and the NIST AI RMF provide useful supporting guidance, but neither replaces application-level controls. For a focused framework overview, see our OWASP Top 10 for LLM Applications guide.

What LLM security covers

LLM security is broader than jailbreak prevention. It is an application-security program for systems in which language models interact with users, prompts, context, enterprise data, retrieval sources, tools, APIs, infrastructure, and downstream workflows.

• Models and endpoints: hosted or self-managed models, gateways, APIs, providers, and model versions.
• Prompts and context: system instructions, user input, conversation history, retrieved material, and application context.
• Data: PII, secrets, proprietary code, enterprise records, training data, retrieval stores, outputs, and logs.
• Users and identities: end users, administrators, developers, service accounts, and identities initiating automated actions.
• Tools and workflows: APIs, databases, files, code execution, browsers, transactions, and other downstream operations.
• Infrastructure and dependencies: networks, containers, cloud services, libraries, third-party models, plugins, and datasets.
• Runtime behavior and governance: model interactions, tool calls, failures, usage, spend, ownership, policies, testing, and incident response.

Traditional identity, API, network, and software-security controls remain necessary. Natural-language inputs and model responses also require careful handling: both are security-relevant application data, not trusted instructions.

Build the threat model before deployment

Inventory the complete AI application rather than examining the model in isolation. Map the path from user input through prompt construction, retrieval, model endpoints, tool calls, downstream systems, and the final response. For a structured approach, see our AI threat modeling guide.

Map components and data flows

• Users, administrators, service accounts, and initiating identities.
• Web, mobile, internal, and API entry points.
• Prompt templates, system instructions, conversation memory, and context assembly.
• Model endpoints, gateways, providers, versions, and fallback models.
• Retrieval sources, document stores, indexes, and data entering context.
• Tools, APIs, databases, file systems, external destinations, and workflow actions.
• Secrets, PII, proprietary code, enterprise records, and other sensitive data.
• Third-party models, plugins, libraries, datasets, and build dependencies.
• Logs, audit records, monitoring systems, budgets, and administrative controls.

Identify trust boundaries and high-impact workflows

Mark every point where data, authority, or trust changes. Important boundaries include the user and application, retrieved documents and system instructions, the model and a tool, a service account and a business system, and development and production environments.

Prioritize workflows that can expose sensitive information, change records, send communications, move money, alter infrastructure, or materially affect people. Revisit the threat model whenever the team changes a model, prompt, retrieval source, tool, workflow, or dataset.

Core threats to LLM and AI applications

Prompt injection matters, but it is not the entire AI application security problem. Evaluate the following risks separately so less visible threats receive appropriate attention.

Use defense in depth across the architecture

No single guardrail, content filter, AI firewall, or framework secures an application by itself. Apply controls across the full system:

• Discovery: inventory components, identities, dependencies, data flows, tools, and trust boundaries.
• Policy enforcement: define who may use each model, data source, tool, destination, and workflow.
• Application layer: validate inputs, separate instructions from untrusted content, validate outputs, and prevent direct execution of model text.
• Model and gateway layer: control endpoints, versions, access, usage, and model changes.
• Retrieval and data layer: restrict access to sources, minimize context, and treat retrieved content as untrusted.
• Tool and workflow layer: authorize each action, restrict destinations, check parameters, and require approval for consequential operations.
• Infrastructure layer: protect networks, containers, credentials, environments, and dependencies.
• Runtime and audit layer: monitor interactions, identities, policy events, tool calls, failures, anomalies, and spend with privacy safeguards.
• Organizational layer: assign owners, maintain incident procedures, test continuously, and retain governance evidence.

Secure inputs, prompts, context, and retrieval

Validate inputs and separate instructions

Validate and constrain user input before it enters the application. Apply suitable limits to request size, context, and token use. Keep application instructions distinct from user-provided and retrieved material so arbitrary text is not implicitly treated as an authorized instruction.

Defend against prompt injection with multiple layers: validation, policy checks, access controls, output handling, and permission boundaries. Do not rely on a single detector or a specially worded prompt. If the model is manipulated, least-privilege permissions should still limit the consequence.

Handle retrieval sources as untrusted content

For retrieval-augmented applications, documents and retrieved passages are untrusted application data. Apply access control before information enters context, retrieve only what the task requires, and preserve the distinction between application instructions and source content. Test retrieval paths for cross-user access and unintended disclosure.

Protect sensitive data and privacy

Protect data throughout its lifecycle: before it enters a prompt, while it is assembled into context, when it is stored for retrieval, when it appears in a response, and when activity is logged.

• Restrict access: enforce identity-based access to data, models, retrieval sources, tools, and administrative functions.
• Minimize context: send only the information required for the task and omit unnecessary sensitive fields.
• Reduce exposure: use masking, redaction, tokenization, or anonymization where appropriate.
• Protect stored and transmitted data: apply encryption and suitable storage protections.
• Control retention: define how long prompts, context, outputs, retrieval records, and logs are kept.
• Scan outputs: check for sensitive information and policy violations before responses reach users or systems.
• Protect logs: restrict telemetry access and avoid collecting more sensitive content than the operational purpose requires.

Security and privacy teams should decide which data may enter each application, endpoint, retrieval source, and log. Encryption and access controls complement minimization and retention policies; they do not eliminate the need to limit sensitive context. For practical safeguards, see our guide to preventing LLM data leakage.

Treat model outputs as untrusted

Model output must not be passed directly into privileged commands or sensitive downstream operations. A response can be incorrect, manipulated, incomplete, or unsafe even when content filters are enabled. For validation and safe-execution details, see our LLM output security checklist.

Before output reaches a user or system:

• Validate its format, type, length, and expected fields.
• Apply content and policy checks for harmful, leaked, or unacceptable material.
• Use explicit authorization outside the model for every sensitive operation.
• Keep generated values separate from executable commands and privileged control parameters.
• Require confirmation or human review for consequential actions.
• Record the initiating identity and relevant policy decision for audit and investigation.

The downstream application must enforce the final security boundary. A model may propose an action, but it must not independently decide whether that action is authorized.

Secure identity, access, secrets, and infrastructure

Apply least privilege

Give each user, service, model endpoint, tool, and workflow only the access it needs. Use scoped permissions rather than shared administrator access. Keep API keys in a secret manager, separate credentials by environment, rotate keys regularly, and monitor unusual access or usage.

Do not place secrets in prompts, source code, logs, retrieved documents, or model-generated responses. Treat model and tool credentials as application credentials: protect them, limit their scope, and revoke or rotate them when exposure is suspected.

Protect environments and infrastructure

Separate development, testing, and production environments. Apply network controls, isolation, private connectivity where appropriate, container and host security, dependency management, and controlled deployment processes. Track model, prompt, tool, configuration, and dataset versions so changes can be identified, investigated, and rolled back where appropriate.

Secure agents, tools, APIs, and workflows

Agent security is primarily an authorization and containment problem. Do not grant broad autonomous access merely because a model can describe a desired action.

• Allowlist tools and destinations: expose only approved capabilities and target systems.
• Scope permissions: limit actions by user, tenant, resource, operation, environment, and time where appropriate.
• Check before execution: validate parameters, destinations, requested scope, and policy before consequential calls.
• Attribute actions: associate each tool call with the initiating user, service, workflow, and authorization decision.
• Contain execution: use sandboxing or other containment when workflows handle untrusted content or affect systems.
• Add approval gates: require human approval or an equivalent additional control for high-impact actions.
• Limit repetition: apply rate, token, quota, and workflow limits to reduce misuse and runaway cost.

Keep authorization outside the model. The model may propose an action, but a separate policy and access-control layer should decide whether it is permitted.

Protect the model, data, and AI supply chain

Review every external component that can influence the application: hosted or open-source models, plugins, libraries, datasets, build tools, and service providers.

• Perform supplier and dependency due diligence.
• Track model, dataset, library, and configuration provenance.
• Maintain software bills of materials where supported.
• Use checksums and controlled artifact sources.
• Prefer reproducible builds where supported by the environment.
• Version models, prompts, datasets, retrieval content, and tools.
• Review changes before production and test their security impact.

Treat data or training-data poisoning as an integrity and supply-chain risk. Provenance, review, versioning, and regression testing help identify unexpected changes without depending on one detection mechanism.

Monitor, log, and control runtime behavior

Production monitoring should cover more than uptime. With suitable privacy safeguards, monitor model interactions, identities, policy events, tool or agent actions, failures, access patterns, usage, and spend. For runtime visibility, see our AI security monitoring guide.

• Privacy-conscious logs: collect necessary security and operational evidence while limiting unnecessary sensitive content.
• Audit trails: record relevant identity, authorization, policy, tool, and workflow events.
• Rate and token controls: limit requests and context or output consumption by user, service, or organization.
• Budgets and quotas: set per-user or organizational spending and usage limits.
• Throttling: slow or reject activity that exceeds defined limits.
• Anomaly detection: look for unusual access, disclosure patterns, tool activity, failures, or spend.
• Alerts: route high-risk events to owners who can investigate or contain them.

Do not assume that logging every prompt, hidden instruction, tool call, or output is always appropriate. Define retention, access, redaction, and review rules that balance investigation needs with privacy and data minimization.

Test continuously and red-team responsibly

Security testing must continue as prompts, models, tools, workflows, and data change. Combine architecture review, threat modeling, adversarial testing, red teaming, and regression tests. For a practical testing process, see our LLM application red-teaming guide.

At a bounded and authorized level, test for:

• Prompt extraction and system-instruction leakage.
• Prompt injection and control-bypass behavior.
• Cross-user or cross-tenant data access.
• Sensitive-data disclosure through prompts, retrieval, outputs, or logs.
• Unsafe or unauthorized tool use.
• Weak authorization and missing approval gates.
• Rate-limit circumvention and excessive usage.
• Cost exhaustion and denial-of-wallet conditions.
• Unexpected behavior after model, prompt, tool, workflow, or data changes.

Turn important findings into regression tests. A control that worked for one model or prompt version should not be assumed to work after the application changes.

Prepare incident response and recovery

Adapt the organization’s incident-response process to AI-specific assets and failure modes. Prepare owners and actions before an incident occurs.

1. Contain: disable or restrict the affected endpoint, workflow, tool, or integration when necessary.
2. Protect access: rotate or revoke exposed credentials and review related permissions.
3. Investigate: examine available logs, identities, policy events, model interactions, data flows, and tool activity.
4. Assess impact: determine which users, data, systems, costs, or downstream actions may be affected.
5. Communicate: notify relevant users, system owners, security teams, and other stakeholders according to established procedures.
6. Recover: validate corrected permissions, prompts, models, configurations, or workflows before restoring normal operation.
7. Improve: document the cause, update tests and controls, and review whether similar systems are exposed.

Use frameworks as supporting structures

Frameworks help organize risks, controls, ownership, and evidence, but they do not replace application-level security work.

No framework, certification, audit report, guardrail, or AI firewall secures an application by itself. Use these references to support risk identification, control design, testing, ownership, and evidence while enforcing the actual controls in the application and its operating environment.

Prioritized LLM security implementation checklist

Design

• Inventory models, endpoints, prompts, context, retrieval sources, tools, users, dependencies, and sensitive data.
• Map data flows, trust boundaries, identities, and high-impact workflows.
• Threat-model prompt injection, disclosure, poisoning, supply-chain risk, excessive agency, and cost abuse.
• Define owners, acceptable use, retention, approval, and incident-response requirements.

Pre-production

• Protect sensitive data through access control, minimization, redaction or tokenization, encryption, and retention controls.
• Implement environment-separated secrets, scoped permissions, allowlisted tools, and containment.
• Validate inputs and outputs and separate instructions from untrusted retrieved content.
• Prevent model output from directly reaching privileged commands or sensitive operations.
• Set rate limits, token caps, quotas, budgets, alerts, and privacy-conscious audit logging.
• Run threat modeling, adversarial tests, red-team exercises, and regression tests.

Launch

• Confirm monitoring for identities, policy events, tool calls, failures, access patterns, usage, and spend.
• Verify approval gates and attribution for consequential actions.
• Test endpoint disabling, credential rotation, communications, and recovery validation.

Continuous operations

• Reassess the system whenever prompts, models, tools, workflows, retrieval sources, or data change.
• Review anomalies, security events, costs, permissions, dependencies, and governance evidence.
• Repeat adversarial and regression testing and update controls from findings.
• Maintain incident-response and governance records without treating compliance status as proof of security.

Use this sequence: map first, protect data and permissions, validate inputs and outputs, constrain tools, monitor and limit runtime behavior, test every material change, and maintain incident-response evidence. That is the foundation of a durable LLM security program.