How to Penetration Test an AI API: A Safe, Practical Checklist

Penetration test an AI API in two layers: first perform authorized standard API security testing using the OWASP API Security Top 10, then test AI-specific behavior and integrations using frameworks such as the OWASP Top 10 for LLM Applications and MITRE ATLAS. Scope the architecture and data flows first.

Obtain written authorization before testing and define the targets, environments, accounts, data, test window, monitoring, and stop conditions. Unauthorized AI API testing may be illegal, disruptive, expose data, or create unexpected provider costs.

The two-layer method for testing an AI API

Layer 1: Conventional API security testing

Assess the service as an ordinary application interface before focusing on model behavior. Review authentication, authorization, input handling, data exposure, business logic, configuration, API inventory, and resource consumption. An AI feature does not replace these controls; it adds another security boundary.

Layer 2: AI-specific behavior and integrations

Then assess prompts, retrieved content, model responses, output handling, logs, tools, and downstream integrations. The sequence depends on whether the service uses a managed provider, self-hosted or fine-tuned model, retrieval-augmented generation (RAG), plugins, or external tools.

1. Confirm authorization, scope, and safety controls

Define written authorization and rules of engagement

• Obtain written permission from the system owner.
• List target domains, API versions, endpoints, model routes, integrations, and excluded systems.
• Specify whether testing is limited to staging or may include production, and document the approved test window.
• Identify approved test accounts, roles, credentials, source addresses, and authentication methods.
• Set request-rate, token, quota, and budget limits to control resource use and provider charges.

Security intent does not make testing automatically lawful. Owner permission and a defined scope are prerequisites, while applicable laws, contracts, and provider terms vary by jurisdiction. Seek appropriate legal advice for questions about your specific testing activity.

Protect accounts, environments, and data

Use dedicated accounts and synthetic or otherwise approved data wherever possible. Confirm that logging, alerting, usage monitoring, and rollback procedures are active. Stop if you observe service degradation, unexpected data exposure, unsafe model behavior, abnormal cost, or contact with an out-of-scope system.

Keep validation non-destructive. Do not steal credentials, extract real sensitive data, delete records, evade monitoring, or run uncontrolled tests against production or third-party systems.

2. Map the AI API architecture and data flows

Identify deployment and model boundaries

Document the components that handle each request and response:

• The public API gateway, application services, and authentication layer.
• The managed model provider, self-hosted model, or fine-tuned model.
• Model routes, system and application prompts, and configuration controls.
• Logs, monitoring systems, storage, and support or debugging interfaces.
• RAG components, vector stores, retrieved documents, and tenant boundaries where present.
• Plugins, external tools, service accounts, and downstream applications that can receive model output.

Trace each request through authentication, prompt construction, retrieval, model processing, output filtering, tool invocation, and logging. Identify which identity and tenant context crosses every boundary.

A text-generation endpoint has a different assessment path from an AI API that retrieves private documents or invokes external services. For managed providers, focus on provider access, keys, model routes, output handling, and application controls. For self-hosted or fine-tuned systems, include the model-serving boundary, configuration, access controls, and data-handling paths in scope.

3. Discover and inventory the API attack surface

Use specifications and observed traffic

Build an API inventory from every available source:

• OpenAPI or Swagger specifications and Postman collections.
• HAR files and observed browser or application traffic.
• Documentation paths, crawling, and JavaScript analysis.
• GraphQL introspection where enabled and in scope.
• Known model routes, webhooks, administrative interfaces, and integration endpoints.

Record each endpoint, method, content type, parameter, response, authentication requirement, role, and API version. Look for undocumented methods, alternate content types, hidden parameters, retired APIs, shadow APIs, and inconsistent behavior between versions. Confirm unexpected endpoints with the owner rather than assuming that undocumented means out of scope.

Preserve sessions and response values across multi-step workflows. Creating, viewing, editing, and submitting a resource may expose chained authorization or business-logic weaknesses that isolated requests miss.

4. Test conventional API security controls

Authentication and authorization

Use approved test accounts to compare permissions across users and roles. Assess:

• Broken authentication, including API keys, JWTs, OAuth flows, sessions, token handling, and expiration behavior.
• Broken object-level authorization (BOLA/IDOR) by checking whether one authorized account can access another account’s permitted objects.
• Function-level authorization, including whether lower-privileged users can reach administrative or restricted functions.
• Property-level authorization, including whether protected fields can be changed.
• Mass assignment, using benign approved requests to verify that hidden or administrative properties remain server-controlled.

Input, exposure, and configuration risks

Assess injection handling, input validation, security misconfiguration, excessive data exposure, SSRF where relevant, undocumented methods, and unsupported content types. Review whether errors, debugging responses, model metadata, logs, or API responses disclose more information than the caller needs.

Test supported HTTP methods, content types, parameters, and request properties for inconsistent processing or authorization. Review every model route as an API endpoint, including chat, completion, embedding, retrieval, administration, and integration paths when present.

Rate limits, resource consumption, and cost exposure

Test rate limits, quotas, 429 handling, concurrency boundaries, request and response size limits, token consumption, provider usage, and potential denial-of-wallet exposure. Use small, controlled request volumes and stop before service degradation or unexpected billing occurs.

Stateful workflows and business logic

Test complete workflows rather than only individual endpoints. Verify that identifiers, roles, ownership checks, and state transitions are enforced at every step. Functional testing confirms expected behavior, but it does not replace vulnerability assessment or penetration testing.

5. Test AI-specific behavior and integrations

Direct and indirect prompt injection

Assess direct prompt injection through the API request and indirect prompt injection through content the system retrieves or processes, where those features exist. Use benign test instructions to determine whether untrusted content can override intended instructions or cross a trust boundary.

Prompt, model, and sensitive-data leakage

Check whether system prompts, internal instructions, sensitive application data, model configuration, or other tenants’ content can be disclosed. Validate isolation and leakage controls without extracting real secrets or attempting to obtain protected production data.

Output handling

Trace model output into every downstream consumer. Determine whether untrusted output is safely treated as data or is incorrectly rendered, interpreted, stored, logged, or passed to another component. Review output filtering, encoding, and error handling wherever a model response can influence application behavior.

Excessive agency and unauthorized tool calls

Where the AI API can call plugins, functions, or external tools, verify that authorization is enforced independently of the model’s decision. Confirm that the caller’s role, tenant, and approved action scope are checked before a tool is invoked. A model response should not be the sole authorization decision for a consequential action.

RAG and document permissions

For RAG or document-retrieval systems, test tenant and document authorization with controlled accounts and synthetic records. A low-privileged user should not receive restricted documents merely because a model can retrieve them. Validate access controls at the retrieval and storage layers, not only in the prompt.

6. Choose tools and conduct controlled validation

API interception and manipulation

Burp Suite or OWASP ZAP can support interception, request modification, fuzzing, and parameter testing. Use them to compare requests across roles, inspect responses, replay approved workflows, and identify differences between documented and observed behavior.

AI-focused evaluation tooling

Use Garak, PyRIT, or comparable authorized evaluation tooling for AI-specific model and prompt-security testing where appropriate. Structure test cases around the components actually present: prompt handling, leakage boundaries, output controls, retrieval permissions, and tool integrations.

How AI-assisted testing fits

AI-assisted testing can help generate, organize, repeat, and compare test cases across endpoints, roles, prompts, and stateful workflows. DAST tools focus on automated application and API checks, while functional tests verify expected behavior and regressions. Human-led penetration testing remains necessary for business logic, authorization boundaries, realistic impact, unusual workflows, and novel attack chains.

Automation supplements rather than replaces human review. No AI-focused scanner provides complete coverage of conventional API vulnerabilities, deployment-specific integrations, or the consequences of a model response in its surrounding application.

GraphQL, when applicable

If the AI service exposes GraphQL, review whether introspection is appropriately controlled, test field-level authorization across roles, and check whether nested-query depth or complexity limits are enforced. Apply the same authorization, data-exposure, rate-limit, and resource-consumption principles used for other API protocols.

7. Report, remediate, and retest

Capture evidence and impact

For each finding, record:

• A concise title, affected endpoint or component, and test date.
• The authorized account, role, tenant, environment, and relevant model or provider.
• Reproducible requests and responses, with sensitive values redacted.
• The failed security boundary and realistic impact.
• Severity, prerequisites, affected integrations, and residual risk.
• Whether the issue involves the gateway, model access, prompt construction, output handling, RAG permissions, a plugin, or an external tool.

Remediate and run regression tests

Match recommendations to the failing boundary. Examples include enforcing server-side authorization, restricting fields, improving token handling, applying quotas, reducing sensitive logging, validating output before downstream use, isolating tenants, and requiring independent authorization for tool actions. Make provider-, model-, RAG-, and integration-specific ownership clear.

Retest the original request sequence after remediation, then run adjacent regression cases across affected roles, endpoints, model routes, and integrations. Confirm that the issue is fixed without excessive blocking, data loss, unexpected cost, or a new authorization gap.

Final completion checklist

• Written authorization, defined scope, approved environments, test accounts, data protections, monitoring, and stop conditions are in place.
• The architecture, data flows, model boundaries, retrieval stores, logs, tools, integrations, and tenant boundaries are mapped.
• The API inventory includes documented and observed endpoints, methods, parameters, roles, authentication, and model routes.
• Conventional API controls were tested against the relevant OWASP API Security Top 10 categories.
• AI-specific behavior, output handling, leakage, prompt boundaries, tool use, and RAG permissions were assessed where applicable.
• Testing remained controlled and non-destructive, with real data protected from extraction.
• Findings include reproducible evidence, impact, severity, remediation, residual risk, and retest results.

The reliable approach is to test the AI API as both an API and an AI system. Automation can extend coverage, but authorized scope, architecture-aware human review, safe validation, remediation, and retesting remain essential.

Watch this in video

How to Do PT for AI: Penetration Testing for AI