How to Pentest an AI Chatbot: A Practical Security Checklist

To pentest an AI chatbot, obtain authorization, define scope, map its model, application, RAG, and tool layers, threat-model the system, test AI-specific and conventional risks, then document evidence, remediation, and retest results.

What an AI chatbot pentest covers

An AI chatbot penetration test assesses more than conversational responses. Depending on the architecture, it can include the LLM, system and developer prompts, web interface, backend, APIs, authentication, authorization, data stores, retrieval sources, vector stores, memory, plugins, tools, integrations, downstream systems, and supporting infrastructure.

A chatbot may produce a harmless-looking answer while its application exposes private data, accepts an unauthorized tool call, mishandles generated output, or fails to enforce tenant boundaries. A useful assessment therefore combines LLM security testing with conventional web, API, identity, data, integration, and infrastructure testing.

Scope by chatbot architecture

Do not assume every test applies to every chatbot. The approved architecture and data flows determine which areas are relevant.

1. Start with authorization, scope, and safety controls

Use only systems you own or are explicitly authorized to assess. Before sending test cases, document the approved targets, environments, interfaces, accounts, and data-handling rules, testing period, and escalation contacts.

Authorization and assessment type

• Obtain written authorization identifying the chatbot, related APIs, domains, environments, integrations, and supporting services in scope.
• Agree whether the assessment is black-box, gray-box, or white-box, and define what documentation, source access, logs, or credentials are available.
• Use staging environments where possible, with synthetic or specially prepared test data instead of real customer records.
• Define prohibited actions, including unauthorized access, destructive changes, data theft, credential use, uncontrolled denial of service, and tests that could create excessive charges.

Rules of engagement

Set request, token, concurrency, and cost limits. Define testing windows, logging expectations, evidence-retention rules, stop conditions, and the process for escalating suspected exposure or service degradation.

Use the minimum evidence needed to prove a finding. Redact sensitive output, avoid copying real protected data, and stop when a test confirms exposure or begins to affect service stability. Do not test third-party systems or connected accounts unless they are expressly included in the authorization.

2. Map the architecture and trust boundaries

Map data flows before testing. Record where user input enters, how prompts are assembled, which identity is used for retrieval or tool calls, and which systems can receive generated content or execute actions.

Model, prompts, frontend, backend, and APIs

• Identify the model, system and developer instructions, prompt templates, conversation memory, model providers, and safety controls.
• Enumerate web, mobile, and embedded interfaces; backend services; public and internal APIs; webhooks; queues; and administrative functions.
• Document how authentication, sessions, OAuth or JWT tokens, roles, and permissions move between the interface, chatbot service, and downstream systems.

Data, retrieval, tools, and downstream systems

Record data sources, document stores, retrieval services, vector stores, memory stores, plugins, APIs, MCP connections, external tools, databases, and business systems. Mark each trust boundary, such as user to frontend, frontend to API, chatbot to retrieval service, chatbot to tool, and tool to downstream system.

Also identify where model output is rendered, stored, queried, or passed to another service. These handoffs often determine whether a model-behavior issue becomes an application-security issue.

Supply-chain review

Where they are in scope, review the provenance and maintenance of third-party models, datasets, RAG components, repositories, dependencies, and intermediary models. Check that untrusted or outdated components cannot introduce unauthorized instructions, data exposure, or altered behavior. Keep this review proportionate to the actual architecture and assessment scope.

3. Threat-model the system and plan the workflow

Threat-model the chatbot as four connected layers:

• Application layer: web, API, session, input-validation, output-handling, and business-logic risks.
• Model layer: instruction-boundary failures, jailbreaks, unsafe or manipulated outputs, hallucination, grounding, and overreliance.
• Infrastructure layer: identity, cloud services, secrets, network paths, logging, rate limits, and resource controls.
• Data layer: prompts, documents, retrieval indexes, memory, private context, training-related content, and tenant separation.

OWASP’s LLM guidance, MITRE ATLAS, and the NIST AI Risk Management Framework can organize the assessment and reporting. They are useful references, not one exclusive testing procedure. Supplement them with established web, API, identity, cloud, and infrastructure testing practices.

Repeatable pentest workflow

1. Reconnaissance: inventory approved interfaces, endpoints, roles, data sources, integrations, and observable behavior.
2. Architecture review: confirm trust boundaries, identities, prompt assembly, retrieval paths, tool permissions, and downstream data flows.
3. Controlled execution: use prepared benign cases against dedicated accounts and synthetic data.
4. Variant testing: replay controlled input variations and fuzzing within the agreed rate and cost limits, then add manual adaptive testing.
5. Analysis: compare observed behavior with the expected security policy. Repeat important cases when responses are nondeterministic.
6. Reporting and remediation: separate model findings from application, identity, integration, and infrastructure findings.
7. Retesting: replay the minimal proof of concept after a fix and record whether the issue is resolved, reduced, or still reproducible.

Automation can support mutation, replay, regression checks, and pass/fail evaluation. It should complement—not replace—manual review of conversation state, authorization, data flows, and tool behavior.

4. Test prompt injection, jailbreaks, and instruction boundaries

Prompt injection is a central AI chatbot pentesting concern. Use controlled cases to determine whether untrusted instructions can override trusted instructions, bypass restrictions, change the chatbot’s role, or cause unauthorized disclosure or action. Do not use real secrets or destructive tool requests. For a broader adversarial-testing checklist covering prompt manipulation and agent behavior, see the AI agent red-teaming checklist.

Direct prompt injection

Test instruction overrides, role-play, encoded or obfuscated inputs, multi-turn conversations, and attempts to make the model ignore its stated boundaries. Compare behavior across fresh sessions, user roles, and conversation histories.

The secure expectation is that user content cannot replace system or developer instructions, expand the user’s permissions, or trigger an action outside the approved workflow. Record the model’s response and any resulting API call, retrieval event, or tool request.

Indirect prompt injection

Place benign test instructions in approved documents, comments, web pages, or retrieved content and observe whether the chatbot treats them as trusted commands. The expected behavior is to keep external content separate from trusted instructions and require authorization before any consequential action.

Test indirect paths wherever content is imported automatically. A document, web page, support ticket, or database field can influence a response even when the user did not place the instruction directly in the chat.

Jailbreak and safety-boundary testing

Assess whether role-play, obfuscation, repeated requests, or multi-turn context causes the chatbot to bypass its safety boundary or produce behavior the application is intended to prevent. Keep cases bounded, non-destructive, and tied to the documented security policy.

5. Test leakage, authentication, authorization, and isolation

Check whether the chatbot discloses system or developer instructions, credentials, API keys, PII, proprietary information, transaction data, conversation histories, training-related content, or another user’s private context. Use synthetic markers so confirmed exposure can be identified and reported safely. For a focused guide to preventing this class of exposure, see LLM data leakage prevention.

• Test direct requests and multi-turn attempts for protected instructions or data.
• Start a new session and use a separate authorized test account to check context retention.
• Compare permitted and unpermitted users to identify cross-user or cross-role disclosure.
• Where tenants exist, verify that prompts, memory, retrieved documents, tool results, and API responses remain tenant-specific.

Record whether the response reveals the protected value, a fragment, metadata about it, or only a refusal. A refusal that still exposes useful sensitive context should be investigated separately from complete non-disclosure.

Authentication and authorization checks

Test the chatbot as an application identity boundary, not only as a model. Verify authentication, session expiration, OAuth or JWT handling, RBAC or ABAC, least privilege, permission boundaries, and access to chatbot functions and data. For a deeper treatment of these identity and authorization controls, see IAM for AI applications.

• Compare behavior between permitted and unpermitted test accounts.
• Check cross-session and cross-user isolation after ending or restarting conversations.
• Verify that administrative functions, hidden endpoints, and backend APIs enforce the same authorization as the visible interface.
• Confirm that a model-generated request cannot grant itself or the user additional permissions.

6. Test RAG and retrieval security

For a RAG chatbot, assess both what the system retrieves and who is allowed to retrieve it. For a focused testing method for retrieval-augmented systems, see the RAG red-team workflow.

• Use approved test documents containing clearly marked misleading or untrusted instructions to assess retrieval poisoning and indirect injection.
• Verify source permissions before and after indexing, chunking, embedding, retrieval, and response generation.
• Test whether one user or tenant can retrieve another’s private documents, citations, metadata, or conversation context.
• Check whether the application distinguishes trusted instructions from retrieved content and preserves source separation across multi-turn conversations.

Do not poison production indexes or alter shared data without explicit approval. Use isolated collections, synthetic documents, and reversible changes instead.

7. Test APIs, plugins, tools, and agentic behavior

Connected capabilities can turn a model mistake into an external action. For every AI chatbot tool, plugin, API, or MCP connection, document the permitted caller, accepted parameters, returned data, side effects, and approval requirements. For a dedicated checklist covering this API layer, see AI API penetration testing.

• Check whether a user can invoke a tool outside their role or alter its parameters through conversation.
• Assess excessive agency, unauthorized actions, privilege escalation, excessive data access, and inadequate downstream isolation.
• Use benign, non-destructive operations to assess command or query injection at tool and integration boundaries.
• Verify that tool calls are authorized independently of generated text and that sensitive actions require the intended human or application approval.
• Check whether tool results expose secrets or private data to a user who could not access the underlying system directly.

For an agentic system, follow the entire action chain: model decision, tool selection, parameter construction, authorization, downstream execution, returned result, and any subsequent tool call.

8. Test input validation and unsafe output handling

Inspect both chatbot and API boundaries. Determine whether generated text is rendered in a browser, inserted into a database, used in a query, passed to a URL fetcher, or supplied to another command or service without suitable validation and encoding.

Use controlled cases in staging to assess potential XSS, SQL injection, SSRF, CSRF, command injection, and related downstream risks. The objective is to verify safe handling, not to execute destructive payloads or access unrelated systems. Treat model output as untrusted input wherever it crosses into another component. For practical controls around validation and safe downstream use, see LLM output security.

Also check whether logging, analytics, citations, conversation export, and error handling preserve sensitive values or execute content unexpectedly.

9. Test grounding, hallucination, and model integrity

Assess safety-boundary failures, harmful or manipulated outputs, hallucination, grounding, overreliance, bias exploitation, and fact-verification failures. Compare responses with an approved trusted source and record fabricated facts, citations, URLs, statistics, or professional guidance.

For a controlled fact-verification test, provide a question whose answer can be checked against a trusted source. Record the expected answer, the chatbot’s response, the citation or reasoning it provides, and whether it clearly indicates uncertainty when verification is unavailable. Repeat the test under the same conditions when responses vary.

Also assess model, data, and retrieval poisoning using isolated, reversible test content. Define the expected security boundary before testing, then use automated variants, replay, manual adaptive testing, pass/fail evaluation, and regression testing to determine whether misleading content changes later behavior.

Compact test-case format

10. Test availability and unbounded consumption safely

Within agreed limits, assess request flooding, excessive API usage, token or context exhaustion, computational strain, throttling, service degradation, and potential financial impact. Use low-volume, monitored tests with explicit stop conditions and cost caps.

Verify that rate limits, quotas, queue controls, timeout behavior, and alerts operate as intended. Do not run an uncontrolled denial-of-service test, recursive request chain, or excessive-billing experiment.

11. Record findings and report remediation

Each finding should be reproducible without exposing unnecessary sensitive content. Separate model-behavior findings from application, identity, access-control, integration, and infrastructure findings.

Map findings to an appropriate framework or taxonomy where useful, but explain the actual affected component and impact rather than relying on a category name alone. A minimal reproducible proof of concept is more useful than a large collection of speculative prompts.

12. Remediate, retest, and continue validation

After remediation, replay the smallest test case that demonstrated the issue. Confirm that the fix addresses the underlying boundary rather than only changing one response, and test nearby variants to check for an equivalent path.

An AI chatbot security assessment is not finished when the initial report is delivered. Replay important cases after model, prompt, retrieval, tool, identity, or application changes. Track remediation, review logs, inspect runtime behavior, monitor for drift, and trigger retesting when integrations or permissions change.

The final plan should remain tied to the chatbot’s actual architecture. A basic conversational system does not require every RAG or agentic test, while a system connected to private documents and external tools requires those additional checks. Keep every follow-up assessment authorized, bounded, evidence-based, and non-destructive.