How to Use an AI Agent Red Teaming Checklist to Find Security Gaps

An AI agent red-teaming checklist should systematically test the full agent lifecycle and attack surface: architecture, prompts, data, tools, permissions, downstream actions, monitoring, remediation, and regression testing.

What an AI agent red teaming checklist covers

An AI agent red-teaming checklist is a reusable worksheet for adversarially evaluating an agent as a complete system—not just testing whether a language model resists jailbreak prompts. It should cover the model, instructions, user inputs, retrieved content, memory, identities, tools, APIs, MCP servers or plugins, permissions, and downstream actions. For the MCP-specific controls in this inventory, see the MCP security checklist.

Prompt testing alone is not sufficient. An agent may produce a safe-looking response while still leaking information through memory, calling an unauthorized tool, exceeding its permissions, or sending unsafe output to a database, shell, API, or renderer.

Organize the checklist into these lifecycle phases:

• Prepare: establish authorization, scope, safety rules, and test controls.
• Map: inventory architecture, assets, trust boundaries, identities, tools, and data flows.
• Model: define threats, scenarios, priorities, and measurable outcomes.
• Test: combine automated adversarial coverage with manual workflow testing.
• Record: preserve reproducible evidence and assess impact and severity.
• Remediate: apply controls, assign owners, and validate fixes.
• Regress: rerun relevant suites after changes to the model, prompts, data, tools, code, or configuration.

Not every item applies to every deployment. Tailor the worksheet to the agent’s architecture, connected capabilities, data, user roles, permissions, and deployment environment.

Preparation: authorization, scope, and safe testing rules

Before conducting an AI red team exercise, define what testers may do and how the environment will be protected. Use explicit permission, an isolated environment where possible, synthetic or approved data, constrained credentials, logging, and rate limits. Do not test live systems or attempt to obtain real secrets without authorization.

Authorization and rules of engagement

• ☐ Record written authorization, the responsible owner, and the security contact.
• ☐ List the systems, agent versions, integrations, accounts, and environments in scope.
• ☐ Define the test window, rate limits, escalation contacts, stop conditions, and emergency shutdown process.
• ☐ Document prohibited actions, including real-secret exfiltration, destructive changes, unapproved production activity, and access to unauthorized third-party systems.
• ☐ Specify approved data sources and data-handling, retention, and deletion requirements.
• ☐ Define how critical findings will be reported immediately rather than held for a final report.

Isolated environment and test controls

• ☐ Use a non-production, isolated environment where possible.
• ☐ Use synthetic or specifically approved data instead of live confidential information.
• ☐ Use non-production credentials with only the permissions required for the test.
• ☐ Enable logs for requests, responses, retrieval, memory reads and writes, tool calls, approvals, authentication, and downstream actions.
• ☐ Apply rate limits, transaction limits, and other controls that reduce the potential blast radius.
• ☐ Confirm rollback or shutdown procedures for agents that can change files, records, accounts, or external systems.

Architecture inventory and threat model

A complete inventory keeps AI agent red teaming from collapsing into generic LLM red teaming. Record each component, owner, version, trust boundary, data flow, and privilege level.

For each connection, identify whether data or instructions cross a trust boundary. Note where a user, document, tool, plugin, or external system can influence the agent’s context or actions. Before testing trust boundaries, use an AI agent threat modeling guide to structure the architecture and attack-path analysis.

Threats, assets, and priorities

• ☐ Identify threat actors, including untrusted users, compromised accounts, malicious content authors, rogue integrations, and users attempting to exceed their role.
• ☐ Identify assets such as credentials, personal data, proprietary information, source code, prompts, memory, tenant data, and external systems.
• ☐ Map trust boundaries between users, the model, retrieval sources, memory, tools, APIs, MCP servers, plugins, and downstream systems.
• ☐ Define likely attack paths, including indirect instructions in retrieved content and multi-step tool misuse.
• ☐ Prioritize tests by privilege, data sensitivity, action reversibility, exposure, and blast radius.
• ☐ Match testers and methods to the objective: automation for breadth and manual analysis for contextual workflows.

Success metrics

Use metrics as configurable fields rather than universal pass/fail thresholds. Relevant measures include attack success rate, planned-scenario coverage, unauthorized access, privilege escalation, data exposure, detection time, exploitability, impact, severity, reproducibility, false positives, and false negatives.

Baseline behavior and defensive controls

Capture expected behavior before adversarial execution. Define which requests should be refused, which actions require approval, what each role may access, and what evidence should appear in logs.

Guardrails and model behavior

• ☐ Test input validation and handling of malformed, oversized, or unsupported inputs where relevant.
• ☐ Test harmful-output safeguards and policy enforcement.
• ☐ Check whether controls can be bypassed through changes in context, role, format, language, modality, or conversation history.
• ☐ Assess reliability degradation, context drift, long-context failures, and inconsistent responses where relevant.
• ☐ Evaluate fairness-related behavior where the agent makes decisions or produces content affecting people or groups.
• ☐ Confirm that output filtering does not replace controls on tools, permissions, and downstream execution.

Authentication, authorization, approvals, and logging

• ☐ Verify authentication for users, services, tools, APIs, and administrative functions.
• ☐ Confirm that authorization is enforced independently of model instructions.
• ☐ Check role boundaries, least privilege, credential scope, and tenant restrictions.
• ☐ Test approval gates for sensitive, high-impact, or irreversible actions.
• ☐ Confirm that logs identify the user, agent, model or configuration version, retrieved sources, tool calls, approvals, outputs, and resulting actions.
• ☐ Test controls individually and in combination, because a safe component may fail when chained with another component.

Prompt injection and instruction-hierarchy test matrix

Test whether the agent can distinguish authorized instructions from untrusted user input, retrieved content, files, tool responses, and conversation history. Use approved test data and do not extract real secrets.

Agent autonomy, permissions, tools, and downstream actions

The defining difference between model testing and full-agent testing is whether the system can take action. Test the boundary between what the agent may suggest and what it may actually do.

Autonomy and authorization boundaries

• ☐ Test goal hijacking and whether untrusted instructions can redirect the agent’s objective.
• ☐ Test excessive agency: actions taken without user confirmation, adequate context, or appropriate approval.
• ☐ Test unauthorized tool calls, privilege escalation, and approval bypasses within the authorized environment.
• ☐ Check whether the agent can access another user’s files, records, memory, credentials, or tenant data.
• ☐ Test file operations, database reads and writes, shell commands, external messages, destructive actions, and transactions according to scope.
• ☐ Verify transaction limits, action sequencing, rollback, and blast-radius controls.

MCP, tool, plugin, and API checks

• ☐ Verify that every tool and API checks caller authorization and enforces least privilege.
• ☐ Test tool manipulation, unexpected tool selection, and multi-step misuse.
• ☐ Check schemas, parameter validation, namespaces, return values, and error handling.
• ☐ Assess poisoning, spoofing, version integrity, and unexpected changes to tool definitions or plugin behavior.
• ☐ Check whether credentials or sensitive tool responses are exposed to the model, users, logs, or other tenants.
• ☐ Test credential scope, expiration, rotation, and separation between users and service identities.
• ☐ Confirm complete tool-call logging, including the function, parameters, authorization decision, result, user, agent state, and timestamp.
• ☐ Test whether a sequence of individually permitted calls produces an unauthorized result.

Unsafe downstream execution

Model output becomes a security issue when another system executes or renders it. Test the complete path from generated content to the receiving component. For practical controls around this execution path, see how to secure LLM outputs before they reach a receiving component.

• ☐ Test output passed to SQL or other database queries for injection and unauthorized data access.
• ☐ Test shell or command execution for unsafe commands, unexpected expansion, and excessive operating-system permissions.
• ☐ Test API parameters, URLs, headers, and request bodies for unauthorized actions or injection.
• ☐ Test HTML and JavaScript rendering for unsafe execution or unintended browser behavior.
• ☐ Test generated or modified files for unsafe content, path handling, permissions, and unintended execution.
• ☐ Verify parameterization, escaping, sandboxing, allowlists, confirmation steps, and least privilege at the receiving system.
• ☐ Record whether the downstream system executed, rendered, rejected, sanitized, or logged the output.

RAG, memory, context, and isolation checks

Retrieval and memory can introduce instructions and data that were not present in the user’s request. Test both integrity—whether the agent grounds answers correctly—and separation—whether one user or session can influence another.

RAG poisoning and grounding

• ☐ Test malicious or altered documents and indirect instructions in retrieved content.
• ☐ Check embedding, ranking, filtering, and retrieval manipulation.
• ☐ Verify source provenance, citation integrity, and factual grounding.
• ☐ Confirm that retrieved instructions are treated as data unless the architecture explicitly authorizes them.
• ☐ Record whether poisoned or irrelevant content changes the answer, tool choice, or downstream action.

Memory, session, context, and tenant isolation

• ☐ Test disclosure of sensitive information across sessions, users, roles, and tenants.
• ☐ Test persistent memory and scratchpad access for leakage, unauthorized writes, and poisoning.
• ☐ Check context-window boundaries and whether stale or unrelated instructions influence a new task.
• ☐ Verify vector-store, retrieval-filter, file, and cache isolation.
• ☐ Confirm that logs, traces, support tools, and error messages do not bypass the same isolation controls.

Capability-specific checks

Include these checks only when the deployment supports the relevant capability.

• ☐ Browser or computer use: test navigation, downloads, forms, external messages, account boundaries, and confirmation before consequential actions.
• ☐ Coding workflows: test repository access, generated code, dependency changes, secret handling, execution permissions, and review or approval gates.
• ☐ Multimodal inputs: test whether visual or document content changes instruction handling, retrieval, output safety, or tool use.
• ☐ Multi-agent workflows: test delegation, inter-agent messages, identity propagation, permission inheritance, untrusted-agent behavior, and cascading failures.

Testing methods: automated, manual, and access models

Use a hybrid method. Automated adversarial testing provides breadth and repeatability, while manual testing is needed for contextual, multi-turn, chained, and architecture-specific workflows.

The access model changes test design and evidence, but it does not remove the need to test the complete workflow. Black-box testing may reveal externally exploitable behavior; gray-box and white-box access can help explain causes, boundaries, and remediation. For a deeper comparison of the access models mentioned here, see black-box vs. white-box AI penetration testing.

Neutral capability criteria for tools and frameworks

Select supporting tools by capability rather than brand. Consider whether a solution provides:

• Automated probing and adversarial test-case generation
• Multi-turn, chained, and agent-workflow testing
• RAG, memory, and retrieval evaluation
• Tool, API, MCP, plugin, and permission-path coverage
• Evidence capture, trace export, and reproducible test records
• CI/CD integration and regression-test execution
• Configurable metrics, severity fields, and reporting
• Clear limitations around coverage, false positives, context, and architecture-specific analysis

Optional mappings to OWASP guidance, MITRE ATLAS, and the NIST AI RMF can organize coverage. They do not replace threat modeling or architecture-specific testing, and no tool or mapping proves that an agent is secure. For a focused comparison of MCP scanning options, see MCP security scanners.

Reusable test-case worksheet

Use one record for every test scenario, including tests marked passed, blocked, or not applicable. The same fields can be maintained in a GitHub-compatible Markdown file, issue template, spreadsheet, or PDF form.

Findings, remediation, and retesting

Each finding should allow another tester to reproduce the behavior without guessing what happened. Severity and scoring should remain configurable to the organization and deployment rather than follow one universal threshold.

Depending on the finding, remediation may include filters and validation, approval gates, least-privilege identities, scoped credentials, tool allowlists, schema validation, sandboxing, transaction limits, isolation changes, logging, alerts, configuration changes, or new regression cases.

Retest checklist

• ☐ Reproduce the original finding under the documented conditions.
• ☐ Test the equivalent case after the fix.
• ☐ Test reasonable variants involving context, role, language, modality, or workflow sequence where relevant.
• ☐ Confirm that the fix does not create an unintended reliability, access, or usability failure.
• ☐ Verify logs, alerts, approvals, and containment—not only the final model response.
• ☐ Update the finding, evidence, owner, closure status, and regression suite.

Continuous regression testing and change triggers

Make the worksheet part of the deployment lifecycle. Integrate suitable adversarial suites into CI/CD and retain manual workflow tests for changes that affect context, permissions, or agent behavior.

Rerun relevant tests after changes to:

• Model, model provider, routing, or inference configuration
• System prompts, developer instructions, policies, or guardrails
• RAG sources, retrieval logic, embeddings, ranking, or datasets
• Memory, session, context, or tenant-isolation logic
• Tools, APIs, MCP servers, plugins, schemas, credentials, or permissions
• Agent code, downstream systems, deployment configuration, or approval workflows

Also rerun relevant coverage after a material incident, newly identified threat, significant integration change, or remediation that affects an adjacent workflow. Continuous testing is regression coverage—not proof that the agent is secure or that future behavior is predictable.

How this differs from generic LLM or traditional penetration testing

Generic LLM red teaming often emphasizes prompts, outputs, jailbreaks, and harmful content. Traditional penetration testing commonly centers on infrastructure, applications, identities, and network controls. AI agent red teaming must include relevant concerns from both while also testing probabilistic behavior, retrieved context, memory, delegated authority, tool selection, multi-step planning, permissions, and automated downstream actions. For a detailed comparison of these complementary approaches, see AI red teaming vs. penetration testing.

The central question is not only whether the model gives a bad answer. It is whether untrusted input, content, or context can cause the complete agent to disclose data, cross a boundary, invoke a tool, exceed authority, or trigger an unsafe action.

Template structure for GitHub or PDF use

For a reusable AI agent red teaming checklist template, keep the following sections in the same order across a GitHub Markdown file, issue template, spreadsheet, or PDF:

1. Rules of engagement and authorization
2. Architecture and attack-surface inventory
3. Threat model and success metrics
4. Baseline controls
5. Prompt and instruction test matrix
6. Tool, API, MCP, plugin, permission, and downstream-action tests
7. RAG, memory, context, and isolation tests
8. Applicable browser, coding, multimodal, and multi-agent tests
9. Test-case worksheet
10. Findings, remediation, retesting, and closure
11. Continuous-testing change triggers

Use the same test IDs, evidence fields, owners, severity fields, and retest status across formats so results remain comparable over time.

A complete worksheet is a testing aid, not a security certification. Tailor it to the agent’s architecture and connected capabilities, preserve reproducible evidence, assign remediation owners, validate fixes, and rerun regression tests after relevant changes.

Watch this in video

How to Start an AI Red Teaming Career in 2026 (with free …, Video, watch on YouTube