AI security monitoring checklist
- Monitor four core areas: prompts and outputs, agent actions and tool use, model safety and drift, and infrastructure/service telemetry; also track third-party AI dependencies and supply-chain risk.
- Establish normal baselines for prompts, responses, tool use, model behavior, and resource consumption.
- Collect and correlate runtime logs, security events, model-quality metrics, and infrastructure telemetry.
- Alert on anomalies and high-risk actions, then apply runtime filtering, approval, pausing, or blocking controls where appropriate.
- Review vendor and dependency inventories continuously rather than monitoring only internally hosted models.
Monitor four core areas: prompts and outputs, agent actions and tool use, model safety and drift, and infrastructure and service telemetry. Also track third-party AI dependencies and supply-chain risk.
AI security monitoring should detect unsafe behavior, unauthorized activity, sensitive-data exposure, model degradation, infrastructure abuse, and risks introduced by external AI services—not just uptime or performance problems.
AI security monitoring checklist
| Monitoring area | Signals to collect | Why it matters |
|---|---|---|
| Prompts and model responses | Prompt injection, jailbreaks, suspicious intent, probing, PII or credential leakage, harmful output, and other unsafe behavior. | These signals can reveal manipulation, sensitive-data exposure, or unsafe generation. |
| Agent actions and tool use | Tool-call sequences, permission use, unusual workflows, bulk exports, unknown API calls, and unauthorized or high-risk actions. | Agent risk may appear in what the system does after receiving an instruction, not only in its final response. |
| Model safety, quality, and drift | Model versions, refusal behavior, groundedness, accuracy, behavioral or statistical drift, and applicable precision, recall, or F1-score trends. | Changes over time can indicate degradation, data problems, retraining effects, or changed safety behavior. |
| Infrastructure and API telemetry | GPU, CPU, memory, storage, API usage, request rates, throughput, latency, resource consumption, and abuse or denial-of-service indicators. | These signals show operational strain, unusual usage, scalability problems, and possible service abuse. |
| Third-party AI dependencies and supply chain | Vendors, hosted models, APIs, direct dependencies, hidden dependencies, and fourth-party relationships. | External services can introduce data-handling, availability, access, and security exposure beyond internally controlled systems. |
1. Monitor prompts and model responses
Monitor prompt and response security for signals of manipulation, data exposure, or unsafe generation:. For practical guidance on reducing exposure in prompts and outputs, see preventing LLM data leakage.
- Prompt injection and jailbreak attempts.
- Repeated probing, suspicious intent, or unusual input patterns.
- PII, credentials, secrets, or other sensitive information in inputs or outputs.
- Toxic, harmful, or otherwise unsafe model output.
- Unexpected changes in the topics, formats, or sensitivity of requests and responses.
Prompt and response logging should fit the deployment’s privacy and data-handling requirements. Collect enough detail for detection and investigation without treating unrestricted content capture as a requirement.
2. Monitor agent actions and tool use
For an AI agent, monitor the activity that follows its instructions—not only the final answer. Important signals include:
- The sequence and frequency of tool calls.
- The permissions used and whether they match the task.
- Unusual workflows, repeated failures, or abrupt changes in normal action sequences.
- Bulk exports, unknown API calls, external transmissions, or access to unexpected systems.
- Unauthorized, high-risk, or out-of-scope actions.
Correlate access logs, failed authentications, authorization activity, the original request, tool calls, and permissions. This helps distinguish expected automation from misuse. For a focused look at unsafe or unauthorized tool activity, read about AI agent tool misuse.
3. Monitor model safety, quality, and drift
Track the deployed model version alongside changes in behavior and evaluation results. Depending on the use case, monitor:
- Safety refusal rates and other changes in refusal behavior.
- Groundedness and answer quality against the information the system is expected to use.
- Accuracy over time.
- Precision, recall, and F1-score where the task and labeled data make those metrics relevant.
- Data quality, retraining cycles, and behavioral or statistical drift.
Accuracy, precision, recall, and F1-score are not mandatory for every AI system. Use metrics that fit the deployment. A classification workflow may use precision and recall, while a generative system may need evaluations for groundedness and response safety.
4. Monitor infrastructure and API telemetry
Monitor the service layer and resources supporting inference. Relevant signals include:
- GPU, CPU, memory, and storage consumption.
- API request rates, usage patterns, throughput, and response times.
- Inference latency and service availability.
- Cloud or API usage and cost patterns where relevant.
- Unexpected resource consumption, traffic spikes, failed requests, and other abuse or denial-of-service indicators.
Infrastructure telemetry supports AI security monitoring; it does not replace monitoring prompts, outputs, agent behavior, or access activity. Resource metrics can reveal anomalies, but they do not by themselves establish the cause of an incident.
5. Monitor third-party AI dependencies and supply-chain risk
Maintain a continuously updated inventory of every external component involved in the AI service, including:.
- Direct AI vendors and hosted-model providers.
- External models, APIs, libraries, and data services.
- Indirect or hidden dependencies.
- Fourth-party relationships connected to a direct vendor.
- Data-handling, privacy, access, availability, and security exposure in vendor environments.
Review vendor and dependency exposure continuously rather than limiting oversight to internally hosted models. Record whether dependencies are confirmed or suspected and retain evidence supporting that assessment. For a structured review of external provider exposure, use this AI vendor security assessment checklist.
Make monitoring actionable
Establish deployment-specific baselines
Define normal patterns for prompts and responses, agent tool use, permission activity, model behavior, API activity, and resource consumption. Baselines make anomaly detection more useful than isolated threshold checks.
Correlate logs and telemetry
Collect and correlate metrics, events, logs, and traces across the model, application, API, identity layer, and infrastructure. Combine runtime logs and security events with model-quality metrics and service telemetry so an unusual response can be examined alongside the request, permissions, tool calls, and resource activity. For help choosing and reporting security measurements, see this guide to AI security metrics and KPIs.
Alert, investigate, and assign ownership
Alert on meaningful anomalies and high-risk actions, then review the surrounding context. Track false positives and nuisance alerts so teams can improve detection. Assign owners for each signal, document the reason for monitoring it, and retain the evidence needed for investigation and review.
Apply safe runtime controls
When monitoring identifies a credible risk, apply controls appropriate to the deployment. Options may include input or output filtering, additional approval, pausing an agent, blocking an action, restricting permissions, rolling back a model or configuration, or engaging the relevant vendor. These are response options—not substitutes for the monitoring checklist. For detailed safeguards when generated content is passed to users or downstream systems, see this checklist for securing LLM outputs.
A practical implementation sequence
- Inventory AI assets and dependencies. Record models, applications, APIs, agents, infrastructure, vendors, and fourth-party relationships.
- Centralize relevant telemetry. Bring together metrics, events, logs, traces, security events, model evaluations, and access activity.
- Establish normal baselines. Define expected prompts, responses, tool use, model behavior, API activity, and resource consumption.
- Choose applicable quality metrics. Use accuracy, precision, recall, F1-score, groundedness, refusal behavior, or other measures only where they fit the system.
- Configure anomaly and high-risk-action alerts. Tune alerts using investigation results and false-positive patterns.
- Connect findings to safe response. Link alerts to investigation, approval, filtering, pausing, blocking, rollback, or vendor-engagement workflows.
- Review exposure continuously. Reassess model changes, vendor dependencies, permissions, data handling, and fourth-party relationships.
There is no single mandatory metric set for every AI deployment. The strongest AI security monitoring program matches its signals and runtime controls to the system’s model, data, agent capabilities, infrastructure, and external dependencies.
