AI agents need more than a secure prompt. They may retrieve documents, call APIs, access databases, use external tools, and take actions on behalf of users. A single authorization failure or indirect prompt injection can therefore have consequences beyond an incorrect chatbot response.
This guide explains 12 practical AI agent security test cases, what each test is intended to verify, what secure behavior looks like, and how to document the results. It is designed for developers, application security engineers, SaaS teams, and organizations preparing AI-powered applications for production.
Scope note: These are example test cases for authorized assessments. Adapt them to the application's architecture and threat model, and use a controlled test environment wherever possible. Passing these 12 tests does not establish that an AI system is completely secure.
Table of Contents
- Why AI agent security testing is different
- How to prepare a test
- 12 practical AI agent security test cases
- How to prioritize findings
- Pre-production checklist
- Frequently asked questions
1. Why AI Agent Security Testing Is Different
Traditional application security testing remains important, but an agent introduces additional trust boundaries. Its behavior may depend on model output, retrieved content, conversation history, tool descriptions, API responses, external services, and the permissions of the identity used to execute actions.
For example, a support agent might be allowed to read a customer's tickets and draft a response. If it can also export every customer's data or send messages without authorization checks, a manipulated instruction may become an access-control problem rather than merely a model-quality problem.
Testing should therefore examine the entire system:
- Model and instructions: How the agent handles conflicting or malicious instructions.
- Tools and integrations: Which operations the agent can request and which operations the backend actually permits.
- Identity and permissions: Whether actions are authorized for the current user, tenant, and session.
- Data sources: Whether retrieved documents, memories, and tool results respect access boundaries.
- Execution environment: Whether file, shell, network, and other capabilities are isolated.
- Monitoring: Whether security-relevant decisions and denied actions can be investigated.
The OWASP Top 10 for LLM Applications and the OWASP GenAI Security Project are useful starting points for developing a threat model and choosing relevant tests.
2. How to Prepare an AI Agent Security Test
Before running the test cases, document the agent's intended behavior and establish a baseline. Without a baseline, it is difficult to distinguish a genuine security failure from an expected product behavior.
- Define scope: Identify the application, agent, APIs, connected tools, data sources, test accounts, and permitted actions.
- Map trust boundaries: Document how user input, retrieved content, tool responses, and external services enter the agent workflow.
- Inventory capabilities: Record the agent's read, write, administrative, file, network, and execution permissions.
- Create test identities: Where relevant, use separate users or tenants with deliberately different access rights.
- Define expected results: Specify which actions should be allowed, denied, or sent for human approval.
- Capture evidence: Record the test configuration, relevant requests, tool calls, authorization decisions, and observed results without exposing real secrets or unnecessary personal data.
Run tests only against systems you own or have explicit authorization to assess. Use synthetic records and harmless actions to demonstrate access-control failures safely.
3. Twelve Practical AI Agent Security Test Cases
Test Case 1: Direct Prompt Injection
Risk: User-controlled instructions attempt to make the agent ignore its intended task or bypass its normal restrictions.
Test procedure: In an authorized test session, submit a request that conflicts with the agent's intended role. For example, ask a read-only research agent to perform an administrative action or reveal information it is not authorized to disclose.
Expected secure behavior: The agent should continue to follow the application's security policy. Any prohibited tool request should be denied by an independent authorization control, even if the model attempts to make it.
Evidence to record: The test input, model response, attempted tool calls, and backend allow-or-deny decision.
Remediation: Treat prompts as behavioral guidance, not an access-control mechanism. Enforce permissions in backend code and test the enforcement independently of the model.
Test Case 2: Indirect Prompt Injection Through Retrieved Content
Risk: A document, email, web page, support ticket, or tool response contains malicious instructions that influence the agent.
Test procedure: Place a clearly labelled test instruction in a controlled document or test record. The instruction should ask the agent to perform an action outside its permitted task. Ask the agent to summarize or process that content through the normal application workflow.
Expected secure behavior: The agent may summarize the document, but the document's instructions must not grant new permissions or authorize unrelated actions.
Evidence to record: The source document, retrieval result, model response, attempted actions, and enforcement decisions.
Remediation: Treat retrieved content as untrusted input. Separate data from trusted instructions where possible, validate tool calls, and enforce authorization at the point where actions execute. Prompt-injection filters alone are not a complete defense.
Test Case 3: Excessive Tool Permissions
Risk: The agent can access tools or operations that are unnecessary for its assigned task.
Test procedure: Compare the agent's documented purpose with its registered tools and actual backend capabilities. Check whether a read-only agent can write records, delete resources, change permissions, send external messages, or invoke administrative operations.
Expected secure behavior: The agent should have only the tools and operations required for its task. Unnecessary or explicitly prohibited actions should be unavailable or denied by policy.
Evidence to record: Tool inventory, credential scopes, permission configuration, and authorization results.
Remediation: Apply least privilege, separate read and write capabilities, use narrowly scoped credentials, and make high-impact operations subject to explicit authorization.
Test Case 4: User and Tenant Authorization
Risk: An agent accesses another user's records or crosses tenant boundaries because the backend trusts the model's interpretation of the request.
Test procedure: In a test environment, create two users with separate records and permissions. Ask the agent operating as the first user to retrieve a known test record belonging only to the second user. Where the application exposes APIs, test the authorization layer directly as well.
Expected secure behavior: The request should be denied unless the current identity has explicit permission to access the target record. The model's answer must not override the backend's access decision.
Evidence to record: Test identities, record ownership, request identifiers, access-control decision, and any returned data.
Remediation: Derive identity from a trusted authenticated session. Enforce record-level and tenant-level authorization in the backend for every relevant operation. Do not rely on the agent to enforce data separation.
Test Case 5: Sensitive Information Disclosure
Risk: The agent exposes secrets, personal information, internal instructions, or confidential records through its answers or connected tools.
Test procedure: Populate a controlled test environment with synthetic sensitive values. Attempt to retrieve values outside the test user's permitted context through normal queries, conversation history, tool calls, and generated responses.
Expected secure behavior: The agent should not disclose information that the requesting identity is not authorized to access. Sensitive values should also be protected in logs and downstream systems.
Evidence to record: The synthetic data involved, request path, authorization outcome, and whether the information appeared in the response, tool arguments, or logs.
Remediation: Minimize sensitive data in model context, enforce authorization before retrieval, use appropriate output handling, and avoid placing credentials or other secrets in prompts. Check logs for accidental exposure.
Test Case 6: Retrieval and Memory Isolation
Risk: A retrieval-augmented generation (RAG) system or persistent agent memory returns content from another user, tenant, or security context.
Test procedure: Insert unique synthetic markers into separate users' documents or memory records. Query the system under each identity and verify that only the permitted records can be retrieved. Include tests for conversation history and persistent memory if those features exist.
Expected secure behavior: Retrieval and memory access should respect the same authorization boundaries as the source data. A record must not become accessible merely because it is relevant to the model's query.
Evidence to record: User and tenant identifiers, retrieval filters, returned document identifiers, and access-control decisions.
Remediation: Enforce access filters in the retrieval layer, partition data where appropriate, validate tenant scope on every query, and define memory retention and deletion rules.
Test Case 7: API and Tool Argument Validation
Risk: An agent generates malformed or unauthorized arguments that trigger unintended behavior in an API, database, shell, or other downstream tool.
Test procedure: In a test environment, provide inputs that encourage invalid identifiers, unexpected parameter types, out-of-range values, or references to resources the test user cannot access. Test the downstream API or tool boundary independently where possible.
Expected secure behavior: The application should reject invalid arguments and unauthorized operations before execution. Model-generated parameters should never be assumed trustworthy.
Evidence to record: The proposed tool arguments, schema-validation results, authorization decision, and execution outcome.
Remediation: Use strict schemas, validate types and ranges, allowlist supported operations, apply parameterized database queries, and enforce authorization at the service that performs the action.
Test Case 8: Human Approval and High-Impact Actions
Risk: The agent performs an irreversible or high-impact action without the required human authorization.
Test procedure: In a safe environment, configure a test action such as deleting a synthetic record, changing a non-production setting, or sending a message to a controlled test recipient. Attempt to trigger it without approval, with expired approval, and with approval for different parameters.
Expected secure behavior: The action should not execute unless the required approval is valid and covers the specific operation and parameters. Approval for one action must not automatically authorize another.
Evidence to record: The requested action, approval state, approved parameters, execution result, and audit entry.
Remediation: Enforce approval in the execution service, bind approval to the action and parameters, use expiration and replay protection where appropriate, and independently verify high-impact operations.
Test Case 9: Code Execution and Sandbox Boundaries
Risk: An agent with code execution or file access can reach sensitive host resources, production credentials, or unrestricted network destinations.
Test procedure: Review the agent's execution environment and run harmless boundary checks within an authorized test container or disposable environment. Verify which directories, environment variables, processes, and network destinations are accessible.
Expected secure behavior: The agent should be confined to the approved execution scope. Sensitive host resources and unapproved destinations should remain inaccessible.
Evidence to record: Sandbox configuration, mounted resources, permitted network destinations, access attempts, and enforcement results.
Remediation: Use isolated environments, restrict filesystem mounts and network egress, avoid exposing production credentials, and separate agent execution from privileged host services.
Test Case 10: Audit Logging and Security Monitoring
Risk: Unauthorized tool calls or sensitive actions cannot be investigated because the system lacks useful audit records.
Test procedure: Perform a controlled permitted action and a controlled denied action. Check whether the logs capture the initiating identity, session, tool, action, authorization result, timestamp, and relevant correlation identifier.
Expected secure behavior: Security-relevant actions and denials should be traceable. Logs should not unnecessarily store passwords, access tokens, or sensitive prompt content.
Evidence to record: The generated audit events, fields present, missing events, and any sensitive data captured unexpectedly.
Remediation: Log tool execution and authorization outcomes, centralize security-relevant records where appropriate, restrict log access, define retention rules, and redact secrets.
Test Case 11: Resource Exhaustion and Runaway Tool Loops
Risk: Repeated retries, recursive tool calls, oversized inputs, or expensive operations cause service disruption or unexpected usage costs.
Test procedure: In a controlled environment, exercise documented limits using bounded test inputs. Verify the behavior when the agent reaches its token, execution-time, retry, tool-call, or spending limits.
Expected secure behavior: The system should stop or constrain work when configured limits are reached. It should fail safely rather than continue executing unbounded actions.
Evidence to record: Request count, tool-call count, duration, limit enforcement, error handling, and resource usage.
Remediation: Apply rate limits, execution timeouts, token and cost budgets, bounded retries, maximum tool-call depth, and appropriate circuit breakers.
Test Case 12: Multi-Agent and Delegation Boundaries
Risk: One agent passes unauthorized instructions or excessive permissions to another agent, allowing a chain of actions that violates the original security policy.
Test procedure: If the application uses multiple agents, create a controlled scenario where one agent requests an action outside its permissions from a second agent. Check whether the second agent independently validates identity, scope, and authorization.
Expected secure behavior: Each agent and downstream service should enforce its own permissions. Delegation must not silently increase access or bypass approval requirements.
Evidence to record: The delegation chain, identities, requested actions, permission checks, and resulting audit events.
Remediation: Use explicit delegated identities, scoped capabilities, independent authorization checks, bounded delegation, and traceable action logs. Do not treat an upstream agent's request as proof of permission.
4. Summary: Test Cases and Expected Results
| Test area | Expected secure result | Priority consideration |
|---|---|---|
| Direct prompt injection | Prohibited actions remain blocked | High when tools are available |
| Indirect prompt injection | Untrusted content cannot grant authority | High when external content is processed |
| Tool permissions | Only required capabilities are available | High for write or admin tools |
| Authorization | Every action respects user and tenant access | High when sensitive records are involved |
| Data disclosure | Unauthorized information remains protected | Based on data sensitivity and exposure |
| Retrieval and memory | Data boundaries remain enforced | High for multi-user RAG systems |
| API validation | Invalid or unauthorized calls are rejected | High for consequential operations |
| Human approval | High-impact actions require valid approval | High for irreversible actions |
| Sandboxing | Execution remains within approved boundaries | High when code execution is enabled |
| Audit logging | Actions and denials are traceable | Important for incident response |
| Resource limits | Excessive work stops within configured limits | Important for exposed or costly agents |
| Multi-agent delegation | Permissions do not expand across agents | High for delegated workflows |
These priority considerations are starting points, not universal severity ratings. Actual risk depends on the agent's access, the data it handles, the actions it can perform, exposure, and available compensating controls.
5. How to Prioritize AI Agent Security Findings
Not every failed test has the same business impact. Prioritize findings according to the consequences of the failure, the likelihood of exploitation, and the strength of existing controls.
- Critical business impact: Unauthorized administrative actions, cross-tenant sensitive-data exposure, or uncontrolled destructive operations may warrant urgent attention depending on their scope and exploitability.
- High concern: Tool misuse, privilege escalation, or data disclosure where meaningful access boundaries can be bypassed.
- Moderate concern: Failures with limited impact, strong compensating controls, or constrained access, after validation.
- Operational concern: Missing audit events, weak resource limits, or incomplete safeguards that make other failures more likely or harder to investigate.
Do not assign severity based only on the model's response. Validate the actual security impact, the backend enforcement, the affected data or operation, and the conditions required to reproduce the issue.
A useful finding should include:
- A concise title and affected component.
- The security condition and relevant test setup.
- Reproducible evidence with sensitive values redacted.
- Practical impact and scope.
- Severity rationale.
- Specific remediation steps.
- Retest criteria and the result after remediation.
6. AI Agent Pre-Production Security Checklist
Use this checklist during release reviews and repeat relevant tests when prompts, tools, models, retrieval sources, permissions, or execution infrastructure change.
- Every connected tool has a documented purpose and owner.
- Tool permissions follow least privilege.
- Authorization is enforced outside the model.
- Direct and indirect prompt injection are included in test coverage.
- Retrieved documents and memory respect access boundaries.
- API arguments are validated before execution.
- Sensitive actions require the intended approval.
- Execution is isolated and network access is restricted where appropriate.
- Credentials are scoped, protected, and rotated according to policy.
- Security-relevant actions and denials are logged.
- Resource and execution limits are tested.
- Previous security failures have regression tests.
- Findings are assigned owners and retested after remediation.
For a structured approach, consult the OWASP LLM security guidance and the OWASP GenAI Security Project. Map the applicable controls to your architecture and risk model rather than assuming that one checklist covers every AI deployment.
7. Frequently Asked Questions
What is AI agent security testing?
AI agent security testing evaluates the security of an AI-powered system that can use tools, retrieve information, call APIs, maintain memory, or perform actions. It tests the model-related behavior as well as the application's identity, authorization, data, tool, and execution controls.
How is AI agent testing different from chatbot testing?
A conventional chatbot may primarily generate responses, while an AI agent can also invoke tools and change system state. Agent testing therefore needs to validate action permissions, backend authorization, execution isolation, approval workflows, and the effects of multi-step tool use.
Can prompt engineering prevent prompt injection?
Prompt instructions can help guide expected behavior, but they should not be treated as a security boundary. Sensitive operations must be protected by independent authorization checks, strict tool permissions, validation, and appropriate isolation.
Should AI agents be penetration tested before production?
Systems that can access sensitive information or perform consequential actions should undergo risk-based security testing before production and after significant changes. The scope should include the agent, its tools, APIs, data sources, permissions, and relevant infrastructure.
Does passing these 12 tests mean an AI agent is secure?
No. These tests cover selected risk areas, not every possible vulnerability. A complete assessment should be tailored to the architecture, threat model, data sensitivity, integrations, business impact, and applicable requirements.
Does automated scanning replace manual AI security testing?
No single scanning approach covers every AI-agent failure mode. Automated checks can help with repeatability and known conditions, while manual analysis can investigate context-dependent behavior, authorization boundaries, complex workflows, and business impact. The appropriate combination depends on the system and assessment scope.
Conclusion
AI agent security depends on controlling what the system can access and do—not just on whether the model produces a safe-sounding answer. The most important tests examine prompt injection, tool permissions, authorization, sensitive data, retrieval isolation, approval controls, and the security boundaries around execution.
Start with the agent's capabilities and trust boundaries, define expected outcomes, test the backend controls independently, and retain evidence that can be reproduced. Then turn validated findings into remediation tasks and regression tests.
For related reading, explore these Nexoryn Security guides:
- AI Agent Security in 2026
- AI Agent Authentication and Authorization
- MCP Security in 2026
- More cybersecurity research from Nexoryn Security
Need Help Assessing Application Security?
Nexoryn Security provides manual-led VAPT and application security testing within an agreed scope. If you need to assess a web application, API, mobile application, network, or related system, discuss your requirements with our team.
AI agent and MCP security testing should be explicitly scoped and confirmed before engagement; do not assume every assessment service or automated scanning product covers these systems.
Editorial note: This article is an educational test-planning guide, not a certification or guarantee of security. Adapt all procedures to the target architecture and obtain authorization before testing.
Comments
Post a Comment