When AI Agents And Attackers Look Alike, What Should Defenders Test?

Jon Baker is VP, Threat-Informed Defense at AttackIQ.

gettyAn AI agent attempts to access a sensitive system outside its assigned role. Its credentials are valid, and it belongs to the organization. Is it trying to complete a legitimate task, following an instruction an attacker planted or running under credentials an attacker now controls?

While that distinction matters for the investigation, it should not determine whether defenders take the access attempt seriously. An internal agent can create an opportunity for an external adversary, and a trusted identity can perform an unauthorized action.

The immediate questions are operational: What could this identity reach? What happened around the attempted access? Which control should have interrupted the activity? Each can be answered with evidence rather than a theory about intent.

An agent’s assigned task and its technical permissions are two different things. A task description might tell an agent to summarize support cases, while the account behind its tools can access a much wider set of records. The task describes intent, but permissions determine what actions connected systems allow.

OWASP identifies excessive functionality, permissions and autonomy as root causes of excessive agency. Its guidance calls for enforcing authorization in downstream systems rather than leaving the model to decide whether an action is allowed.

For defenders, the useful starting point is a map of the agent’s effective access. Include every account’s tools use, along with delegated access and shared service accounts. A dedicated agent identity provides little isolation if one of its tools connects through a broadly privileged account. Then compare that access with the business task. Any difference gives the team a specific boundary to test.

Consider a hypothetical support agent that reads customer cases, retrieves troubleshooting documents and sends a response. Each capability supports a legitimate workflow, but it also raises a question: Could the agent retrieve material the customer isn’t entitled to see and include it in the response?

A permissions review of each application may miss that combination. The test needs to follow the workflow from the customer’s request through retrieval to the outgoing message, checking where the customer’s authorization is preserved or lost.

Start with a critical resource and work backward. Which tool can reach it? Which identity does that tool use? What conditions allow the agent to invoke it? The resulting path gives defenders a concrete test objective, rather than a general instruction to assess AI risk.

An unusual resource request is a signal to investigate, not proof of malicious intent. Examine the activity before and after it. Did the agent encounter untrusted content? Did it switch tools, request additional privileges or begin collecting information unrelated to the task? Was an attempted transfer blocked, or did the workflow continue through another route?

The ForcedLeak vulnerability that Noma Security disclosed in September 2025 shows the full sequence in a production product. An attacker could place hidden instructions in the description field of a Salesforce Web-to-Lead form. When an employee asked an Agentforce agent to handle the lead, the agent carried out the planted instructions alongside the legitimate request, queried the CRM for lead records and sent the results to an external domain inside an image URL. That domain sat on the Content Security Policy allowlist and had expired, so the outbound control that should have stopped the transfer did not. Salesforce has since enforced a trusted URL allowlist on agent output. MITRE ATLAS documents the same pattern as case study AML.CS0059, EchoLeak, a zero-click prompt injection against Microsoft 365 Copilot that used the same image-URL route to move data out.

The investigation should connect the initiating request, tool calls, identities and downstream actions. Where that connection is missing, record the visibility gap as a test finding. A team that cannot reconstruct the sequence will struggle to explain what happened or decide which access to revoke.

Adversarial exposure validation (AEV), the category Gartner uses for breach and attack simulation and automated penetration testing, can help establish whether a plausible route can succeed and whether the defenses along it operate as expected. For the support-agent example, use synthetic records to test whether access restrictions prevent retrieval and whether outbound controls stop an unauthorized disclosure.

Define the expected result before running the test. A blocked request, a detection and a completed response are separate outcomes. If the activity generates an alert but the information still leaves the system, the detection worked while prevention failed. That distinction should shape the corrective action.

Test containment as well. Can responders suspend the relevant workflow and invalidate its access? After narrowing permissions or changing a control, rerun the scenario to confirm the route is interrupted while legitimate support work still succeeds.

Continuous threat exposure management (CTEM) provides a way to keep this work tied to business priorities as the environment changes. A new connector, broader permission or revised workflow should trigger a review of the paths affected and the evidence needed to validate them.

Give each failed test an owner and a clear closure condition. Closing a ticket should mean the relevant route has been retested successfully, or the remaining opportunity has been explicitly accepted. That keeps the work focused on reducing threat debt, the adversary opportunity the organization carries.

Defenders may not immediately know why an agent crossed a boundary. They should know where that boundary is enforced, what evidence shows it held and who can stop the workflow when it does not. Those are questions a security team can test before granting an agent more authority.

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?