Microsoft Agent Safety treats secure agent operation as shared responsibility between the framework and application developer. The model cannot be the authority that protects itself.
An agent is secure only when a manipulated model still cannot exceed its deterministic permissions.
What threats target AI agents?
Attackers can use prompt injection hidden in an email, webpage, file, or tool result to redirect the agent’s goals. A compromised step may try to extract data, poison agent memory, misuse credentials, or persuade another system to accept an unsafe action. Every external input and tool result remains untrusted, even when it looks like an instruction the agent should follow.
Agent failures can also begin without an attacker. Ambiguous goals, malformed tool outputs, excessive permissions, and repeated retries can produce the same harmful action. Security therefore has to limit impact as well as detect malicious intent.
Where should the security boundary sit?
The boundary belongs between model output and every capability that can read sensitive data or change state. Tool schemas validate structure, authorization checks validate identity and scope, and policy code decides whether the requested operation is allowed. The model may request an action, but deterministic code must authorize and execute it.
Guardrails can detect risky content or suspicious patterns, but they do not replace access control. Credentials should be scoped to the smallest dataset and operation the agent needs. Sandboxed code execution should restrict files, processes, and outbound network access when the agent can run code.
How do teams contain an agent failure?
Use least privilege, isolated execution, outbound network limits, immutable logs, and tool approval before actions involving money, external communication, deletion, or production data. Approval must bind to the exact action and parameters a person reviewed. Approving a draft should not grant the agent permission to change the recipient or amount afterward.
Recovery also matters. Operators need a way to stop the agent, revoke credentials, identify completed actions, and reconcile external writes whose outcome is uncertain. Durable execution can preserve workflow state, but it must not blindly repeat an action after a crash.
How should AI agent security be tested?
Start from the trust boundaries, not a list of clever prompts. Test whether untrusted content can change tool selection, arguments, memory, destinations, or data access. Then test whether each deterministic control blocks the attempt. A security test passes only when the harmful action is impossible, not when the model politely refuses once.
AI red teaming should prove that every important boundary rejects realistic attacks, including indirect instructions inside retrieved documents and compromised tool responses. LLM observability should make attempted violations visible without logging secrets that create a second exposure.