AI Agents Shouldn’t Set Their Own Security Boundaries

AI agents use tools, reach into internal systems and get work done with barely any human involved. New security problems turn up almost weekly. What if an agent’s own instructions block it, and it finds a way around them just to finish the job?

At NVIDIA’s GTC Berlin I was at a session where Ali Golshan, Senior Director of AI Software at NVIDIA presented an approach that treats agent security as a problem spanning the entire software stack, from the application and runtime to the underlying infrastructure.

His central argument was that security cannot depend solely on whether a model follows instructions.

Enforce rules where agents can’t reach

One big issue Golshan pointed out is that agents don’t understand restrictions the way developers meant them to. If an agent can’t get to a tool or resource it might look for another way to get the job done. In systems this can lead the agent to drift from its original goal or even find a hidden path to sensitive data. As he said:

They just see friction versus no friction.

The point is that an agent may treat a security restriction as an obstacle to overcome rather than a boundary it must respect. If controls exist only in the application layer, where the agent can reason about them, they may not provide a sufficiently reliable enforcement mechanism.

Ali argued that security controls should therefore be moved into layers that operate independently of the agent. The agent can propose an action, but a separate mechanism should determine whether that action is permitted and enforce the decision. This means separating an agent’s ability to plan from its authority to act.

Every agent can follow the rules while the system as a whole breaks them

The problem becomes more complicated when multiple agents collaborate. Golshan described a scenario in which one agent can access GitHub but cannot access the public internet, while another can publish content online but has no access to GitHub.

Each agent operates within its individual permissions. Yet if the first agent delegates work to the second, or passes information between them, their combined capabilities may exceed the limits an organisation intended to enforce:

However, you have three agents that their collective capabilities now exceed your global policy. How do you verify these unintended exfiltration paths?

This illustrates why checking permissions for individual agents may not be enough. In multi-agent systems, security controls must also account for delegation and the combined effects of actions across an entire workflow.

Longrunning agents introduce another challenge. As they work, call tools or delegate tasks, their access requirements can change. Persistent credentials and permissions may accumulate unnecessary risk if they remain active after a task is complete.

A more restrictive approach is to grant agents only the permissions they need, for as long as they need them, and revoke access when the relevant work is finished.

The risks also grow as agents operate for longer periods. Ali pointed to long-running agents and multi-agent fleets as two developments that make security harder to manage. Over time, an agent may accumulate access, credentials and session state, while collaboration introduces risks that cannot be assessed by looking at each agent in isolation.

How to solve this?

He advocated a layered approach that combines isolation, narrowly scoped permissions, monitoring and the ability to revoke access quickly.

The principle is to reduce the damage an agent can cause if it makes a mistake, is manipulated by malicious input or exploits a vulnerability in its environment. Sensitive credentials should remain outside the agent’s direct reach wherever possible, while access to external systems should pass through independently enforced controls.

NVIDIA presented its own implementation of this approach through OpenShell, an agent runtime, alongside infrastructure-level controls associated with its BlueField platform and Sentry security framework.

The proposed architecture aims to move some enforcement beyond the software environment in which an agent operates. The broader idea is that an agent should not be able to change or bypass the security controls governing its own execution.

Ali described this as a progression from application-level controls to runtime isolation and, ultimately, hardware-level enforcement. The reasoning is that a runtime is still software and may itself contain vulnerabilities. NVIDIA’s proposed architecture aims to place some security functions in a separate infrastructure layer that the agent cannot modify or bypass directly.

Agent security can’t be solved one vendor at a time

Enterprise environments combine different models, cloud platforms, tools and internal systems. If each provider implements its own incompatible approach to identity, permissions and delegation, organisations may struggle to enforce consistent controls across their infrastructure. Shared approaches to agent identity, authorisation, delegation and auditability could let them apply one security policy across different systems. NVIDIA pointed to the Open Secure AI Alliance as work toward open standards for agentic systems.

Before deploying agents, ask what each one can access, what it can change, and whether those limits still hold when several agents work together.