For decades, software security has been built around a relatively stable assumption: a program executes instructions according to rules defined by its developers, while security systems attempt to control which instructions it can execute, what data it can access and which external systems it can reach.
AI agents complicate that model.
An agent is not simply another application with a larger feature set. It combines a probabilistic model with tools, external data, memory, permissions and the ability to determine what action to take next. Instead of executing a predetermined sequence of instructions, it can interpret an objective, choose among available tools, observe the results and continue operating.
That architectural change creates a different security problem.
The question is no longer only whether software contains a vulnerability. It is also whether an AI system can be induced to use legitimate capabilities in an unintended way.
This is why agent security is emerging as a distinct engineering discipline. The central challenge is shifting from protecting software from unauthorized instructions to controlling what an autonomous software system is capable of doing when instructions, data and circumstances change.
From deterministic software to adaptive systems
Traditional software can be complicated, but much of its security architecture is based on predictable execution. Developers define functions, permissions and interfaces. Authentication determines who can access a system, authorization determines what that identity can do, and security controls restrict access to resources.
AI agents introduce another layer between authorization and execution.
A simplified agent architecture looks like this:
User objective → model reasoning → tool selection → external action → result → new reasoning → next action
The model may decide which tool to invoke based on information it has encountered during execution. That information might come from a trusted database, a webpage, a document, an email or another software system.
This creates a crucial distinction between instructions supplied by the user and instructions encountered by the agent.
OpenAI has described prompt injection as a major security problem precisely because external content can contain instructions designed to influence an agent’s behavior. In an ordinary browser, a webpage is primarily information for a human. In an agentic system, that same webpage can become input to a machine capable of taking actions.
The security boundary therefore becomes much less straightforward.
A malicious instruction does not necessarily need to compromise the underlying application. It may instead attempt to persuade the model to misuse capabilities that the application legitimately provides.
Why a sequence of legitimate actions can become dangerous
One of the most important differences between conventional software and agents is that security failures can emerge from sequences of individually permissible actions.
Consider a simplified example.
An agent is authorized to read a document. It is also authorized to send an email. It can access a web service and create a file.
Each permission might be reasonable on its own.
But if the agent is manipulated into reading sensitive information, transferring that information into a new file and then sending it externally, the resulting behavior may violate the user’s actual intent even though no individual permission was necessarily illegal or technically broken.
This is a systems problem rather than a single-function vulnerability.
The risk emerges from the interaction between:
- the model’s interpretation of instructions;
- the information it receives;
- the tools available to it;
- the permissions associated with those tools;
- the sequence of actions it chooses;
- and the safeguards applied between actions.
Research from the UK’s AI Security Institute illustrates how consequential this distinction has become. Its analysis of 177,436 agent tools found a substantial shift toward tools capable of taking actions in external environments. Action tools accounted for 65% of observed tool usage by the end of the period studied, compared with 27% at the beginning. The research also identified tools capable of higher-stakes activities, including financial transactions.
The significance is architectural: an agent’s security exposure grows as its action surface grows.
The tool layer is becoming part of the security perimeter
For conventional applications, developers typically think about APIs and software components as interfaces that execute authorized commands.
With agents, tools become something closer to extensions of the model’s operational capabilities.
A web-search tool provides access to information. A file-management tool provides access to local data. A coding tool can modify software. An email tool can communicate externally. A financial tool can potentially initiate transactions.
The model itself may not directly contain the capability to perform these activities. Instead, it chooses when to invoke external tools.
That creates a new security layer between the AI model and the outside world.
The AI Security Institute’s research is significant because it suggests that understanding agents may require monitoring not only the model but also the tool ecosystem surrounding the model. Its dataset showed that software development dominates current agent-tool activity, while tools enabling consequential actions are expanding.
This is one reason agent security cannot simply be treated as another form of model evaluation.
A highly capable model with no external permissions has a different risk profile from the same model connected to corporate databases, production systems and financial services.
Capability and access are separate variables.
Security architecture has to manage both.
Memory changes the security model again
Memory introduces another important dimension.
A conventional software process may maintain state, but AI agents can combine persistent information with probabilistic reasoning about what that information means and what should happen next.
This creates new questions around the lifecycle of information.
If an agent remembers a previous instruction, should that instruction remain authoritative indefinitely?
If it stores information obtained from an untrusted source, can that information influence later actions?
If several tools or agents share context, how should the system determine which information is trustworthy?
These questions connect AI security to longstanding principles in information security, including least privilege, data isolation and access control. But the implementation becomes more complicated when the system must also reason over the information it stores.
The important change is therefore not simply that AI “has memory.” It is that memory can become part of an agent’s future decision-making process.
That makes data provenance and context management security concerns rather than merely usability features.
Human approval is not a complete solution
A natural response to autonomous-system risk is to put a human in the loop.
For some actions, this remains an important safeguard. But human approval has a structural limitation: people cannot reliably inspect every low-level decision made by a system that operates continuously.
Anthropic’s engineering research provides an unusually concrete example. The company reported that users approved approximately 93% of permission prompts in one of its agentic coding environments. The company describes this as a form of approval fatigue: if users encounter frequent requests, they can become less attentive to individual decisions.
This creates an important security lesson.
A security control that requires humans to evaluate too many decisions can gradually become a rubber stamp.
The implication is not that humans should be removed from the system. Rather, human oversight needs to move toward the decisions that actually require human judgment.
Routine actions can be constrained technologically.
High-impact actions can require stronger authorization.
Exceptional behavior can trigger human review.
This resembles the evolution of other safety-critical systems, where controls are designed around the consequences of failure rather than requiring a person to manually approve everything.
Containment becomes an engineering discipline
As agents become more capable, the question becomes not simply “Can the agent make a mistake?” but:
How much damage can the mistake cause?
This is the concept of blast radius.
Anthropic’s published engineering work describes containment as one of the primary mechanisms for limiting that blast radius. Instead of attempting to predict every undesirable action, systems can restrict what the agent is technically able to reach through mechanisms such as sandboxes, virtual machines and network egress controls.
This represents a major architectural shift.
Traditional security often attempts to determine whether an action is legitimate before allowing it.
Agent containment increasingly asks a different question:
What if the agent makes the wrong decision?
If the system is isolated from sensitive resources, the consequences can be limited even when behavioral safeguards fail.
This is particularly important because AI behavior is probabilistic. Even a model that performs extremely well under evaluation cannot be assumed to behave perfectly in every possible environment.
Containment therefore provides a second line of defense.
Identity and authorization have to evolve
The emergence of autonomous agents also exposes a problem that conventional software infrastructure has historically handled through user identities and service accounts.
Who is the agent?
What is it authorized to do?
Who authorized it?
How long should that authorization remain valid?
Can it delegate permissions?
Can one agent authenticate another?
NIST’s 2026 concept paper on software and AI-agent identity and authorization explicitly identifies these questions as an emerging standards problem. It proposes applying identity and authorization practices to software agents because agents increasingly interact with diverse datasets, tools and applications.
This is potentially one of the most consequential infrastructure changes associated with agentic computing.
If AI agents become persistent participants in enterprise software, they cannot simply operate using broad credentials assigned to the human who launched them.
Agent identity will need to become a first-class infrastructure concept.
That points toward more granular authorization, scoped credentials, audit trails and policies describing what an agent is permitted to do in a particular context.
Security evaluation must become sequential
Traditional software testing often asks whether a specific function behaves correctly.
AI-agent testing must account for sequences.
An agent can behave correctly during one interaction and fail after several rounds of tool use. An apparently safe action can become dangerous when combined with information received later.
Anthropic’s research on agent evaluations emphasizes precisely this problem: agents operate across multiple turns, modify state, call tools and adapt based on intermediate results. Consequently, evaluating them requires methods that reflect the complexity of their execution rather than testing only isolated responses.
The UK’s AI Security Institute has reached a similar conclusion through large-scale red-team testing. In one competition involving 22 frontier agents across 44 realistic deployment scenarios, researchers collected 1.8 million prompt-injection attacks, with more than 60,000 successful policy violations. The study found that robustness did not consistently correlate with model size or capability, indicating that simply making the underlying model larger is not a sufficient security strategy.
This matters because it separates model intelligence from system security.
A smarter model may be better at recognizing some attacks, but security ultimately depends on the architecture surrounding the model.
Agent security is becoming its own technology layer
Taken together, these developments point toward a new software-security stack.
At the bottom are conventional controls:
identity → authentication → authorization → network security → operating-system isolation
Above them sits the AI-specific layer:
model behavior → context → memory → tool selection → policy enforcement → agent evaluation
And surrounding both is a containment layer:
sandboxing → data boundaries → egress controls → transaction limits → monitoring → auditability
This layered approach is becoming necessary because no single control can reliably solve the problem.
Model training cannot eliminate every unexpected behavior.
Prompt filtering cannot prevent every manipulation.
Human approval cannot scale indefinitely.
Permissions cannot anticipate every sequence of actions.
Containment cannot make an unrestricted system safe, but it can reduce the consequences of failure.
The architecture therefore has to assume that individual defenses will sometimes fail.
Industry implications: security moves closer to the operating system
The short-term effect is likely to be an expansion of security tooling around AI agents. Developers will need mechanisms for managing tool permissions, evaluating agent behavior, isolating execution environments and monitoring actions.
OpenAI is already developing agentic security capabilities for software development, while Anthropic has described containment architectures across its agentic products.
The longer-term implication is deeper.
If agents become a standard interface for enterprise software, security controls may increasingly move below the application layer. Instead of trusting an application simply because it has been authenticated, infrastructure will need to determine what an autonomous agent is allowed to do at each stage of its operation.
That could create a new ecosystem around agent identity, policy engines, secure tool registries, runtime monitoring, isolated execution and automated security evaluation.
In other words, agent security may eventually resemble the evolution of cloud security: initially treated as an extension of existing infrastructure, then becoming a specialized architecture with its own standards, vendors and operational practices.
The future trajectory: controlling capability rather than predicting behavior
The most important technological shift is not that AI agents can make mistakes. Software has always made mistakes.
The difference is that an AI agent can interpret ambiguous information, choose among multiple actions and continue operating without a developer explicitly specifying every step.
That makes perfect behavioral prediction an unrealistic security foundation.
The more durable approach is architectural.
Systems need to constrain what agents can access, distinguish trusted instructions from untrusted information, limit the consequences of individual actions, establish clear identities and permissions, and continuously evaluate behavior across realistic sequences.
This does not eliminate uncertainty. It changes where security responsibility resides.
Instead of asking only whether the model can be trusted, engineers can ask whether the entire system remains acceptably safe when the model is wrong.
That is a much more useful engineering question.
Conclusion: The Security Boundary Is Moving
AI agents are forcing software security to evolve because they combine probabilistic reasoning with tools, memory, permissions and autonomous execution.
The resulting risks cannot be addressed solely by improving the underlying model. The critical security boundary increasingly exists around the complete agent: its identity, context, tools, data, network access and ability to act.
This explains why containment, authorization, agent evaluation and runtime controls are becoming increasingly important.
The deeper transition is from securing software that follows instructions to securing software that interprets instructions and decides what to do next.
That is the fundamental architectural challenge of agentic computing.
The future of AI security will therefore depend less on finding a model that never makes a mistake and more on building systems in which mistakes remain bounded, observable and recoverable.
