AI/ML

The highest risk for AI agents is the software development organization

A hand interacts with AI agents on a laptop screen displaying various DEV OPS interfaces.

COMMENTARY: The enterprise security conversation around agentic AI has focused heavily on customer service, finance, legal and other business functions where autonomous systems are beginning to make decisions or execute workflows. But the organization that may face the greatest immediate risk is much closer to the technology itself — software development.

Development teams are adopting agentic systems faster than almost any other part of the enterprise. Coding assistants have already evolved from tools that suggest snippets into agents that can inspect repositories, modify source code, install dependencies, execute shell commands, run tests, open pull requests and interact with cloud infrastructure. Increasingly, developers are not simply asking AI systems to write code. They are assigning objectives and allowing agents to determine how to accomplish them.

[SC Media Perspectives columns are written by a trusted community of SC Media cybersecurity subject matter experts. Read more Perspectives here.]

That shift creates a uniquely dangerous combination of autonomy and privilege.

Software development environments have traditionally operated with looser controls than many other corporate systems because developers need broad access to do their jobs. Engineers routinely interact with source repositories, package registries, CI/CD systems, development databases, cloud consoles, Kubernetes clusters, API credentials and infrastructure automation tools. Restricting that access too aggressively can slow development, so organizations frequently accept a higher level of operational freedom in exchange for productivity.

Agentic systems are now inheriting that same freedom.


Related reading:


The result is that many companies are unintentionally creating highly privileged networks of autonomous software inside their development organizations while maintaining security controls designed primarily for human users. A developer may initiate a task, but an agent can subsequently make dozens or hundreds of decisions without human involvement. It may inspect files, download packages, call external services, authenticate to infrastructure or launch additional processes as part of completing a single assignment.

The Hugging Face agentic attack provides an important warning about where this can lead. What made that incident particularly significant was not simply that an AI system was involved, but that autonomous activity could move through infrastructure and perform actions at machine speed. That type of behavior becomes especially difficult to distinguish from legitimate activity inside a development environment because agents and developers often use exactly the same tools.

A developer cloning repositories, installing packages, invoking cloud APIs or running shell commands is completely normal. An agent performing those same actions may also be normal. The security problem emerges when the frequency, sequence or context of those actions changes. An agent that suddenly begins enumerating credentials, interacting with repositories unrelated to its assigned task, downloading dependencies from unfamiliar sources or reaching infrastructure outside its expected scope may indicate something very different from ordinary development activity.

Traditional security systems are poorly equipped to understand that distinction because they generally focus on identities, devices, network connections and individual events. They can tell a security team that a process contacted GitHub or that an API credential was used, but they may not understand which agent initiated the action, what objective that agent was attempting to accomplish or what sequence of decisions led to the event.

That is why software development organizations need a much more aggressive approach to agentic network activity monitoring than most companies have implemented today.

Monitoring has to begin with visibility. Organizations need to know which agents are operating inside development environments, who initiated them, what models they are using, which tools they can invoke and what systems they are permitted to access. That sounds elementary, but many organizations cannot currently produce a reliable inventory of the AI agents being used by their development teams.

The next requirement is behavioral context. Security teams need to understand not only that an agent executed a command but whether that command was consistent with the task the agent was assigned. The same command can be harmless in one context and extremely suspicious in another. Agentic network activity monitoring therefore has to connect actions with identities, objectives, tools and downstream consequences rather than treating every event as an isolated security signal.

Detailed activity logging is equally important. Organizations should be able to reconstruct what an agent did during a session, including the tools it invoked, commands it executed, systems it contacted, files it modified and credentials or APIs it attempted to use. Without that visibility, incident response becomes an exercise in reconstructing autonomous behavior from fragments scattered across endpoint, network and application logs.

This does not mean developers should be prevented from using agents. That would be both unrealistic and counterproductive. Software development has always adopted automation aggressively because automation improves speed, consistency and productivity. Agentic development is a continuation of that evolution, and organizations that refuse to adopt it will probably place themselves at a competitive disadvantage.

The security challenge is to make agent autonomy observable.

Development organizations should therefore be treated as the first priority for enterprise-wide agentic monitoring programs. They combine the highest concentration of technical privileges with some of the fastest adoption of autonomous systems and, in many companies, comparatively lax oversight of how those systems operate.

The biggest mistake enterprises can make is assuming that because an agent was launched by a trusted developer, its subsequent behavior should also be trusted.

Agents do not merely execute instructions. They interpret objectives, choose tools and make decisions. Once they are operating across source repositories, cloud platforms and software delivery pipelines, those decisions become security-relevant events.

The software development organization is rapidly becoming the largest experimental agentic network inside many enterprises. It should also become the most closely monitored one.

An In-Depth Guide to AI

Get essential knowledge and practical strategies to use AI to better your security program.
Dennis Zimmer, Codenotary

Dennis Zimmer is a co-founder and the chief technology officer of Codenotary, provider of tools for notarization and verification of the software development lifecycle attesting to the provenance and safety of the code. He has more than 25 years of experience in the IT industry.

Get daily email updates

SC Media's daily must-read of the most current and pressing daily news

By clicking the Subscribe button below, you agree to SC Media Terms of Use and Privacy Policy.

You can skip this ad in 5 seconds