Hard to handle: Securing AI-generated code and the AI agents that write it

The adoption of AI-generated code is transforming software development, but it's also introducing a new class of security vulnerabilities that organizations are only starting to recognize.

In a recent SC Media webcast, host Mike Shema and Legit Security CTO Liav Caspi discussed how AI agents are speeding up code production to previously unseen levels and also changing the developer's role from writing code to guiding and reviewing machine-generated output.

"Today, we have agents speed-running software development. They're creating more code more quickly," said Shema. "But the security challenge doesn't just come from the code being generated. It also comes from the generators themselves."

Application-security practices built around human developers are having trouble keeping pace, especially because AI-generated code often introduces subtle logical flaws rather than obvious coding mistakes.

Application security must adapt to this new development paradigm. Static application-security testing (SAST) and other traditional approaches are still useful, but they're insufficient on their own, due to the volume and complexity of AI-generated code.

Instead, organizations must use AI-driven security tools to detect vulnerabilities in AI-generated code and validate behavior at scale. At the same time, attention must shift from just the code to the agents producing it.

"The level of trust you're going to have in this code contribution is going to be proportionate to, 'Was this agent really secure as it wrote the code? What was the prompt? What were the skills?'" explained Caspi.

Another problem is the unpredictability of AI systems. Unlike traditional software, AI is inherently non-deterministic, meaning the same inputs can produce very different outputs. If you're using AI to generate code, you can imagine that this predictable pattern of chaos might tremendously complicates validation and reproducibility.

Additionally, AI agents are themselves susceptible to manipulation through techniques such as prompt injection. Caspi likened prompt injection to phishing attacks against humans and suggested that an AI overseer might help detect such attempts.

"You're trying to fool the agent to do something, and you need some counterintelligence," he said, "like an intelligent system, to look at the original intent and say, 'Okay, I'm seeing that something is off here.'"

The upshot of the discussion was that while AI-powered code development introduces new risks, it doesn't invalidate foundational application-security principles.

However, organizations must augment these with new strategies like securing the agents themselves, controlling third-party integrations, improving observability into AI activity, and automating security testing to keep pace with development speed.

As AI continues to reshape software engineering, the organizations that succeed will be those that treat AI not just as a productivity tool, but also as a new attack surface that requires continuous oversight and innovation.