Why security needs to become an integral part of AI development

In this article:

Companies developing AI models are innovating so quickly that it's hard to keep up. Chipmakers and cloud-storage companies make billion-dollar deals with AI makers that dazzle Wall Street. Organizations from the largest enterprises to the smallest mom-and-pop shops use AI to streamline efficiency, boost productivity and find new paths to profits.

One important aspect of software development is being left behind, however. Most companies that use AI don't really know how to secure it. Many developers of AI models treat security as an afterthought or, worse, as an obstacle in the race to be the first across the finish line.

"AI is moving very fast. Whoever gets there first is going to be the pioneer," says Damien Lim, AI Product Marketing Evangelist at Tenable, adding that AI companies may "have less constraints because they have to build fast and get it out the door."

For more information:

A common refrain heard at this year's security conferences was that AI developers are ignoring all the hard-earned security lessons of the past few decades — and that AI security today is what Windows security was like in the 1990s.

That clearly needs to change. Information-security teams need to work with AI developers, but as collaborators, not critics. AI models need to be penetration-tested and red-teamed like any other kind of software, but because AI is fundamentally unpredictable, it will take additional measures to make AI secure.

Why AI developers may disregard security

Any software developer wants to get their project completed as quickly as possible and doesn't want to be held up by a security checklist.

"Security best practices, like any developer who is asked to do documentation, it's very tedious," says Lim. "If I have to apply security practices to what I'm working on, it's going to slow me down."

But, Lim adds, those best security practices are often built into the development process at established companies where the value of stopping to check before production begins is understood.

"In most companies, at least the larger enterprises, there is a process," he explains. "There's check-ins. You develop this, you know who's doing dev tests or dev QA. Even before you get to the finish line, there need to be some checks and balances."

Those checks and balances may not always be present in the "move fast, break things" start-up culture of AI model building, where developers often work 12-hour days, six days a week.

"It's not like a traditional large enterprise," says Lim. "For some of these younger startups, is that process there?"

It may also be that AI developers lack familiarity with the kinds of tried-and-true best practices that have been applied in the recent past.

"They're new-generation coders, developers and programmers, and they don't have the same level of experience," Lim says.

Some AI companies may also be deliberately downplaying security at the highest levels, Lim worries. He cited examples of AI security managers who brought up safety issues being replaced by more compliant managers.

"The CEO didn't agree. It's like out the door, let's get another person in," he says. "That's a cause of concern for me when people rotate like that and they are bringing up legitimate concerns."

How AI is different and more dangerous than regular software

It's even more worrisome when you consider that generative AI models and the agentic AI models that use them are fundamentally different from other types of software.

With most kinds of software, if you input the same data or commands, you'll always get the same results. It's deterministic and predictable. It follows the rules.

With AI, you never know what you're going to get. It's unpredictable and "non-deterministic," a nice way of saying chaotic. And you can't always tell whether it's chaotic good or chaotic evil.

Large language models, which input and output text, can hallucinate false information, deny the validity of true information, or be tricked into divulging secrets. AI agents further expand the attack surface because they use LLMs to help make decisions and interact with other applications to gather data and then take actions.

"Typically, what happens in agentic AI is there's an orchestrator, they take whatever input from the user, but then they distribute that workload to different agents. And you don't know what's going on with the agents," explains Lim. "The most important thing is they are able to take action. That's the real kicker. You don't know what actions they are taking."

He gave an example of a financial AI gaining access to Excel files to gather data, which sounds fine — until the AI is misconfigured so that it can not only read, but also write data in those files.

"As part of me analyzing and being a good AI, I'm going to go ahead and update some of these rows," Lim imagined the AI as saying to itself.

 Or, he said, the AI agent might be publicly exposed, letting anyone manipulate its output. This year's Black Hat conference documented several real-life examples.

"I think there are some of these consequences if you don't treat AI agents properly with the right security concerns," says Lim. "They can go rogue and that could be a huge issue."

Because of this unpredictability and the wide potential for harm, generative AI and AI agents need to be treated as their own threat models.

As the OWASP GenAI Red Teaming Guide puts it, "GenAI Red Teaming must accordingly expand the definition of adversary to include the model itself, and the output generated by it, and must include an evaluation of the risks from harmful or misleading responses produced by the underlying models."

But, as Lim points out, not enough people are thinking that way.

"I don't think anyone has really thought about threat models or how to even address that, at least not today," he says. "Right now, a lot of people are just scrambling to build AI agents as fast as they can because that is the hot thing. Everyone wants it."

How to safeguard your AI development

Thanks to lots of hard work being done at OWASP, Tenable and other organizations, frameworks for testing the security of AI models are being developed.

Besides its Red Teaming Guide, OWASP has developed a Top 10 vulnerabilities list for LLMs, is working on a similar list for AI agents, has codified best practices for LLMs and GenAI and has listed agentic AI threats and mitigations.

Tenable has developed a cloud-based AI security posture management (AI-SPM) tool and more recently brought it down to Earth with AI Exposure, a component of the Tenable One exposure-management platform.

Both can be used, Lim explains, to check the security of and provide visibility into AI models and tools in development as well as in production.

"A lot of solutions today cannot give you visibility to the agent level," he says. "But AI Exposure is able to then showcase that so you can see exactly what [the AI agents] are doing, who they are talking to and what tools are integrated with that."

"It's no longer a black box," he adds. "We show you everything. And not only that, AI Exposure also provides remediation. If you think, 'Hey, one of your AI agents has gone rogue,' you can isolate the agent, you can even delete the agent."

As for AI-SPM, he says, "it looks at the AI ecosystem, how you connect to data stores, how the libraries work with the AI models and are there any misconfigurations happening? Who has access to your data store?"

Beyond the tools and frameworks, Lim says, there are a few steps that can be taken to make AI code more secure.

Developers should use AI code-analysis tools to inspect every piece of third-party AI code they get from GitHub, Hugging Face or other open-source repositories.

"When you first pick up the code from Hugging Face, for example," he says, "I think it behooves whoever the developer is to really do a quick analysis and say, 'Hey, you know what, this code is safe, it's fine, let me download the repository and work on it,' versus, 'I'm going to trust that that code that I'm going to get is secure.'"

Developers and infosec teams need to share AI code-analysis tools and see the same outputs.

"They should share that platform, whatever the AI tool is," Lim says. "If I notice an AI developer has downloaded something that looks suspicious, I would tap the shoulder of the developers and say, 'Hey, you know what, we just checked, maybe there's a low severity or medium or even high. Like, why are you doing this? I just want to ensure that we are on the same page.'"

The infosec and development teams need to collaborate, not point fingers at each other.

"The underlying key takeaway is that the security teams hopefully are not obstacles. They're not creating friction," Lim explains. "Because if you create friction, what's going to happen is then you get shadow AI development, which is not what you want. Because they can easily download it on their laptops and say, 'You know, I'm going to do it myself. Goodbye.'"

Use AI-based tools to pen-test and fuzz AI models.

"If you have an AI tool that understands what proper guardrails are, and it can adapt to new attacks, you could use this AI tool to then test your AI," Lim says. "I don't think traditional security tools are able to do that."

Use social engineering to pen-test AI models.

One of the most common attacks on AI models is prompt injection, essentially tricking an LLM to divulge sensitive information or otherwise break its own rules. Another is data poisoning, in which you add malicious or false information into the training data to poison later output. Neither of these attacks involve sophisticated code; you're just trying to fool the AI like a con man would fool a mark.

Apply statistics to make sure you're getting it right.

Because AI is non-deterministic, you can't tell what's going to happen every time. But you can make a good guess about what will happen most of the time.

"Unlike traditional systems, GenAI involves probabilistic outputs, which requires statistically rigorous testing methods to assess vulnerabilities," says OWASP's Red Teaming Guide. "The stochastic nature of the generative AI systems means determining successful attacks vs normal model behavior variations is more complex than traditional red teaming."

Consider real-world threats and real-world implications when evaluating GenAI and agentic AI.

"To create a threat model, you should apply social, political, and cultural contexts against common adversarial test scenarios to ensure that tests consider technical vulnerabilities in the system," says the OWASP Red Teaming Guide, "plus the wider implications of how that system may be used or misused in diverse environments and communities."

Keep the humans in the loop

Those are the best practices so far when it comes to testing and analyzing AI models primarily developed by humans. But what about when AI itself develops AI? Lim says it's already happening.

"The reason why [the Chinese AI model] DeepSeek was so fast in its innovation is it's actually AI training AI," he explains. "Using a second generation of AI that the first AI model in DeepSeek was training."

And that's very risky, he adds, because "we do not know if we've trained AI properly."

"My scary thought is this is from a Chinese company," he says. "They don't have any guardrails. They're like, yeah, go for it."

For now, Lim thinks humans need to remain part of the AI development process.

"I think responsible AI is a consideration," he says. "How humans can be part of that process and ensure that we build agents securely."

Please visit our exposure management topic page.