COMMENTARY: On October 16, CISA, the NSA and the Commerce Department released joint guidance for AI developers. It was prompted by incidents in which AI models escaped their testing environments (think Hugging Face), and several of the companies involved said so publicly.
When regulatory agencies combine to issue guidance urging AI developers to strengthen security, we should pay a lot more attention than we do. These warnings reflect the difficulty of keeping increasingly capable systems within the boundaries we defined for them.
[SC Media Perspectives columns are written by a trusted community of SC Media cybersecurity subject matter experts. Read more Perspectives here.]
They are rooted in the reality that teams are failing to do so. And they are alarms that we can only ignore so long before the industry changes and we realize why we should have been paying attention sooner.
New technology, same old mistake
After more than 40 years in technology, I have watched us repeat the same mistake in cycles: if a technology accelerates progress, we assume that progress is inherently good. But speed, capability, adoption, and even economic growth are seldom a net benefit to the tech industry. The world does not operate on “Keep what works safely, discard what’s risky regardless of performance.” It works on “What’s easiest and most profitable, regardless of risk.”
Step by step that’s becoming our undoing.
Just look at real comparisons: the internet connected the world, but it also amplified misinformation and gave cybercrime unprecedented reach. Smartphones put limitless information in our pockets, while turning human attention into a commodity. Social media created powerful mechanisms for influencing behavior at scale under the guise of "connection." Nuclear weapons offered deterrence, while introducing the possibility of civilization-ending consequences.
Every one of these technologies delivered genuine benefits. That’s precisely why their risks were so easy to rationalize, defer, or as history has repeatedly shown, ignore. But we need to start recognizing the negative consequences that come along with those benefits before they become destructive. Failure to do so is short-sighted at best -- and reckless in almost all cases.
AI routinely surprises its builders
Today, we face a major challenge with AI, but with a critical difference. We are developing systems whose capabilities can evolve beyond our ability to reliably anticipate their behavior, or even understand. When AI developers themselves raise concerns about models testing boundaries or behaving unexpectedly, we cannot simply trust that the next generation will run more safely, or that the next AI will counter the threat.
In fact, the next model often becomes progressively better at evading our understanding and ignoring parameters. The systems are designed to do that! Yet they are operated as if the creator should be an exclusion, and the systems often do not recognize that distinction.
The agencies calling for stronger safeguards recognize the core problem. They’re being responsible by warning everyone. Technical controls must form part of the architecture from the start, and we must make them immutable, unable to be changed without physical access external to the system.
Developers need independent testing before they deploy systems, with monitoring in place to detect failures. They also need reliable shutdown mechanisms that they have tested under realistic conditions.
Will we heed this warning while we have the ability to act, risking that competitive pressure and immediate commercial advantage will once again persuade us that the risks can wait. We missed the boat on all the other technologies and have had to struggle to contain what they mean to privacy and security for decades. If we do not get a handle on AI soon, we’ll soon do this even more at a larger scale, with power we cannot even begin to imagine yet.
The guidance gives developers a place to start, but following it will require time and money, and a commitment from the top down. When looked at through the lens of resilience vs. profit, pausing to understand seems more reasonable than profiting our way to our own undoing.
Will business leaders support that decision, even if the pause means safer business foundations vs. immediate growth?
There’s a first time for everything, right?
Poison can be medicine and medicine can be poison, but metered dosing controls the outcome.
Right now I believe AI is poison with medicinal potential. So, will we recognize that potential, or keep drinking it?
Gene Moody, Field CTO, Action1
SC Media Perspectives columns are written by a trusted community of SC Media cybersecurity subject matter experts. Each contribution has a goal of bringing a unique voice to important cybersecurity topics. Content strives to be of the highest quality, objective and non-commercial.
