Can we build something great with unreliable materials? This is the question that the industry is currently grappling with on a daily basis. As an AI security company in 2025, we’ve seen little indication that the AI adoption curve has slowed. Quite the contrary, in fact: top-down pushes have escalated:
This is not a small effort, but it’s a solvable one that will be critical to developing AI solutions that provide value for the long-term. If you’re part of an organization that’s currently rushing into the AI fray, do you have the controls and guidance in place to handle hallucinations when they come to call? Or are you trusting that your software or model will prevent them?If it’s the latter, maybe it’s time to phone a friend.
- Meta has announced that “AI-driven impact” will be a part of employee performance exceptions next year.
- Workday requires all employees to have an AI skill development goal.
- In internal memos Microsoft has stated that “Using AI is no longer optional” and is considering performance metrics for the next fiscal year.These are just a sampling of examples in the public eye; in my professional capacity I speak regularly with companies of all sizes that are experiencing similar “adopt-or-else” pressures.
Cause and effect
This aggressive approach shouldn’t come as a huge surprise for some companies; Mark Zuckerberg famously coined the phrase “move fast and break things” to encourage innovation through rapid iteration. That said, as an industry we are breaking things. We have aggressively adopted CI/CD-focused agents that turn around and delete database content during a code-freeze. Startup Enrichlead found itself playing catch-up after its Cursor AI-built application was found to be filled with basic security flaws allowing users to bypass subscriptions and modify data.Fundamentally at issue here is that this technology is new. Companies such as Anthropic that produce the models we are coming to rely on are explicit about the fact that they do not even fully understand how they work. It is a powerful tool, but having a hammer does not make every problem a nail. As industry pressures to adopt AI increase it creates an incentive to apply the tooling regardless of the suitability to the task.Elephant in the room
The technology has certain fundamental risks that must be addressed in any use case. The term ‘hallucination’ has become ubiquitous even among segments of the populace that don’t typically concern themselves with technical details for a reason — from lawyers citing fake cases to Google-provided suggestions to consume rocks, AI hallucinations make themselves known. The challenge we run into as an AI Security Consultancy is a pervasive believe among industry professionals that hallucinations can be prevented.In September, researchers from OpenAI and Georgia Tech put that to rest. The researchers establish hallucinations as an unavoidable component of the system: “…we show that even if the training data were error-free, the objectives optimized during language model training would lead to errors being generated. With realistic training data containing shades of error, one may expect even higher error rates.”What it means for you
As such, this should be a driver for a change in behavior: it is important that we shift focus away from trying to build systems that “can’t” fail into building policies, governance, and operational frameworks that are robust to the irreducible quantity of errors that AI will product. We are not unfamiliar with this challenge—businesses in 1985 operated with similarly unreliable components in their processes (humans are terrible about hallucinating). A robust AI governance model will require a robust set of controls and guidance:- Policy layer: What use-cases are allowed? What level of harm is acceptable? Who owns oversight?
- Technical controls layer: How is data sourced? Do we use technical controls such as retrieval augmentation, uncertainty estimation, prompt design, human review, and logging/monitoring?
- Operational layer: Do we have clear roles/responsibilities (AI safety officer, model owner, domain expert reviewer)? Is there training for users, incident response when hallucination causes harm, or periodic audits?
- Feedback & improvement layer: Do we monitor model outputs, collect data on hallucination incidents, refine prompts? How do we revisit our controls as technology changes and improves?
- Transparency & communication: Do we formalize internal communication (to staff) and external communication (to users/clients) about model limits, risk of hallucination, and where the AI is in use in our tooling?In effect: any company using AI today needs to have a clear framework for both the technical and nontechnical management of the tooling.
