Why regulated industries can’t afford to skip data privacy for their AI projects

COMMENTARY: Picture a compliance officer at a mid-sized regional bank, staring at a Slack message that says: "we're piloting an AI chatbot for customer support."

Their first question isn't: "will it work?" The compliance office wants to know: "where's our customer data going?"

That question, more than any model accuracy score, decides whether the project survives contact with a regulator.

[SC Media Perspectives columns are written by a trusted community of SC Media cybersecurity subject matter experts. Read more Perspectives here.]

Financial services, healthcare, and insurance don't get the same luxury as everyone else. They can't treat data privacy as a "nice-to-have" feature bolted on after launch. A retailer can ship a chatbot, notice it's leaking PII, and patch it in the next sprint. A hospital running the same playbook could likely experience a HIPAA violation, a breach notification to thousands of patients, and a very uncomfortable board meeting.

Large language models (LLMs) are hungry. They'll happily ingest whatever gets fed to them: training logs, chat transcripts, support tickets, spreadsheets someone forgot to scrub. None of that data knows it's protected health information or a customer's Social Security number. It's just tokens to the LLM.

That's exactly why teams building anything AI-adjacent need to think about data flow before they think about prompts: Where does the data live? Who can query it? Does the vendor retain conversation logs for training, and for how long?

Anthropic and OpenAI both publish enterprise data handling terms for a reason: banks and hospitals ask, every single time. Most of these questions have concrete answers sitting in a data processing agreement somewhere. Unfortunately, almost nobody reads that document past the first page.

The paper trail regulators actually want

Auditors don't want a promise. They want evidence: a data flow diagram, a documented retention policy, a record of who accessed what and when. The NIST AI Risk Management Framework gives a decent skeleton for this, even if it reads like it was written by committee. It won't tell us exactly what to build, but it gives auditors a shared vocabulary to check our work against, which matters more than it sounds like it should.

At least four points tend to separate the organizations that pass their audits from the ones that scramble. Here's what they do:

None of this will show up in a demo. But it's the difference between an AI project that quietly ships and one that ends up as a case study in what not to do. Most breach postmortems come down to a shortcut taken at exactly one of these four steps.

It's tempting to assume a big-name AI vendor has got privacy handled, since surely they've thought of everything. They haven't – at least not for our specific regulatory obligations. Of course, a tool like OneTrust or Vanta can help track compliance posture, but the responsibility for what data goes into a model, and what comes out of it, sits with the organization deploying it. Shared responsibility isn't a cop-out: it's just how cloud and AI contracts actually work, whether the pipeline was built in-house or by an AI development company already versed in HIPAA and PCI DSS requirements.

Regulators in the EU, under GDPR and now the EU AI Act, have made this explicit: accountability doesn't transfer just because we bought a subscription. Fines under GDPR can run up to 4% of global annual revenue, which tends to get a CFO's attention fast. U.S. regulators are getting there too, sector by sector: the FTC has already gone after companies for quietly repurposing customer data to train models nobody signed-off on. None of these agencies care whether the model was built internally or bought off the shelf. The obligation travels with the data, wherever it moves inside the organization.

Data privacy isn't the boring paperwork part of an AI project: it's the part that decides whether the project gets to exist past its pilot phase. Regulated industries don't have the luxury of moving fast and fixing it later; the fixes come with fines, breach notices, and a lot of explaining to do.

Organizations must build in the privacy work early, document it honestly, and treat every vendor claim as a starting point for questions rather than an answer. Those are the habits that keep a good AI idea from turning into next year's cautionary tale.

David Balaban, owner, Privacy-PC

SC Media Perspectives columns are written by a trusted community of SC Media cybersecurity subject matter experts. Each contribution has a goal of bringing a unique voice to important cybersecurity topics. Content strives to be of the highest quality, objective and non-commercial.