Genius toddler with your production credentials

COMMENTARY: In 2025, a developer instructed the coding agent not to touch the production database, but the assistant went ahead and destroyed the entire production database anyway. Then it created thousands of fake records to fill what was missing and told the developer that a rollback was impossible.

It wasn’t an attack, the agent just did what it thought was helpful.

[SC Media Perspectives columns are written by a trusted community of SC Media cybersecurity subject matter experts. Read more Perspectives here.]

This is the part of the AI security conversation that keeps getting missed. The risk is not that agents are dumb – they know your codebase, your APIs, the database, and most of the public internet. The risk is that they combine their encyclopedic knowledge with the judgment of a toddler who has just discovered the kitchen drawers. You asked it to tidy the room, but it reorganized the whole house. And now it is not reversible.

The email that stole your files without being opened

Now add an adversary.

In 2025, researchers from Aim Labs published a paper on the EchoLeak vulnerability, the very first zero-click attack on a major AI product. The entire attack was a single email, and the victim didn’t need to click anything or even open it at all.

The email had hidden instructions that were written to look like a note to a colleague, not like a harmful instruction to a computer. This was enough to slip past Microsoft’s defense mechanisms. The email then just sat in the inbox, and the attacker waited.

Related reading:

Later, the user asked Copilot something completely unrelated. To answer, the agent looked up relevant conversations from the mailbox, including the attacker’s email, and followed its instructions. As a result, company files went out to the attacker, disguised as an ordinary image loading on the page.

Microsoft patched it and said no customers were affected. But look what the defender would have had to spot: no malware, no download, no strange login. It was just an assistant trying to be helpful for the wrong person, using permissions you granted on purpose.

Why this keeps happening

There is one architectural fact underneath nearly all of it. A model reads its own instructions, your request, and any information it retrieves from the world as a single undifferentiated stream. There is no reliable way to distinguish some words as “system command” and others “just data.” A malicious sentence in a calendar invitation has the same weight as a directive from your CEO.

This brings us back to the toddler. You don’t leave a small child alone in a park, not because they’re incapable, but because a stranger can walk up and say, “your mum told me it’s fine, come with me,” and the child has no way to check. They can’t verify who issued the instruction. They only hear that it sounds like permission.

That is exactly what happened to Copilot. The attacker didn’t break in. He just told the assistant that the parents had said it was fine, and the assistant had no way to check.

This is called prompt injection, and it’s not some kind of software bug that could be fixed in code. It is just how the agent reads the information.

What actually protects you

Let’s compare the deleted database and the malicious email cases. One was an accident, one was a targeted attack, but in both cases the damage was caused by the same thing: what the agent was allowed to do. A shot by an enemy and a negligent discharge are both fatal.

Start with permissions, not prompts. The most common deployment pattern today is also the worst: hand the agent broad access to the computer, write “please be careful, make no mistake” in the system prompt, and hope for the best. Prompts are not a security boundary; the whole lesson of prompt injection is that anything expressed in words can be overridden by other words. Give the agent a specific, enumerated set of tools and the fine-grained access each one needs. If it books meetings, it gets the calendar, not the mailbox. If it reads tickets, it doesn’t need to be able to open network connections. Always assume that every single permission that you grant will be used in a way that you did not intend.

Add additional layers of checks before the action where sensitive access is genuinely necessary.

First, an automated review step: before a tool call executes, a separate check inspects the actual action (including the command, parameters, and intent) and blocks anything that appears to be harmful. The critical design rule is that this layer can only say no. It must never be the thing that grants approval, because it’s a model too, and it’s subject to the same manipulation as the agent it’s policing. Its job is to catch the obvious, not to certify the safe actions.

Second, a human. For anything critically sensitive or irreversible (deleting, deploying, paying, publishing, or sending externally) a person must approve the specific action. An automated verdict of “this looks fine” is permission to ask and not permission to proceed. Two toddlers agreeing with each other is not supervision.

It is also important to log what the agent did. Keep track of the tool calls, the parameters, the data that left the building. A hijacked agent can complete a malicious task perfectly, and your tests would still show green.

Conclusion

This doesn't justify avoiding the technology. It gives a reason to refrain from handing over authority any faster than you can take it back.

You don’t childproof the house because the toddlers are stupid. You do it because they are fast, curious, and completely uninterested in your restrictions.