Black Hat: Phishing training is pretty pointless, researchers find

LAS VEGAS — Phishing training for employees as currently practiced is essentially useless, two researchers said at the Black Hat security conference on Wednesday.

In a scientific study involving thousands of test subjects, eight months and four different kinds of phishing training, the average improvement rate of falling for phishing scams was a whopping 1.7%.

"Is all of this focus on training worth the outcome?" asked researcher Ariana Mirian, a senior security researcher at Censys and recently a Ph.D. student at U.C. San Diego, where the study was conducted. "Training barely works."

At the beginning of Mirian's presentation, Mirian asked how many people in the audience of cybersecurity professionals believed that phishing training worked. About half raised their hands, to her mock dismay.

Lab tests vs. the real world

Her research partner Christian Dameff, co-director of the U.C. San Diego Center for Healthcare Cybersecurity, explained that he, Mirian and other research partners noticed that existing studies on the efficacy of phishing training were contradictory, with some showing remarkable improvement and others none at all.

Many of those studies were controlled lab tests, or as Dameff described them: "Pull in some grad students, test 'em, train 'em, test 'em again."

That didn't sound very conclusive, he explained, especially when cybersecurity insurance often requires phishing training and almost every large organization practices it.

Dameff and Mirian wanted scientifically rigorous, real-world results. (You can read their academic paper here.) They enrolled more than 19,000 employees of the UCSD Health system and randomly split them into five groups, each member of which would see something different when they failed a phishing test randomly sent once a month to their workplace email accounts.

Over the eight months of testing, however, there was little difference in improvement among the four groups that received different kinds of training. Those groups did improve a bit over the control group's performance — by the aforementioned 1.7%.

Not what was expected

However, there were some lessons learned — not all expected. The first was that it helped a lot to change up the phishing lures. Most subjects saw right through a phishing email that urged the recipients to change their Outlook account passwords, resulting in failure rates between 1% and 4%.

But about 30% of users clicked on a link promising information about a change in the organization's vacation policy. Almost as many fell for one about a change in workplace dress code.

"Whoever controls the lures controls the failure rates," said Mirian. "It's important to have different lures in your phishing training."

Another lesson was that given enough time, almost everyone falls for a phishing email. Over the eight months of the experiment, just over 50% failed at least once.

"Given enough time, most people get pwned," said Mirian. "We need to stop punishing people who fail phishing tests. You'd end up punishing half the company."

Finally, she said, it turns out that a lot of people completely ignore phishing training. Many of the test subjects closed the post-click page they were shown so quickly that the researchers couldn't measure their dwell time.

Maybe someday

Dameff explained that these results don't mean that phishing training will never work. There just needs to be more research into what does work.

"We need to empirically measure these outcomes and share the data to achieve better security," he said.

And, he said, you shouldn't blindly trust the claims made by vendors of phishing training.

"Ask them to back up their claims with data," Dameff said.

At the end of the presentation, Mirian asked how many people in the audience now thought that phishing training works.

"That's far fewer than half an hour ago," she said.