A practical guide to measurable phishing simulation testing

For as long as the field has been in existence, cybersecurity pros have looked for ways to remove the human element from the security equation — to create a tool, architecture, or solution capable of warding off any and every phishing email before it ever reaches the end-user’s inbox.

And while we’ve most definitely made major strides towards that goal, no email security product can claim a 100% success rate. Even the most advanced, AI-driven enterprise email security products can only boast a 99.998% success rate in preventing phishing emails from reaching inboxes.

While that’s obviously a phenomenal track record, it’s important to keep two points in mind: phishing is a numbers game; and it takes only one successful attack to cause serious damage to an organization.

The average enterprise today gets targeted by hundreds of phishing emails per day, averaging hundreds of thousands of attempts per year. So, even at a high success rate an organization at the low end of the spectrum will see more than 200 phishing emails make it through their defenses. That’s 200 instances a year in which the only thing standing between the organization and a potentially devastating security breach is a single employee and his or her ability to identify a malicious email. And with the rise of generative AI, phishing attacks are quickly becoming both more sophisticated and more numerous.

With all this in mind, it’s hard to question the utility of security awareness training. It’s not merely useful, it’s absolutely essential. Phishing training for employees has become instrumental in creating a security-conscious workforce, reducing the risk of successful phishing attacks, and fostering a resilient organizational culture that can effectively respond to evolving cybersecurity threats.

Pros and cons of phishing simulation programs

While there are certainly some cons worth noting, it’s important to remember that they are far outweighed by the benefits the programs offer. What’s more, we can overcome many of these cons through policy changes and programs designed with them in mind. Nevertheless, let’s start with the potential drawbacks:

Cons:

False sense of security: If not handled carefully, simulated phishing tests can create a false sense of security among the workforce. Employees may become overconfident in their ability to recognize phishing attempts, leading to complacency. However, it’s almost exclusively the case when the training and testing program is either too easy, too repetitive, and/or too outdated. To overcome this, ensure that the testing and training program meets the degree of expertise of the workforce, contains significant variations, and is frequently updated to reflect the latest threats.
Phishing fatigue: Overuse of simulated phishing tests may lead to "phishing fatigue," where employees become desensitized to the exercises, reducing their effectiveness over time, and even leading some to mistakenly assume a real attack is just another simulation, and possibly failing to report the attack as a result. Companies can overcome this by making sure to use training and testing at a rate that ensures effectiveness without becoming overbearing. Avoid fatigue by ensuring variety and avoid exposing employees to the same simulations and materials over and over again.
Potential for backlash: If not communicated properly, simulated phishing tests can lead to resentment among employees who feel they are being targeted or tested without adequate context, information and training. This can also stoke feelings of anxiety in some employees about failing tests. The best way to avoid this: ensure the workforce gets properly trained and prepared for the testing beforehand. Be mindful of their level of expertise and select training and testing that offers a challenge without being unfair or beyond their level of expertise.

Pros:

Awareness improvement: This one goes without saying and it’s very much the bottom line when it comes to phishing training’s advantages: it makes employees more effective allies in the fight against cyberattacks. Simulated phishing tests raise awareness among employees about the existence and potential risks of phishing attacks. With a more vigilant and knowledgeable workforce, organizations are much less likely to fall victim to phishing attacks.
Behavioral change: By experiencing simulated phishing tests, employees are more likely to change their online behavior and become more cautious when interacting with emails, links, and attachments. It’s important to point out that these improved behaviors often carry over beyond the workplace and into the home, meaning fewer compromises to personal devices and a more secure workforce.
Realistic training: We can design simulations to closely mimic real-world phishing scenarios, offering employees practical, hands-on experience in a controlled environment. Moreover, simulation training lets employees be exposed first-hand to the newest, most advanced phishing techniques across the threat landscape. As opposed to watching countless training videos (which are more often than not outdated before their second or third viewing), simulation training helps to ensure the team better understands the latest threats out there.
Quantifiable metrics and targeted improvements: Organizations can track metrics such as click rates, open rates, report rates, and response times, creating quantitative data to measure not only the performance of their employees, but also the effectiveness of the training program itself. With this data, organizations can more readily identify areas that need improvement and opportunities to improve their team’s performance.

The proper cadence

There’s no magic number to this algorithm, but the more phishing tests run, the more employees can recognize the signs and patterns to look out for on these attacks. Companies should conduct phishing tests every other week or once a month to give accurate training and for businesses to receive accurate data on who is more susceptible to these attacks.

For the best efficacy, we recommend a combined security awareness training approach coupled with phishing simulation testing. For example, once someone fails a Phishing Simulation Training (PST) the company can send them an additional video that offers more training on how to spot these attacks. PST can help train employees on the latest attacks, and they will learn how to effectively spot and report phishing. So, while training videos might help as supplemental tools for training, never treat them as a replacement for hands-on phishing simulations.

Set measurable KPIs and live by them

When it comes to assessing the effectiveness of any security training and testing program, it’s important to identify the most important key performance indicators (KPIs). Without them, it’s impossible to gauge how well the team performs, nor can the company gauge the effectiveness of the training program.

Most people think they need to get the team’s click rate as close to zero as possible. At first blush, this makes sense — after all, the fewer people who click, the fewer have fallen for the simulation. However, we feel the goal should not be to only minimize click rates, but also to maximize reporting rates. Why? Because it’s through reporting that we can help our systems perform better and help increase the awareness level of our employees.

Also, for those organizations looking to make their training more enjoyable for their workers, there are many easy ways to introduce gamification to a phishing training program. For example, establish leaderboards (by individual employees and/or by teams) and offer prizes for the highest performers each quarter.

Security awareness training does work: it elevates an organization’s security posture, strengthens defenses, and lets them stay one step ahead. However, like any other companywide initiative, organizations have to approach them strategically, in a well-thought-out, and measurable way.

Eyal Benishti, chief executive officer, IRONSCALES