AI-generated emails make up 40% of BEC lures, security firm says

AI-generated emails now account for an estimated 40% of business email compromise (BEC) attempts, VIPRE Security Group said in its Email Threat Trends Report for Q2 2024, published Wednesday.

The security firm says it ran a sample of BEC emails sent to its customers through multiple tools designed to detect AI-generated content, including GPTZero, Quillbot, ZeroGPT, Sapling and Scribbr, leading to the 40% figure.

BEC lures in total made up about 49% of 226.45 million spam emails detected by VIPRE in Q2 2024. SC Media reached out to VIPRE to ask how many of these emails were analyzed for AI-generated content and did not receive a response.

However, security experts who spoke with SC Media said there’s little doubt that generative AI use in phishing campaigns is a growing concern.

“The rise of AI-generated business email compromise (BEC) attacks, as reported by VIPRE Security Group, aligns with our observations of a significant increase in sophisticated phishing lures over the past year,” SlashNext Email Security Field CTO Stephen Kowski said in an email to SC Media.

SlashNext released two reports over the past year pointing to the role of GenAI apps like ChatGPT in making phishing emails easier to produce and more likely to evade detection.

The first, in October 2023, noted a 1,265% increase in phishing attacks between Q4 2022, when ChatGPT was first released, and Q1 2023. The second, published in May of this year, found a 856% surge in phishing attacks and 27% growth in BEC over the previous six months, which SlashNext researchers attributed to increasing adoption of GenAI.

GenAI has seemingly cemented itself as part of the cybercriminal’s toolkit, with Hoxhunt Co-founder and CTO Pyry Åvist telling SC Media that AI-based phishing kits are becoming a popular option for online scammers.

“We’ve seen a significant uptick in attacks that are likely from a new generation of blackhat AI phishing kits available on the dark web. The prices of these kits are dropping and they produce phishing emails with comparatively better localized text, graphics and landing pages than the older generation of phishing kits,” Åvist said.

What latest email threat trends mean for email defenses

In addition to insights regarding use of GenAI in email spam, VIPRE reported a 74% increase in malicious links sent via email year-over-year, possibly due to evolving link masking tactics used to bypass traditional email security scans.

VIPRE noted that these links, totaling 16.91 million, were caught by its Link Isolation tool, which tests links in a sandbox in order to catch attacks that may otherwise evade safety scans.

The rise of AI-generated BEC scams and other sophisticated phishing tactics calls for solutions that leverage AI to detect malicious content, links and attachments where less advanced solutions, and human users, may fail to pick up on the red flags.

“Organizations must adopt advanced threat detection solutions that leverage AI and machine learning to identify and block these increasingly convincing AI-generated scams in real-time. Additionally, implementing continuous security awareness training for employees and utilizing multi-factor authentication can significantly enhance an organization’s resilience against BEC attacks in the AI age,” said Kowski.

Darktrace Vice President of Strategic Cyber AI Nicole Carignan agreed that defenders will need to fight AI with AI, as cyberattackers’ use of the technology continues to expand from basic phishing emails to more specific spear-phishing lures and the use of GenAI to craft malware scripts.

“With the increasing use of generative AI by threat actors, our dependency on traditional threat intelligence or rules and signature-based defense systems will diminish as threat actors can now rapidly adopt and change signatures, hashes, and indicators of compromise to evade defenses,” Carignan said. “Organizations must use machine learning-powered tools that can understand how their employees interact with their inboxes and build a profile of what activity is normal for users, including their relationships, tone and sentiment, content, when and how they follow links, etc.”