Cybercriminals are increasingly jailbreaking popular AI models like Grok from xAI and Mixtral from Mistral AI to create phishing tools, malicious code, and hacking tutorials, researchers at Cato Networks reported, according to The Record, a news site by cybersecurity firm Recorded Future.
These jailbroken versions are circulating on dark web forums like BreachForums under names such as WormGPT and FraudGPT, marketed for up to $5,000. Instead of exploiting software vulnerabilities, threat actors manipulate LLM behavior using specially crafted system prompts that steer responses past safety guardrails. Cato researchers emphasized that these misuses dont stem from flaws in the models themselves, but from how attackers reconfigure their context. The proliferation of open-source models makes it easy to host and distribute modified versions, making detection and takedown a challenge. Dave Tyson of Apollo Information Systems warned that there are likely hundreds of uncensored LLMs already in circulation, while Darktrace's Margaret Cunningham pointed to a rise in "jailbreak-as-a-service." The trend highlights growing concern over AI misuse, particularly as LLMs become more powerful and widely accessible.
These jailbroken versions are circulating on dark web forums like BreachForums under names such as WormGPT and FraudGPT, marketed for up to $5,000. Instead of exploiting software vulnerabilities, threat actors manipulate LLM behavior using specially crafted system prompts that steer responses past safety guardrails. Cato researchers emphasized that these misuses dont stem from flaws in the models themselves, but from how attackers reconfigure their context. The proliferation of open-source models makes it easy to host and distribute modified versions, making detection and takedown a challenge. Dave Tyson of Apollo Information Systems warned that there are likely hundreds of uncensored LLMs already in circulation, while Darktrace's Margaret Cunningham pointed to a rise in "jailbreak-as-a-service." The trend highlights growing concern over AI misuse, particularly as LLMs become more powerful and widely accessible.
