Security engineer fools AI chatbots with fake world championship claim

AI chatbots that rely on web searches can be tricked into presenting fabricated information as fact, as demonstrated by a security engineer who convinced several bots he was the world champion of a non-existent German card game. The engineer created a fake Wikipedia entry and a corresponding website to support his claim, which was then readily accepted by multiple AI chatbots when queried. This highlights a vulnerability in how AI models process and trust information retrieved from the internet, according to The Register.

The engineer, Ron Stoner, fabricated a Wikipedia entry for the card game "6 Nimmt!" (also known as "Take 5"), listing himself as the 2025 world champion. He also created a website, 6nimmt.com, which hosted a press release announcing his fictional victory. Despite the lack of independent corroboration and the fact that both the Wikipedia entry and the website were created by him, several AI chatbots confidently stated he was the world champion when asked. Stoner explained that this issue, termed "retrieval-augmented generation" poisoning, occurs because AI models often prioritize whatever information ranks highest in search results, regardless of its veracity. He noted that this tactic is akin to old-school SEO and misinformation techniques, but the new interface of AI makes the false information appear authoritative.

Stoner identified three failure modes: the retrieval layer can immediately return bad data, model training corpora can be poisoned if the fake information is scraped before correction, and AI agents could be manipulated into taking harmful actions based on false retrieved data. He emphasized that the attack cost only $12 for a domain registration and minimal time, suggesting a significant attack surface for motivated adversaries.

Source: The Register