AI systems vulnerable to ‘command hijacking’ via visual prompts

Per The Register, researchers have demonstrated a new class of attack called CHAI (command hijacking against embodied AI) that exploits indirect prompt injection by manipulating visual information presented to AI systems. This vulnerability could allow malicious actors to hijack the decision-making processes of autonomous systems, including self-driving cars and drones.

Academics from the University of California, Santa Cruz, and Johns Hopkins University showed that AI systems, particularly large vision language models (LVLMs), can be tricked into following illicit instructions displayed on signs. These instructions, manipulated by AI for maximum efficacy in various languages and visual formats (fonts, colors, placement), can override intended programming. For instance, self-driving cars could be made to ignore pedestrians, and drones programmed to track police vehicles could be diverted. In simulated trials, CHAI achieved an 81.8% success rate with GPT-4o in tricking self-driving car models, and up to 95.5% success in misidentifying vehicles for drone tracking. Real-world tests with RC cars also showed high success rates, particularly with GPT-4o.

Further research is planned to test CHAI under adverse conditions like rain and visual noise, aiming to understand the full scope of these attacks and develop robust countermeasures against them.

Source: The Register