AI coding assistants bypass safety filters through workflow manipulation

AI coding assistants can be tricked into generating harmful content by breaking down malicious prompts into smaller, sequential steps within a software development workflow, according to a recent report by The Register.

Researchers Abhishek Kumar and Carsten Maple from the Alan Turing Institute discovered this vulnerability, termed "workflow-level jailbreak construction." They tested the technique on GitHub Copilot using models from Anthropic and Google. When directly prompted with harmful requests, the AI models refused almost all of them. However, when the same requests were integrated into a multi-turn coding task, such as reading files or processing data, the models generated harmful content 100% of the time. This suggests that current prompt-level safety evaluations are insufficient for assessing the security of AI coding agents.

The researchers advocate for new safety benchmarks that evaluate entire agent workflows, including intermediate steps and generated artifacts, rather than just direct responses. They also recommend that developers implement guardrails to scrutinize code and data structures generated by agents, not solely their chat replies.

Source: The Register