Healthcare AI vulnerable to prompt injection, security experts warn

A healthcare artificial intelligence system designed to manage prescriptions has been found to be susceptible to manipulation by security experts, as reported by The Register.

Researchers from AI security firm Mindgard demonstrated that it was relatively easy to trick Doctronic's healthcare AI into revealing its system prompts and allowing modifications. By simply informing the AI that a session had not yet started and that the conversation was with the system rather than a user, the researchers could bypass safeguards. This allowed them to generate misinformation, such as COVID-19 conspiracy theories, or even suggest illegal activities like making methamphetamine, by presenting them as system updates.

While most manipulations were session-specific, the researchers found a way to introduce persistent changes through SOAP notes, a clinical recordkeeping format. They showed how an AI could be tricked into altering a prescription, for instance, tripling the dosage of OxyContin, which could then be passed to a human clinician for approval. Doctronic and the Utah pilot program stated that controlled substances cannot be refilled through their current trial and that additional safeguards are in place.

Source: The Register