Large language models (LLMs) are revolutionizing the way we interact with digital systems — but as their use grows, so do the risks. Recent research revealed troubling security flaws in how LLM plugins are designed, including vulnerabilities that could enable
data leaks, remote code execution, and even full takeover of AI sessions.
One high-profile example comes from
HiddenLayer’s recent discovery of prompt injection vulnerabilities in
Google’s Gemini Advanced Workspace plugin. Researchers found that Gemini Pro and Gemini Ultra could be manipulated into leaking hidden system instructions and executing unauthorized actions — all triggered by carefully crafted user input or malicious content stored in shared documents.
The Gemini exploit
According to HiddenLayer, Gemini Pro could be tricked into revealing its hidden system prompt by rephrasing queries and formatting responses as code blocks. Despite instructions to conceal a “secret passphrase,” the model disclosed it under indirect questioning. Gemini Ultra, the premium version, went further — when paired with the Workspace plugin, it was made to ask users for their passwords after reading instructions embedded in a Google Drive file.
“This isn’t just a technical bug — it’s an architectural vulnerability,” said a HiddenLayer spokesperson. “Any time an LLM blindly trusts plugin input or lacks validation, it opens the door to abuse.”
[Editor's Note: This is part SC Media's partnership to unpack OWASP's Top 10 for LLM Applications.] The Gemini case is a textbook example of indirect
prompt injection: an attacker plants malicious instructions inside a document, then tricks the LLM into interpreting them as legitimate. The result? A hijacked session that could be used for phishing, data exfiltration, or impersonation.
Broader implications: A systemic design flaw
Security experts warn that Gemini is not alone. Most LLM plugins are REST APIs that accept freeform or poorly validated input. If a plugin blindly processes SQL queries, connection strings, or document parameters, attackers can exploit those features to escalate privileges or leak sensitive data.
For instance, a plugin that connects to a vector database might accept connection strings without validation — allowing an attacker to access and exfiltrate embeddings from other tenants. Another plugin that accepts SQL WHERE clauses as advanced filters might be vulnerable to injection attacks if it appends them directly to a query.
“LLMs interpret and act on input dynamically. If a plugin treats everything as user-generated, without confirming the source or intent, the entire system can be misled,” a researcher noted.
How to defend against insecure plugins
OWASP suggests applying longstanding software security best practices:
- Enforce strict input validation: Avoid freeform strings when possible. Use parameterized inputs with type and range checks.
- Add a validation layer: Where freeform input is required, implement secondary parsing and sanitization before execution.
- Follow OWASP ASVS guidelines: Apply access control and input validation standards consistently across plugin design.
- Use authorization tokens per plugin: Require OAuth2 or API keys that bind user identity to specific plugin actions.
- Limit plugin capabilities: Follow the principle of least privilege—expose only necessary functionality.
- Test thoroughly: Use static, dynamic, and interactive security testing during development.
HiddenLayer recommended keeping sensitive data out of system prompts entirely and said developers should fine-tune their models to specific tasks to minimize deviation. Google, for its part, said it regularly conducts red-teaming exercises and applies filters and input sanitization measures to detect and prevent malicious prompts.
Still, these efforts haven’t closed every gap — and as LLMs gain more control over business workflows, the risks will only grow. Gemini’s vulnerabilities may have been responsibly disclosed and patched, but they offer a stark reminder: the true threat may lie not in how powerful LLMs are, but in how little oversight exists over the tools that extend them.
This article is part of SC Media’s 10-part editorial series on the OWASP Top 10 for LLM Applications 2025. Produced in partnership with the OWASP Generative AI Security Project, the series highlights actionable steps for secure, transparent GenAI application development.
