Mitigating attacks against AI-enabled Apps, Replacing the CIA triad, Enterprise News – David Brauchler – ESW #429
Segment 1: David Brauchler on AI attacks and stopping them
David Brauchler says AI red teaming has proven that eliminating prompt injection is a lost cause. And many developers inadvertently introduce serious threat vectors into their applications – risks they must later eliminate before they become ingrained across application stacks.
NCC Group’s AI security team has surveyed dozens of AI applications, exploited their most common risks, and discovered a set of practical architectural patterns and input validation strategies that completely mitigate natural language injection attacks. David's talk aimed at helping security pros and developers understand how to design/test complex agentic systems and how to model trust flows in agentic environments. He also provided information about what architectural decisions can mitigate prompt injection and other model manipulation risks, even when AI systems are exposed to untrusted sources of data.
More about David's Black Hat talk:
- Video of the talk and accompanying slides: https://www.nccgroup.com/research-blog/when-guardrails-arent-enough-reinventing-agentic-ai-security-with-architectural-controls/
- Talk abstract: https://www.blackhat.com/us-25/briefings/schedule/#when-guardrails-arent-enough-reinventing-agentic-ai-security-with-architectural-controls-46112
- Slide presentation only: https://i.blackhat.com/BH-USA-25/Presentations/USA-25-Brauchler-When-Guardrails-Arent-Enough.pdf
Additional blogs by David about AI security:
- Analyzing Secure AI Architectures: https://www.nccgroup.com/research-blog/analyzing-secure-ai-architectures/
- Analyzing Secure AI Design Principles: https://www.nccgroup.com/research-blog/analyzing-secure-ai-design-principles/
- Analyzing AI Application Threat Models: https://www.nccgroup.com/research-blog/analyzing-ai-application-threat-models/
- Building Security‑First AI Applications: A Best Practices Guide for CISOs: https://www.nccgroup.com/building-security-first-ai-applications-a-best-practices-guide-for-cisos/
- Building Trust by Design for Secure AI Applications: Tips for CISOs: https://www.nccgroup.com/building-trust-by-design-for-secure-ai-applications-tips-for-cisos/
- AI and Cyber Security: New Vulnerabilities CISOs Must Address: https://www.nccgroup.com/ai-and-cyber-security-new-vulnerabilities-cisos-must-address/
Segment 2: Should we replace the CIA triad?
An op-ed on CSO Online made us think - should we consider the CIA triad 'dead' and replace it? We discuss the value and longevity of security frameworks, as well as the author's proposed replacement.
Segment 3: The Weekly Enterprise News
Finally, in the enterprise security news,
- Slow week for funding, older companies raising via debt financing
- A useful AI framework from the Cloud Security Alliance
- two interesting essays, one of which is wrong
- Folks are out here blasting unencrypted data to and from Satellites, while anyone can sniff and capture it
- getting hacked during a job interview
- LLM poisoning is far easier than previously thought
- F5 got breached
- Be careful when patching your Jeep (’s software)
All that and more, on this episode of Enterprise Security Weekly.
David is an enthusiast for all kinds of technological development and enjoys learning new attacks and methodologies as much as he enjoys teaching and training others in those processes. He enjoys researching and using emerging technologies including AI and machine learning, blockchain/smart contracts, and novel developments in computing hardware. He was previously an adjunct professor for the Cyber Security graduate program at Southern Methodist University. David has a master’s degree in Security Engineering and the Offensive Security Certified Professional (OSCP) certification. David published Multi-Level Access Protection for Future IEEE P1687.1 IJTAG Networks in IEEE ITC 2020, which proposed a low-cost security protocol for the IEEE P1687.1 IJTAG standard. He was previously the host of NCC Group’s monthly Talking Cyber podcast series.
Don't miss InfoSec World 2025 — October 27 to 29 at Disney’s Coronado Springs Resort! Cybersecurity pros, workshops before and after, and endless networking. Save 25% with code ISW25-SW at securityweekly.com/ISW2025!
Adrian Sanabria
- FUNDING and M&A: via the Security, Funded newsletter, issue 215 – There’s No Governance in this Dojo
Last week's vibe check
Asks: "How would you describe your organization's approach to overseeing third-party AI vendors?"
The result is a nearly 3-way tie with:
- Comprehensive AI vendor risk assessment program
- Standard vendor process with AI considerations
- Same process as any other vendor (slightly less popular than the first two)
Funding
- Authentic8, a United States-based remote browser isolation platform, raised a $12.0M in Debt Financing from Vistara Growth.
- DigiCert, a United States-based digital certificate provider, raised a $9.3M in Debt Financing from Runway Growth Capital.
Acquisitions
- Inky, a United States-based email security platform, was acquired by Kaseya for an undisclosed amount. Inky had previously raised $31.8M in funding.
- FRAMEWORKS: Introducing MAESTRO: A framework for securing generative and agentic AI
I missed this framework when it first came out. Very useful to make sure you're not missing anything while going through a design/architecture review and lots of great examples!
- ESSAYS: Three Security Invariants Could Prevent 65% of Breaches: Analyzing 70 Incidents and Building CISO Challenge
I love everything about this post, except the conclusion. There are no three of anything that can stop 65% of breaches. Fixing a common misconfiguration or broken security control doesn't stop attackers, it simply forces them down a different path.
Sure, if they fail enough times down enough paths, they may give up and go after a different target, but as we've seen in many, many cases - if one of these "invariants" stops them, they just go around it. And that's assuming the average enterprise can pull off these three invariants, which are not simple controls to tackle:
- hardware-based MFA
- strong egress filtering
- application control on endpoints
But again, check out this writeup, he gets a lot right, and there's a very interesting bit discussing CISO incentives and how to fix them. He's also building a board game to simulate CISO dilemmas and incentives.
- ESSAYS: Autonomous AI hacking and the future of cybersecurity
An op-ed written by industry vets Bruce Schneier, Gadi Evron, and Heather Adkins, warning about generative AI tipping the scales entirely in the attacker's favor (and the scales were already in their favor). But in an unexpected "glass is half full" take, they also share how they think AI could benefit defenders as well, particularly in AppSec.
Phase One: AI accelerating vuln discovery and fixes Phase Two: 'VulnOps' - tooling designed to streamline AI-assisted vuln discovery and fixes Phase Three: Software dev disruption - all this shifts left - vuln discovery and repair is done earlier in the dev process Phase Four: The self-healing network - defenders don't need to wait on patches from vendors, they can fix it or mitigate it themselves.
- RESEARCH: Satellites Are Leaking the World’s Secrets: Calls, Texts, Military and Corporate Data
UC San Diego and University of Maryland researchers revealed that they were able to collect sensitive network data from satellites, using less than $1k of off-the-shelf equipment.
They found a variety of traffic, ranging from military comms to in-flight Internet and even internal corporate network communications. Note that, while they were able to capture the traffic, a lot of modern network traffic is encrypted (like nearly 100% of website communications). That in-flight Internet traffic was likely no more vulnerable than if you were able to capture traffic at a coffee shop.
That's why the researchers were shocked by how much of the rest of the communications they captured were not encrypted.
- Call and SMS data from T-Mobile - unencrypted
- internal corporate network traffic - they captured cleartext authentication data and cleartext emails
- comms from critical infrastructure
- military comms - unencrypted voip data and ship tracking data
- unencrypted voip calls from voip gateways
Classic "nobody will ever see this" security by obscurity thinking.
- SQUIRREL: Jeep Owners Are Reporting That An OTA Software Update Is Disabling Their Vehicles – The Autopian
What happened: Jeep owners received an OTA update to the UConnect infotainment system with an option to install or defer. Those that installed it reported being able to drive a short distance before their vehicles lost all drive. Here's how one owner described the situation:
“We just came to a screeching halt. Every light in the thing came on… service stability control… now we’re stuck. I can’t even come out of Drive now.”
For some owners, drive failed on the HIGHWAY.
The fix is almost more concerning - apparently it installs with NO visible prompt or indication. Owners report that when they turned on their Jeeps for 10 minutes or so, a fix is installed.
JUST INSTALL THE PATCH THEY SAID
Sean Metcalf
- RESEARCH: A small number of samples can poison LLMs of any size
From Adrian: also check out the thoughts on this video, particularly his take on the future of SEO - https://www.youtube.com/watch?v=o2s8I6yBrxE
- BREACHES: F5 says hackers stole undisclosed BIG-IP flaws, source code
China allegedly hacked F5 and had long term access to its network and made off with source code. They also got undisclosed info about vulns and details about customer implementations.
How big a deal is this?
- CISA is giving civilian agencies just one week to patch
- F5’s BigIP traffic management products are used by 80% of the F500
