OWASP Global AppSec: Risk management may be a pointless waste of time

WASHINGTON, D.C. — Trying to measure and manage risk is a waste of time and effort, threat-modeling expert Adam Shostack said during his keynote address Friday (Nov. 7) here at the Open Worldwide Application Security Project (OWASP) Global AppSec conference.

"We gravitate toward the idea of risk. It's treated as if it's an unquestionable axiom," Shostack told the audience of cybersecurity professionals. "But it does not solve our problems. It leads to ugly fights inside of companies. It leads to problems we can avoid."

Shostack clarified that he isn't talking about risk in the general English-language sense of the term. Instead, he meant the narrower cybersecurity and business usage, in which "risk" means a quantified, quasi-mathematical factor that can be calculated by multiplying likelihood by impact.

Risk reduction, by extension, can supposedly be quantified by comparing the amounts of risk before and after a mitigation action.

Educated guessing

The problem, Shostack said, is that you can't properly measure the likelihood of an adverse event. You can't truly measure potential impact before it happens.

You can only guess, and you shouldn't have to make difficult business decisions and purchases — whether to buy that expensive exposure-management platform, how exactly to reconfigure your networks — by just guessing.

"Have you ever used risk to quantify a cybersecurity issue?" Shostack asked the audience. Lots of hands went up.

"Did you have the data you needed?" he asked. Far fewer hands were raised.

"Did that lead to a decision?" Shostack asked. A moderate number of hands went up, but fewer than in answer to the first question.

"We're putting risk into a privileged position, with the belief that risk works," Shostack said.

The notion of quantifiable risk is as pervasive as it is questionable, he said. The current NIST Cyber Security Framework declares in its first sentence that it "provides guidance to industry, government agencies, and other organizations to manage cybersecurity risks." The European Union's Cyber Resilience Act uses the word "risk" 178 times across its 112 pages.

"They want to deliver resilience through risk," Shostack said.

"Risk" is intertwined with "management," he pointed out, adding that regardless, even the people who wrote books on risk measurement knew they lacked definite numbers.

Yet American business practices are suffused with the notion of quantifiable, manageable risk. Financial investments, the insurance industry, and pension plans are all based on it. And probabilities, statistics, and actuarial data do tend to gravitate toward the mean, given a national population of hundreds of millions.

Moving beyond risk management

But it's impossible to quantify risk when you're dealing with incidents that may happen only a few times, or even only once.

"Applying risk is empirically hard," said Shostack. "It's hard to quantify likelihood and impact, hard to draw lines between vulnerabilities and the chance of a data breach."

Many organizations do just fine without using quantifiable risk, he added. Microsoft, where Shostack worked for eight years, instead uses "bug bars," which prioritize problems according to severity, not likelihood.

NIST's own Common Vulnerability Scoring System (CVSS) says right on its front page that "CVSS is not a measure of risk."

As one of the architects of threat modeling, Shostack prefers using that method instead of quantified risk. According to him, threat modeling involves answering four questions:

But notably, this does not ask what an attacker would do, or which assets the attacker would target. Focusing on those answers didn't work because they were unpredictable and invariably led to the problem of trying to quantify risk.

"Stop doing risk," Shostack urged. "Single incidents are impossible to predict. It's all just guessing."

Instead, he said, we should accept that there are some things we cannot accurately forecast. We should set norms, standardize answers and use other prioritization tools.

"We use risk because we believe it will help in prioritization, not because it actually does," Shostack said. "Real prioritization involves the cost of a fix, the impact of fixes, and the security load — none of which are risk."