Statutory restrictions hindered federal response to SolarWinds, Microsoft Exchange

The SolarWinds and Microsoft Exchange incidents improved coordination between the government and private industry, but also exposed worrying gaps in the government’s information sharing, auditors concluded in a new Government Accountability Office report released Thursday.

Specifically, officials from two agencies (the Office of the Director of National Intelligence and the Cybersecurity and Infrastructure Security Agency) told auditors that information-sharing protocols in the wake of both incidents were “slow” and “a challenge,” largely due to statutory restrictions. Many exchanges with stakeholders of information around the vulnerabilities took place manually through email, instead of through dedicated or automated channels.

The review reinforces the significant damage that both incidents had on federal networks. Exploiting the Exchange vulnerabilities, GAO said, would have given Chinese hackers and other threat actors access to “email accounts and data, as well as [the ability to] install malware on systems and harvest user credentials, which could have been used to gain persistent unauthorized access to other networks at an impacted agency.”

On SolarWinds, the GAO has reiterated a point that others in government have made, chiefly that the government still can’t conclusively say federal networks that downloaded the corrupted Orion update are now safe or that the intruders have been kicked out.

“Even though CISA’s efforts to work with agencies have provided a degree of confidence that the threat actor is no longer present, the threat actor may have established undiscovered persistent access within affected agencies and private companies’ networks,” the report notes. “Failure to perform comprehensive and thorough remediation activity will expose those networks and potentially cloud environments to substantial risk for long-term undetected APT activity.”

They also found that logging practices at many federal agencies were woefully insufficient, with half of the 24 agencies evaluated saying gaps in network and log coverage prevented a quicker response to the flaws. The National Security Council concluded that the need to increase detection and response activities for significant cybersecurity incidents was one of the chief lessons of the two incidents, along with further building on past information sharing and public/private collaboration efforts.

“Agency officials also told us that the varying levels of data log preservation among agencies and a lack of data collection tools limited evidence collection for the incidents,” auditors noted.

One of the Biden administration’s mandates to federal agencies last year — a requirement to identify all devices connected to their networks and implement data logging — appears to be a direct response to this lack of visibility from federal agencies.

One federal official noted that “log retention was a particular challenge for investigators responding to the SolarWinds incident as the threat actor was in agencies’ networks months before it was detected and evidence may not have existed at all agencies based on an agency’s log preservation activities.”

The White House and the Office of Management and Budget have said that their efforts to improve logging capabilities over the next three years will not only help lay the groundwork for the widespread use of technologies like Endpoint Detection and Response and Security Orchestration and Automated Remediation systems, it will also help CISA as it looks to leverage new statutory authorities to conduct proactive threat hunting operations on other agency networks this year and beyond.

Last year, Jen Easterly told Congress that putting such systems in place around federal networks would “allow us to not just focus on the perimeter but really to focus in-depth, all the way down to the host level, at the workstation, at the server, to ensure that we can see what threats are out there, detect suspicious activity and ensure we’re able to mitigate and remediate it as soon as possible,” the CISA director said in a Senate hearing last week.

In the wake of the emergency order the agency issued in late 2020, at least six agencies reported that they were unable to generate enough telemetry to even identify or detect potentially anomalous behavior related to the flaw, while 11 agencies said they had networks where such activity “could” have occurred. Just nine agencies were able to confidently assert the presence or absence of such activity.

The report also underscores how statutory language can sometimes obscure the seriousness of an incident. To wit: 19 agencies did not report SolarWinds as a “major incident,” with some saying the incident did not qualify under the definition provided by OMB. Two agencies said they did not report it as such because they had determined that no systems or data were compromised, but 16 agencies provided no additional explanation.

Positive takeaways from cyber incidents

The incidents did accomplish one thing: improving the muscle memory of the federal government and private sector when it comes to coordinating in the wake of a newly discovered breach or vulnerability.

Multiple officials said the information sharing with industry allowed the government to quickly ascertain the scope of the SolarWinds campaign, increased visibility around servers vulnerable to the Microsoft Exchange bugs and patching gaps and allowed the government to build trust with stakeholders outside the government in ways that are repeatable for future incidents. In fact, CISA, FBI and other agencies are currently conducting many of the same incident and asset response activities regarding the Log4j vulnerabilities, which are broadly present in both public and private IT networks.

Meanwhile, those same agencies have completed all their mandated responses regarding the Microsoft Exchange vulnerabilities, with the FBI leading the government’s response in many areas, like investigating and gathering intelligence, disrupting the IT infrastructure of threat actors, sharing information with the private sector.

The findings are based on interviews officials from CISA, the Department of Justice and FBI, the Office of the Director of National Intelligence the NSA, and the White House National Security Council. Auditors also relied on reporting documentation from all 24 CFO Act agencies, descriptions and timelines from federal agencies and research private cybersecurity vendors around both incidents.