AI can help the industry finally get SOC automation right

COMMENTARY: Security operations centers (SOCs) are the nerve centers of cybersecurity. However, despite massive investment in tools and technologies, many SOCs still find themselves overwhelmed by the very chaos they aim to control.

Analysts are drowning in data, jumping between disconnected tools, and trying to make sense of endless alerts. The result? An epidemic of burnout among the talented security professionals who are critical to keeping organizations safe.

[SC Media Perspectives columns are written by a trusted community of SC Media cybersecurity subject matter experts. Read more Perspectives here.]

This has become particularly acute for state and local government security teams that must protect critical infrastructure and sensitive citizen data with typically smaller budgets and staff than their federal or private-sector counterparts.

Despite this challenge, today we're seeing states significantly increase cybersecurity investments, with initiatives like the proposed $88 million Cyber Command in Texas and New York's enhanced cybersecurity funding for its Joint Security Operations Center.

The root cause lies in a fundamental misconception about security operations. For decades, we've tried to impose rigid structure on inherently unstructured problems. Various products promised to bring order through centralization and automation. Instead, they often added layers of complexity, transforming threat hunting from finding a needle in a haystack to finding the right needle in a stack of needles.

Advances in Generative AI (Gen AI) and reinforcement learning are transforming how humans interact with data, offering capabilities that were previously out of reach. AI doesn't just tolerate chaos – it thrives in it. Unlike its predecessor technologies, AI can reason over unstructured data without requiring rigid frameworks or predetermined patterns. It adapts to disorder rather than trying to eliminate it. If done right, AI can finally deliver on the promise vendors have been making to analysts and CISOs for decades.

As a former Air Force officer, I often think of SOC workflows in terms of the OODA Loop – Observe, Orient, Decide, Act – a decision-making framework developed by Col. John Boyd in the 1950s. While this concept isn't new to cybersecurity, it perfectly illustrates how technology should enhance rather than hinder our analysts' natural problem-solving processes. The future of SOC automation lies not in forcing structure onto chaos, but in embracing tools that can navigate that chaos alongside the analyst and accelerate the OODA Loop.

Why SIEM, SOAR and XDR fell short

Before we head into the future, let's take a quick journey through the evolution of security operations automation. It's a story that unfolds in three major chapters over the past two decades: SIEM, SOAR, and XDR. In a recent episode of CSO Perspectives by N2K CyberWire, I had the opportunity to take a deep-dive into these previous attempts.

It began in the early 2000s with security information and event management systems (SIEMs), which replaced manual log review with centralized data aggregation and analysis. But as data volumes exploded, SIEMs generated overwhelming false positives and unsustainable storage costs. If we put 100 CISOs in a room today, we’d get nearly 100 complaints about the costs associated with SIEMs.

By the mid-2010s, security orchestration, automation, and response (SOAR) platforms emerged, attempting to capture human decision-making in predefined playbooks. However, their rigid "if-this-then-that" logic couldn't adapt to rapidly-evolving threats and enterprise environments. In mid-2024, Gartner declared the death of the SOAR and the dire need for something better.

Extended detection and response (XDR) followed, promising real-time integration across endpoints and networks. While it improved upon previous solutions, XDR still couldn't fully address analysts' core challenges. Similarly, threat intelligence platforms, meant to enhance decision-making with contextual data, often created more noise than clarity. Rather than reducing analyst workload, these tools contributed to growing burnout across SOC teams by adding complexity without producing concomitant value.

Embrace chaos with AI

These past failures have paved the way for AI – a fundamentally different approach that thrives on unstructured data. Unlike traditional automation, AI can reason over chaos in ways that feel almost human-like. Picture an AI system proactively suggesting detection rule updates, or guiding junior analysts through investigations with insights from world-class threat hunters.

For state and local governments facing threats to election systems, public utilities, and municipal services, this capability represents a potential game-changer—allowing resource-limited security teams to achieve enterprise-grade threat detection and response. Even the dreaded late-breaking Friday threat report could be swiftly analyzed by an AI-powered platform, freeing analysts to focus on what truly matters – like time with family – instead of tilting at windmills.

This isn't about replacing human reasoning, but amplifying it. Recent advances in Gen Ai and reinforcement learning have transformed how analysts interact with data. By wrapping AI technology around our analysts, we can radically accelerate the OODA Loop, enabling better, faster decisions with richer context.

After two decades of unfulfilled promises to solve security operations challenges, we finally have technology that can deliver. But the big lesson from past failures is clear: automation works only when it enhances human capabilities rather than attempting to replace them.

Success lies not in eliminating analysts from the equation, but in combining AI's ability to process vast amounts of unstructured data with human insight, intuition, and strategic thinking. We simply can't replace the secret ingredient of a highly efficient quality analyst. That will remain true for the foreseeable future. This human-technology symbiosis is the true future of security operations and how we’ll get automation right.

William MacMillan, chief product officer, Andesite

SC Media Perspectives columns are written by a trusted community of SC Media cybersecurity subject matter experts. Each contribution has a goal of bringing a unique voice to important cybersecurity topics. Content strives to be of the highest quality, objective and non-commercial.