AI has transformed intelligence collection by letting organizations gather, process, and analyze information at unprecedented speed and scale.
But while machines excel at identifying patterns and surfacing signals, intelligence remains fundamentally a human discipline. It’s still not possible to automate context, judgment, attribution, prioritization, and decision-making.
For better or worse, AI has had a massive impact on the way we can collect and analyze security intelligence. The most obvious change: the way threat intel collections are amassed; bigger collections with more detailed analysis.
More fundamentally, network defender operations are changing, allowing for a holistic picture of the threat landscape and possible countermeasures. It changes the role of the defender to one focused on making sense of data rather than harvesting it.
It also changes the way threat actors operate. Armed with AI intelligence-gathering tools, attackers can automate the attack process, gaining a massive increase in the number of attacks in the wild. Additionally, AI tools can improve social engineering as it allows for more believable phishing messages and conversations with targets.
There are several advantages to the use of AI for intelligence. Larger collections of data allow a better understanding of the threat landscape. More intel can reduce the chances of attackers taking advantage of blind spots. Improved pen tests can uncover vulnerabilities before threat actors do.
Then there’s the advantage of granularity. With larger and more detailed intelligence, security professionals can gain more precise insight into individual threats and incidents and tailor their defenses accordingly.
When deployed correctly, security professionals should also have much more time to perform all those tasks. With AI deployments able to automate everyday maintenance and operational tasks, data defenders are free to focus their efforts more on analyzing their gathered intelligence. For example, not having to spend as much time on understanding the what, they can spend more time working on understanding the why.
With every innovation comes a new set of challenges, and when it comes to intelligence the saying definitely holds true.
One of the biggest problems has become the gathering of "slop intel." With the larger net to gather data comes a larger pool of returns that are inaccurate, redundant, or just plain useless to analysts. AI has gathered information from sources known for disseminating state sponsored disinformation. Threat actors have also published “research” to influence both AI and search engines to return results favorable to them.
There’s also the risk of over-reliance on automation. Security professionals can make the mistake of thinking their AI deployments are "set it and forget it" solutions, rather than tools they have to use and maintain. Experts have admitted to me that, even though they understand the need to check and verify AI output, after running a few projects without noticing any major mistakes, they tended to spot check the results at best.
The third challenge is a bit broader: a lack of expertise. The introduction of AI intelligence also brings a need for new skillsets. Aside from knowing how to efficiently use the AI tools themselves, defenders will need to change their approach from focusing on technical know-how to developing their chops in analytics, knowing how to properly understand the scope and context of a specific threat or incident.
It's wrong for us to only focus on what intelligence analysts and data defenders can do to leverage AI. The bad guys use it as well.
First, there’s the problem that AI tools lower the barrier to entry. What once required an attacker to develop technical skills before carrying out attack campaigns they can now automate with AI services.
AI now finds vulnerabilities that had never been unearthed by people, sometimes finding vulnerabilities that had been around for decades. Unfortunately, AI has been much better at finding vulnerabilities than finding adequate patches.
Likewise, AI gives threat actors the ability to scan through terabits of data and find information that attackers can use for malicious purposes. Not only credentials, but intellectual property, information that can be used for physical attacks, data that can constitute insider information, and many other insights valuable to threat actors. In the past, many ransomware gangs were limited by the ability of their own workforce to analyze the data they stole for potential leads to other attacks. Now AI offers threat actors with everything from data analysis, to carrying out the attack, to helping with ransom negotiations.
The larger reach of automated attack techniques can also pose problems when it comes to effectively analyzing intelligence. Analysts could have a harder time picking out new threats from the sea of spammed out automated attacks.
Finally, there’s the social engineering risk. In addition to making phishing and social engineering attacks easier to set up, the use of sophisticated communication tools can make it easier for those without strong English skills to expand their attacks on Western targets. While most AI attacks are not new but rather improvements on existing attacks, the advent of convincing Deep Fakes that can now sometimes pass the Turing Test is a truly original new AI weapon.
While there are several challenges and risks that have been raised, there are some long-term changes that the community can make to address the challenges of AI.
For starters, intelligence analysts should focus on better quality of detection. This could effectively address the AI slop problem and help for more efficient and timely threat analysis.
Intelligence analysts can also reshape the way they handle collected intel. As a new generation of professionals enters the market, emphasizing training on analytics and behavioral science could become just as important as technical know-how.
Now that we see agents behaving in increasingly unpredictable and unexpected ways, such as by agents escaping sandboxes and creating communication channels and communities, the question of who’s ultimately responsible for agent behavior will loom large. Many of the models used by threat actors are the same models created by major companies and available to the public. The bad guys were just able to jailbreak them or to bully them into doing tasks outside their assumed or intended guardrails. Agents are becoming better at inferring what the client wants. If a user issues a request that doesn’t direct an agent to break the law, but creates a situation where the best or easiest way to accomplish a task is for the agent to break the law, and it does so, who’s responsible? Is it the user, the company that created the model? Those who trained the model?
Ultimately, it’s important to keep in mind that threat intelligence will always require a human being. AI can never fully replace human intelligence. Security intelligence still requires a human brain and intuition to properly understand context and reach conclusions.
Roman Y. Sannikov, global research coordinator, iCounter
