AI-assisted SOC analysts up to 61% faster, 29% more accurate, study finds

A study on the use of AI to assist security operations center (SOC) investigations found that using AI improved speed and accuracy compared with manual methods.

The study by Dropzone AI and the Cloud Security Alliance, published Tuesday, involved 148 security operations center (SOC) analysts who were randomly assigned to complete escalated security alert scenarios either with or without AI assistance.

Each group had a similar distribution of analyst experience levels and tackled the same simulated scenarios in the same order: an Amazon S3 bucket alert and a Microsoft Azure failed login alert.

The AI group used the Dropzone AI platform in their investigations while the control group used traditional manual methods. After completing each scenario, the participants were scored for accuracy, completeness, report length and investigation time and asked for their perspectives on the difficulty of the scenarios, their confidence in their performance and their attitudes toward AI in security.

Accuracy was measured based on whether participants correctly determined if further action was needed in the scenarios, with the AI assisted group being 22% to 29% more accurate overall.  

The results showed that participants who used AI had a 97% accuracy rate for the AWS S3 bucket scenario, which involved a successful malware upload, and an 85% accuracy rate for the Microsoft Entra scenario, which was a brute-force attempt with no successful compromise. By comparison, the manual control group had 68% and 65% accuracy rates, respectively.

AI assistance was also found to significantly lower the time taken to investigate each scenario, with an overall 45% to 61% speed increase. The Amazon S3 scenario took AI-assisted users 58 minutes to complete on average, compared with 105 minutes for the control group, while the Microsoft Entra scenario only took 30 minutes for the AI group and 78 minutes for the control group.

Completeness, which was measured based on the extent to which participants addressed seven core investigative criteria expected for a thorough investigation, was nearly even between the AI and manual groups for the first AWS S3 scenario, but then reduced by 16% for the AI group and 29% for the control group in the second scenario.

The report authors stated that the AI-assisted group’s lower decline in completeness indicated a potentially higher resilience to fatigue between scenarios. They drew a similar conclusion regarding average report length, which reduced nearly 25% between scenarios for the control group (163 words to 123 words) but stayed almost the same between AI-assisted scenarios.

The AI-assisted and non-AI-assisted groups did not show significant differences in their perception of scenario difficulty or in their confidence in their investigative findings, and both groups had a mostly positive view of the use of AI in cybersecurity (8.6 out of 10 for the AI group and 8 for the control group).

The AI group was also asked whether their perception of AI in security changed after completing the scenarios, and 94% said they viewed AI more positively following the exercises.

“The findings underscore the benefit these tools can deliver in high-volume SOC environments, where consistency and efficiency are critical to success,” Cloud Security Alliance Associate Vice President Hillary Baron said in a statement.