SecurityScorecard Inc. has launched an open benchmark designed to evaluate how artificial intelligence models perform in third-party risk management tasks while resisting adversarial manipulation, based on information published by Silicon Angle.
The new Third-Party Risk Management Benchmark assesses AI capabilities in areas like exposure triage and fourth-party technology risk. Unlike generic benchmarks, this tool uses real vendor telemetry and includes deceptive self-attestations to test if models can identify manipulated data. The framework assigns autonomy readiness levels from L0 to L4, indicating the degree of human oversight required. Additionally, a Model Security suite tests for vulnerabilities such as prompt injection and jailbreaks, assigning letter grades from A to F. Early testing revealed that while open-weight models are improving in supply chain security, they remain highly susceptible to adversarial attacks. Security experts emphasize that deploying untested AI agents creates significant new attack surfaces for enterprises. By open-sourcing this framework, the company provides a transparent method for security teams to evaluate model reliability. Organizations can now run their own assessments or submit results to a public leaderboard to compare performance across various frontier and open-weight models.
Source: Silicon Angle
