How to Evaluate AI Claims in Identity Platforms

The Credibility Problem

Identity platform vendors have converged on a common vocabulary: risk scoring, behavioral analytics, anomaly detection, automated remediation, predictive access. The marketing language is nearly uniform. The implemented capabilities are not.

Machine learning-based risk scoring can adapt to individual user behavioral baselines over time, adjusting thresholds as patterns shift. A rule engine with a fixed threshold on consecutive failed logins does not do this — it produces a binary trigger based on a number an engineer chose during configuration. Both can appear in a product brief under the same heading.

This gap matters operationally because the failure modes differ entirely. A genuine adaptive model degrades gracefully when edge cases appear; it may produce lower-confidence scores that surface for human review.

A static rule engine either fires or it doesn't — and when it doesn't fire on a credential-stuffing campaign that stays below the threshold, there is no fallback signal. The consequence isn't an abstract capability shortfall. It's a missed detection with no alert.

Practitioners evaluating identity platforms cannot distinguish these implementations from a vendor demo. A structured evaluation approach changes that.

What This Decision Actually Involves

The inflation patterns follow recognizable forms. Knowing them lets you design POC tests that expose the gap before purchase.

"AI-powered risk scoring" as fixed threshold logic. Some platforms calculate a risk score by summing weighted rule conditions — failed login count, IP reputation lookup, time-of-day deviation from a static population average — and label the result a risk score. The score changes only when an administrator adjusts the rule weights. There is no model, no training data, no feedback loop. Ask the vendor to show you the score recalculate in real time after you demonstrate a new behavioral pattern for a test account. If the score doesn't shift within the session, you're looking at a rule engine.

"Behavioral analytics" as time-of-day rules. Behavioral analytics, properly implemented, requires a per-user baseline built from historical activity. Time-of-day anomaly detection that compares a login against a population-level schedule — "logins outside 8 AM–6 PM are anomalous" — is an engineer-authored rule, not behavioral analysis. The test: provision a test account with a clearly irregular but consistent usage pattern (logins at 2 AM daily for two weeks). A behavioral system should stop flagging those logins as anomalous once the pattern is established. A static rule engine flags them indefinitely.

"Automated remediation" as ticket creation. Automated remediation in identity context means the system takes a direct action on the identity object — suspending a session, stepping up authentication, revoking a token — without human initiation. Creating a ticket in a SIEM or sending an alert to a SOC queue is not remediation; it is notification. Verify by simulating a high-confidence anomaly during POC and confirming what action the platform takes autonomously, with what latency, and under what conditions it defers to human review.

"Predictive access" as scheduled access review. Predictive access controls suggest entitlements based on peer-group analysis or role inference from behavioral signals. Scheduled access review workflows that surface unused permissions after 90 days are governance hygiene — valuable, but not predictive. Ask whether the recommendation engine can explain why a specific entitlement was suggested or flagged, and whether the model updates recommendations as peer-group composition changes.

Evaluation Criteria

Frame each criterion as something verifiable in a proof of concept, not a feature checkbox.

Individual versus population-level baselines. Require the vendor to demonstrate that the risk model builds a per-user behavioral profile, not a population threshold. The test is straightforward: create two test accounts with opposite usage patterns, confirm the risk scores diverge over time without manual configuration.

Explainability of flagged sessions. A model that produces a score without a reason code forces your SOC to treat every flagged session as equivalent — they cannot triage by signal quality. Require the platform to produce a human-readable explanation for each flagged event during POC. If the vendor cannot demonstrate this, the operational overhead of false positives will fall entirely on analyst time.

Production metrics, not demo metrics. The NIST AI Risk Management Framework explicitly identifies trustworthiness characteristics for AI systems including validity, reliability, and the importance of evaluating AI performance against real-world operational conditions rather than controlled demonstrations (Source: NIST AI Risk Management Framework AI RMF 1.0, https://www.nist.gov/itl/ai-risk-management-framework). Ask vendors for precision and recall data from production customer deployments. Vendors with genuine ML deployments can generally produce these numbers. Vendors with rule engines do not have a concept of precision and recall because there is no probabilistic model to measure.

Fallback behavior. Require documentation of platform behavior when the AI component is unavailable — model inference latency spike, API timeout, or upstream dependency failure. A well-designed platform degrades to a defined policy state. A platform where AI is load-bearing infrastructure may fail open or block all authentication. Neither outcome is acceptable without knowing it in advance.

Core functionality without AI. Disable the AI components in a POC environment and verify that authentication, provisioning, MFA enforcement, and SSO operate without degradation. If disabling the AI module breaks protocol-layer identity functions, the architecture has no clean separation between ML inference and identity infrastructure. That is an operational risk, not a feature distinction.

Questions to Ask Vendors

These questions are designed to surface implementation depth rather than sales positioning.

On model architecture: Ask whether the risk scoring component uses a trained statistical model or a rule engine with weighted conditions. Ask when the model was last retrained, on what data, and how training data is isolated from customer environments. If the vendor cannot answer these questions technically, escalate to their engineering team. Marketing cannot answer them.

On false positive rates: Ask for false positive rates from three production deployments with different user population sizes. A vendor with real ML deployments can pull this from operational telemetry. Expect variance across customer environments — if the numbers are identical across customers, they may be illustrative rather than measured.

On availability fallback: Ask what the platform does when the risk scoring service is unavailable for 30 seconds, 5 minutes, and 30 minutes. The answer should be a documented policy, not a shrug. Options include fail-open (permit with logging), fail-closed (block until service recovers), or step-up (require MFA for all sessions during the outage window). Each has a consequence. Failing open creates a detection gap; failing closed creates an availability incident.

On AI component isolation: Ask whether AI features are modular and independently disableable. Ask whether doing so requires a platform redeployment or a configuration toggle.

NIST SP 800-63 Digital Identity Guidelines define requirements for authentication assurance levels based on the strength and reliability of the authentication mechanisms in use. Ask vendors whether their AI risk scoring influences assurance level assignment at runtime, and if so, what the documented fallback assurance level is when the model is unavailable.

Consolidation and Integration Considerations

Identity platforms that market AI capabilities often position those capabilities as reasons to consolidate identity functions — governance, access management, and analytics — onto a single platform. The consolidation argument can be valid, but the AI claim does not validate it.

Consolidation reduces integration surface and operational overhead when the platform's core identity functions meet requirements. Adding AI-adjacent claims to the evaluation does not change whether the platform handles joiner-mover-leaver lifecycle correctly, integrates with your SCIM endpoints, or enforces step-up authentication at policy thresholds.

Separate the AI evaluation from the platform evaluation. Score the platform on identity functionality first. Then evaluate AI capabilities as additive features with their own verification criteria. Conflating the two creates a risk that a vendor with compelling AI marketing displaces a platform that handles identity protocols reliably.

Integration architecture for AI-based risk signals varies by platform. Some platforms produce a risk score that is consumed internally to influence authentication decisions. Others expose the score via API so downstream systems — SIEM, SOAR, access management policy engines — can act on it. If your architecture depends on consuming risk signals externally, verify the API contract, the latency characteristics, and whether the signal schema changes between model versions without versioning guarantees.

Evaluation Checklist

AI Claim Verification by Category

AI Capability Verification Matrix

Tier 1: Must-Verify (Claims central to the purchase decision)

Tier 2: Important (Claims that affect operational value)

Tier 3: Nice-to-Have (Forward-looking roadmap items)

Roadmap items in Tier 3 should not influence the current purchase decision unless the vendor contractually commits to capability delivery with defined timelines and remediation terms for non-delivery. Marketing slide commitments do not constitute a contract.

Sources