Break-Glass Access That Is Actually Usable Under Pressure

A Path That Only Exists On Paper

The on-call engineer gets the alert at 2:47 a.m. The control plane is unreachable. The IdP is returning 503s. She opens the break-glass runbook, navigates to the credential retrieval step, and finds it points to the secrets vault — the one that authenticates via the same IdP that is currently down. Break-glass paths that depend on the infrastructure they are designed to bypass offer no recovery value when that infrastructure fails.

This is not an edge case. It is the default failure mode for break-glass implementations that were designed during normal operations, tested never, and documented in good faith by people who could not reach the cloud console at 2:47 a.m. An untested break-glass path is an assumption with a runbook attached to it — and in an actual outage, assumptions do not open consoles.

The claim here is narrow: break-glass is a control only if it works when the normal path has failed. Everything that follows is about closing the gap between the documented path and the operable one.

The Circular Dependency Problem

Break-glass credentials are retrieved from somewhere. That somewhere typically uses authentication, network access, and authorization logic. When the failure that triggered the break-glass event also takes down any of those retrieval dependencies, the credential is effectively locked behind the incident it was meant to resolve.

The most common form: a secrets vault that enforces access via short-lived tokens issued by the primary IdP. If the IdP is unavailable, new tokens cannot be issued, and the vault denies the retrieval request — even to the break-glass account holder. Depending on vault configuration and token caching behavior, a previously issued token may still be valid within its TTL window; whether that window overlaps with the incident is not a control, it is luck. Vault configurations that allow a local recovery path — authenticated by a hardware-backed credential held offline and out-of-band from the IdP — reduce this dependency, though they introduce their own custody and rotation requirements.

A second circular dependency involves the approval channel. Some break-glass procedures require a second party to confirm the retrieval — via a chat platform, a ticketing system, or a dedicated approval queue. Chat platforms and ticketing systems are SaaS services with their own availability envelopes, and a significant cloud incident can affect multiple SaaS services simultaneously, particularly when they share common infrastructure providers or network dependencies. If the approval channel requires the same network path that is degraded, or if the approver's identity resolves through the same broken IdP, the check-and-release process stalls. When the designated approver cannot be reached through the primary out-of-band channel, the procedure should define an explicit escalation path — a named alternate approver with documented contact information and authority — rather than leaving the responder to improvise. That escalation chain must itself be reachable without the degraded infrastructure.

A related dependency that extends into the supporting controls: when an offline credential is retrieved using a hardware-backed method that bypasses normal telemetry, the retrieval event may not be captured by the same log pipeline that covers normal operations. This is not a reason to avoid offline retrieval — it is a reason to specify how that retrieval is recorded. A physical access log, a witnessed acknowledgment, or a secondary out-of-band log entry are all acceptable mechanisms; the requirement is that the record exists independently of the systems affected by the incident, and that it is reconciled against normal telemetry once those systems are restored.

Break Glass Dependency Table

Related coverage: IdP failover behavior, federation resilience, and fallback authentication architecture are covered in IAM-RESIL-003. CLOUD-IAM-002 covers privileged access control design, including standing-access reduction and entitlement risk scoring — do not conflate break-glass design with those controls.

What Break Glass Must Not Depend On

A useful constraint: list everything the break-glass path touches, then ask whether each item could be down for the same reason the incident occurred. If the answer is yes, that dependency is a single point of failure in the recovery path. Any dependency that can be removed by the same event that removes the primary path negates the independence that break-glass is meant to provide.

The specific items that frequently introduce this problem in cloud environments:

Break-glass accounts can often be configured to avoid the IdP dependency at the authentication step itself, but the implementation differs by provider. AWS and Azure support native local accounts — IAM users and directory-independent accounts respectively — that authenticate directly with the provider rather than through a federated IdP. In Google Cloud, the equivalent approach uses Cloud Identity or Google Workspace accounts that are exempt from the external SSO policy enforced on normal users; these are not "local accounts" in the same sense, but they serve the same function of bypassing the federated authentication path. In all cases, the tradeoff is the same: accounts outside the IdP's lifecycle management require independent rotation, monitoring, and access review.

NIST SP 800-53 Rev. 5 control AC-6(9) requires auditing the execution of privileged functions. (Source: NIST SP 800-53 Rev. 5, https://csrc.nist.gov/publications/detail/sp/800-53/rev-5/final) That logging requirement applies here: the break-glass account's use must be captured in a log store that is not itself dependent on the systems the break-glass event is affecting. AU-9(2) addresses protecting audit information by storing it on a separate system; AU-5(5) addresses alerting when audit logging capacity or availability is at risk. Both are relevant to designing a log architecture that survives the same failure the break-glass account is responding to.

Distinguishing Legitimate Use From Abuse

A break-glass path that is indistinguishable from an attacker's lateral movement is not a break-glass path — it is an unmonitored privilege escalation vector. The audit trail for break-glass use must capture who retrieved the credential, from which system, at what time, using what authentication method, and what actions were taken after access was granted. Assertion alone — "I used it for the incident" — is not a control.

The mandatory components of a usable audit trail:

The post-use review is not optional and should not be scheduled — it fires on use. A review that depends on the responder initiating it will frequently not happen, particularly after a stressful incident recovery. The automation should require the reviewer to attest that the use was authorized and the access was reverted, with a defined SLA — the appropriate window depends on organizational policy, risk tolerance, and any applicable regulatory obligations, not a fixed number.

Incident-time grants and their reversion are covered in CLOUD-ENT-OPS-006. The review process described here applies specifically to break-glass accounts, which have distinct characteristics: they are pre-staged, not dynamically granted, and their use signals that normal controls were bypassed.

Test question for your implementation: If an attacker obtained the break-glass credential and used it for 20 minutes, how long before your detection pipeline fires, and what would the alert contain? If the answer involves manual log review initiated by someone noticing something was off, the detection gap is real.

Testing Without Creating The Incident

A break-glass path that has never been exercised under realistic conditions has not been validated — it has been assumed. Documentation is not evidence of function.

Testing has two distinct phases that serve different purposes and require separate approval. The first validates the mechanical path; the second validates that the account actually works. Neither requires deliberately disabling the production IdP or making destructive changes to production systems.

Phase one: retrieval and process validation. This phase validates custody, retrieval mechanics, approval channel reachability, and log routing. It covers:

This phase stops before using the credential against a production system. Gaps found here are far cheaper to remediate than gaps found during an actual incident.

Phase two: functional validation. A credential can be retrievable and appear current while the account cannot authenticate or perform the required recovery actions. Phase one does not surface this failure. Phase two requires a separately approved, narrowly scoped exercise that verifies actual sign-in to the relevant console or API, confirms that the account holds the expected administrative authorization, validates that the session generates the expected alerts, and confirms that session termination and access reversion are logged correctly. Microsoft's emergency-access guidance explicitly includes testing sign-in and the ability to perform administrative tasks as part of account validation. AWS similarly recommends verifying the level of access obtained, not only that credentials can be retrieved. This exercise should be planned, time-bounded, and reviewed post-execution against the audit log record it produces.

Drill cadence follows the risk and criticality of the access path and the rate of change in the environment, not the calendar. For a high-criticality path a quarterly retrieval drill is a reasonable baseline; functional validation can follow a longer cadence but must occur on a defined schedule and after any significant change to the break-glass account, its permissions, or the underlying authentication infrastructure.

NIST SP 800-207's treatment of per-session, least-privilege access applies at the boundary where break-glass access ends. (Source: NIST SP 800-207, https://csrc.nist.gov/pubs/sp/800/207/final) The moment the incident is resolved, the break-glass session should close and access should revert. "Revert" here means restoring the approved emergency-access baseline — which retains the pre-staged account and its recovery privileges in a dormant state — rather than removing the account entirely. That reversion is not automatic in many environments — verify it is explicit and logged.

What Break Glass Cannot Compensate For

Break-glass access is a last resort for access path failure, not a substitute for IAM hygiene. It does not compensate for missing monitoring, over-permissioned standing accounts, or an absence of network segmentation. If the normal privileged access path is so fragile that break-glass fires frequently, the problem is in the primary design — see CLOUD-IAM-002 for privileged access control design and entitlement risk analysis.

Periodic access reviews of the break-glass account itself — not just auditing its use — are a necessary control. The review should confirm the account's permissions match the documented scope, the credential has been rotated within the defined period, and custodians are current. This can surface excessive permissions and stale entitlements. It does not establish whether a separate standing privileged account has been compromised before or during the incident. Detecting compromise of other privileged accounts is a function of monitoring, behavioral detection, and threat investigation — not break-glass access review. Keep those concerns separate.

The Break Glass Test

Run this in the next 24–72 hours. No privileged action required.

If any step fails, you have found the gap before the incident did. Remediate the specific failure — dependency, log gap, custodian confusion, stale credential, or unvalidated account function — and re-test that step before closing the finding.

Sources