Telling a design deficiency apart from an operating deficiency

Why the distinction determines the remediation

A vendor access review ran on schedule. The reviewer completed it, signed the attestation, and the evidence was filed. Ninety days later, the same over-privileged accounts were still active. The remediation assigned after the first review was retraining — the reviewer, it was argued, had failed to act on what they found. After the second occurrence, a harder look at the procedure revealed that it contained no step requiring the reviewer to submit a deprovisioning request. The review was designed to produce attestation. Deprovisioning was assumed.

That is a design deficiency. Retraining a person to perform a step that the procedure does not contain cannot fix the control. When a finding is misclassified as an operating failure and remediated with training or coaching, the procedure that caused the failure remains intact, the next review cycle runs the same way, and the finding recurs.

A design deficiency exists when a control, even if executed exactly as written, cannot achieve its stated objective. An operating deficiency exists when a properly designed control fails because the people or systems responsible for it did not execute it as intended. The remediation paths do not overlap. Design failures require changing the control — its scope, its procedure, its coverage, or its triggering conditions. Operating failures require changing how the control is performed — training, monitoring, accountability, or tooling that supports execution.

The PCAOB's AS 2201 draws the same structural line in the context of financial reporting controls: a deficiency exists when a control is either not designed to prevent or detect misstatement, or is designed adequately but fails in operation (Source: PCAOB AS 2201, https://pcaobus.org/oversight/standards/auditing-standards/details/AS2201). AS 2201 governs integrated audits of internal control over financial reporting, and that is the scope of what it establishes. We carry the distinction across to security and privacy control assurance by analogy, and the analogy holds for a specific reason: the design/operation split is a property of how any control can fail, not of financial reporting. A control that cannot meet its objective even when performed exactly as written has failed in design whatever framework it sits under. The definition is the PCAOB's; the extrapolation is ours.

Testing design before testing operation

You cannot test whether a control operates effectively if you have not first confirmed that it is designed to achieve its objective. Testing operation before confirming design produces evidence that may be formally clean and substantively useless.

NIST SP 800-53A Rev. 5 establishes examine, interview, and test as the three assessment methods for control effectiveness, and applies depth and coverage as attributes to each (Source: NIST SP 800-53A Rev. 5, https://csrc.nist.gov/publications/detail/sp/800-53a/rev-5/final). Design adequacy sits in the examine layer — reviewing the procedure, the policy, the configuration standard, or the system setting against the objective the control is meant to meet. If that review does not happen before operational testing, a passing operational test on a badly designed control produces a false assurance signal.

A practical sequence: confirm the objective the control is supposed to meet, review the procedure or configuration against that objective, identify whether a correctly executed instance of the control could satisfy the objective, and only then test whether execution meets the procedure. Skipping to execution testing in environments that have not done design review creates a gap when findings are later appealed — the evidence of design adequacy does not exist.

The control performed correctly that still fails

This is where the classification argument carries its real weight. A control can be performed exactly as documented and still fail to meet its objective if the design is inadequate. The people responsible for the control did nothing wrong. The procedure was followed. The evidence of execution exists. And the risk the control was meant to address remained unmitigated.

Consider a change-approval control designed to prevent unauthorized changes to production systems. If the scope of the control covers scheduled changes but the procedure does not address emergency or out-of-band changes, then emergency changes proceed without approval — not because anyone bypassed the control, but because the control was never designed to cover that population. Auditors testing only scheduled changes against the procedure will find no exceptions. The gap is invisible to operational testing.

The COSO Internal Control—Integrated Framework holds that internal control is effective only when each relevant principle is present and functioning, and all five components operate together as an integrated system (Source: COSO Internal Control—Integrated Framework, https://www.coso.org/guidance-on-ic). A control that covers part of the population it is meant to govern is not present in the relevant sense — it is partially absent by design.

The diagnostic question is: could a fully compliant execution of this control, by a competent and motivated person, have prevented the failure? If the honest answer is no, the deficiency is in the design, regardless of what the execution evidence shows.

When the procedure is silent on the failed step

Procedure silence is one of the clearest indicators of design deficiency. If the procedure does not contain the step that failed, the failure cannot be attributed to the person who skipped it.

The approval step with no defined approver is a common example. An access request workflow requires manager approval, but the procedure does not specify who approves requests when the manager is the requestor. Requests from managers either stall or are approved by the requestor — both failure modes are predictable, and neither is an operating failure. The design left a population uncontrolled.

When classifying a finding where the failed step is absent from the procedure, document the absence explicitly. A screenshot of the procedure with the relevant step missing, paired with the evidence of execution, supports a design classification and prevents downstream reclassification by a reviewer who was not present at the assessment. GRC-AUDIT-003 addresses the risks of reconstructed evidence and owner-controlled evidence without independent monitoring — the same documentation discipline applies here.

Repeated operating failures as a design signal

A single operating failure is usually an execution problem. A pattern of operating failures on the same control step, across different performers and different periods, is often a design signal — the procedure may be ambiguous, the required input may be unavailable, or the execution burden may exceed the capacity built into the design.

Tracking failure recurrence by step, not just by control, surfaces this pattern. If the same step fails across multiple test cycles, the question to ask is whether the procedure gives the performer what they need to execute it reliably. If it does not — if the step depends on a report that is not always produced, or on a system that does not retain the necessary data — the control has a dependency on an uncontrolled input, and the design requires remediation.

GRC-FOUND-003 covers the failure pattern of screenshot assurance and framework coverage theater. Repeated operating failures that are closed with evidence of one compliant execution without investigating the pattern can drift into the same territory.

Misclassification and what it costs

Misclassifying a design deficiency as an operating deficiency produces a remediation plan that cannot succeed. Training, coaching, and monitoring address execution. They do not add steps to procedures, expand control scope, define approvers, or secure uncontrolled inputs.

The audit cost is concrete: a finding closed with the wrong remediation type remains open as a risk even after the auditor accepts the closure. If an external auditor or regulator subsequently tests the same control and finds the same failure, the prior closure becomes evidence that the remediation process is ineffective — a secondary finding that is often more difficult to address than the original.

Under PCAOB AS 2201, a combination of deficiencies can constitute a material weakness even if no single deficiency does, when together they create a reasonable possibility that a material misstatement will not be prevented or detected on a timely basis (Source: PCAOB AS 2201, https://pcaobus.org/oversight/standards/auditing-standards/details/AS2201). Misclassification can suppress aggregation that would otherwise trigger escalation to that severity level.

What the distinction does not resolve

Classifying a deficiency correctly does not complete the remediation — it starts it. Design remediation requires someone with authority over the control procedure to change it, test the revised design, and produce evidence that the change was made and validated. Operating remediation requires evidence that execution has improved and that the improvement is sustained across more than one cycle.

Neither classification resolves questions about compensating controls in an assurance context — that territory is currently unowned in this corpus and is held for a future asset. Do not substitute design or operating classification for compensating-control analysis when the question is whether an alternative control meets the same objective.

The distinction also does not settle severity. Design deficiencies are not automatically more severe than operating deficiencies. A control that is well-designed but chronically not performed is a serious operating failure. Severity depends on the risk the control was meant to address, the population left unprotected, and the detection capability that remains. GRC-FOUND-001 owns the conceptual framework for how those factors interact.

The classification test

Before closing any deficiency finding, apply this sequence:

Documenting this sequence in the finding record creates an audit trail that supports the classification if it is later challenged. The documentation does not need to be elaborate — a brief statement of the objective, a reference to the procedure reviewed, and the answer to step 2 is often sufficient. What matters is that the logic is visible and attached to the finding before the finding is closed.

Deficiency classification table

Sources