A deviation surfaces in a quarterly access review and the control owner disputes it. The discussion that follows isn't about whether the deviation indicates a real failure — it's about whether ten samples from a population of 400 can support any conclusion at all. You've lost the argument before you've made it. The sample size was never derived; it was guessed.
What a sample is standing in for
A sample is a representation of control performance across a population you haven't fully examined. It substitutes for 100% testing when exhaustive review is impractical, which is the case for most operationally active controls. What the sample is standing in for determines how large it needs to be, how it must be selected, and what conclusions it can support.
The distinction between sufficiency and appropriateness matters here. PCAOB AS 1105 defines sufficiency as a measure of the quantity of evidence and appropriateness as a measure of quality — encompassing both relevance and reliability (Source: PCAOB AS 1105, https://pcaobus.org/oversight/standards/auditing-standards/details/AS1105). A larger sample doesn't compensate for evidence that doesn't speak to the control objective. Pulling more screenshots of a misconfigured process gets you more of the wrong thing. Before sizing anything, confirm the evidence type is appropriate to the assertion.
Define the sampling unit before sizing the sample
Before deriving a population from control frequency, define what you are actually testing. The sampling unit is the discrete item that corresponds to the assertion under evaluation, and it is not always self-evident from how a control is described.
A quarterly access review illustrates the problem. The control fires four times per year, which might suggest a population of four. But the assertion being tested could be any of the following: that each review cycle was completed, that individual access decisions within each cycle were documented and acted upon, or that revocations resulting from the review were executed in a defined timeframe. Each assertion implies a different sampling unit and a different population. If the review covers 300 accounts across three systems, the population for testing revocation completeness is not four — it may be in the hundreds, depending on how many accounts were flagged during the period.
Automated controls introduce a different complication. A continuous encryption control operating at the infrastructure layer does not necessarily generate a sampling problem involving millions of log events. PCAOB AS 2201, paragraphs B28–B33, recognizes that automated controls with consistent execution logic may be assessed differently from manual controls — focusing on configuration, relevant system changes during the period, exception handling, and the IT general controls that protect the automated process. Sampling from a log population may not be the right approach at all; the relevant evidence may be a combination of configuration screenshots, change records, and exception reports. Separate the question of what needs to be tested from the question of how many items to pull.
Defining the sampling unit before touching a population export prevents two recurring problems: deriving a population size that doesn't match the assertion, and selecting sample sizes that satisfy coverage requirements for the wrong thing.
The inputs that determine size
Sample size is an output, not a starting point. Three variables drive it: how much deviation you can tolerate before concluding the control is failing, how much deviation you expect to find based on prior results and control design, and how willing you are to reach the wrong conclusion about control effectiveness.
PCAOB AS 2315 formalizes exactly these three inputs — tolerable rate of deviation, expected rate of deviation, and the allowable risk of assessing control risk too low — as the determinants of sample size for tests of controls (Source: PCAOB AS 2315, https://pcaobus.org/oversight/standards/auditing-standards/details/AS2315). AS 2315 sets those factors for financial-statement audits of controls. They transfer usefully to security control testing — not because the PCAOB says they do, but because each one names a judgment a GRC manager is already making implicitly when deciding whether a control's performance is acceptable. Borrowing the vocabulary makes that judgment explicit and reviewable.
Lower tolerable deviation → larger sample required. Higher expected deviation → larger sample required (because you need statistical confidence where error is likely). Lower acceptable risk of a wrong conclusion → larger sample required. These relationships are directional and stable.
One clarification worth making explicit: for attribute sampling of large populations, increasing population size alone often has little effect on the required sample. A defensible sample does not need to represent a fixed percentage of the population. What matters is the combination of tolerable deviation, expected deviation, and acceptable risk — not coverage as a percentage.
Where many teams fail is treating size as a convention — pulling a number from prior engagements without re-evaluating whether the inputs have changed. If the control population grew, or prior results showed a deviation rate you chose to ignore, last year's number may no longer be defensible.
Statistical versus non-statistical sampling
Not every defensible sampling approach is statistical in the formal sense. PCAOB AS 2315 explicitly permits both statistical and non-statistical sampling when properly applied. The distinction matters because it affects what you can legitimately claim about the result.
Statistical sampling requires that every item in the population has a defined, non-zero probability of selection, and that the evaluation uses probability theory to quantify sampling risk. Random selection is necessary but not sufficient — if the results are evaluated by judgment rather than probabilistic inference, the approach is non-statistical regardless of how items were chosen.
Non-statistical sampling can be defensible when the methodology is documented, the selection is tester-controlled, and the conclusions drawn are appropriately scoped to what non-statistical inference supports. The key requirements are the same in either case: the tester controls which items are selected, no discretionary substitution is permitted after selection, and the population from which items were drawn is validated for completeness.
The owner-selection problem is more specific than a prohibition on owner involvement. A control owner choosing which favorable records to submit for review introduces selection bias that undermines any inference. A control owner extracting records that the tester has already identified — by date, transaction ID, or other tester-defined criteria — is a different situation. What must be prevented is the owner's ability to exercise judgment about which items are returned. Require tester-controlled selection criteria, validate that returned records match those criteria, and confirm against the population listing that no items were substituted or omitted.
Control frequency, population and coverage
Frequency shapes the population available for testing, but it is not the same as the population relevant to the assertion — as the quarterly review example above illustrates. With that distinction established, frequency does provide useful guidance on which testing approaches are practical and what coverage decisions need to be documented.
The table below illustrates how control characteristics interact with sizing decisions. Item counts are intentionally omitted — the point is the direction of each factor, not a number. What constitutes an adequate sample is a judgment calibrated to the specific engagement context, not a lookup.
Sample derivation table
| Control Frequency | Population Characteristic | What Drives Size Up | What The Tester Must Document | Residual Limitation |
|---|---|---|---|---|
| Continuous / Automated | Configuration and exception population; log-based sampling may not be the primary approach | Risk sensitivity of control; frequency of configuration changes during period | Sampling methodology where applicable; change records; exception handling; IT general controls supporting the automated process | Log-based samples confirm log existence, not process fidelity — configuration testing and manual overrides require separate evidence |
| Daily | Large; potentially thousands per year | High expected deviation from prior results; coverage across full period | Stratification across weeks/months; rationale for sub-period coverage | Dense populations can mask clustering of failures in a specific sub-period |
| Weekly | Moderate; roughly 52 per year | Low tolerable deviation rate approaching zero; any failure is material | Randomization or selection method; demonstration that all weeks had selection opportunity | Small-period gaps (e.g., holidays, transitions) may create untested windows |
| Monthly | Small but enumerable | Low tolerable deviation rate; any gap is visible | Full population listing before selection; demonstration of selection method | Thin population limits statistical inference; one failure significantly affects rate |
| Quarterly | Very small at the cycle level; may be large at the transaction or decision level depending on the assertion | Determined by the relevant sampling unit, not cycle count alone | Sampling unit and population definition; rationale for coverage approach; 100% review rationale where applicable | Cycle-level population of 4 supports exception identification, not rate-based inference; see note below |
| Event-driven | Variable; depends on trigger frequency | Population size uncertainty; trigger rate variability across period | How the population was enumerated; how trigger rate was validated | If trigger count was lower than expected, low population size must be disclosed |
A note on quarterly and annual controls: When the cycle-level population is four or fewer items, statistical sampling to generalize about a deviation rate is not a viable approach — the population is too small for rate-based inference. Full population review is typically the only defensible option. This does not mean individual exceptions are indefensible. If a control fires four times and one instance is missing or inadequately performed, that is a valid exception supported by complete coverage of the annual population. The limitation applies to drawing statistical inferences about a broader rate of failure, not to identifying and documenting specific deviations. Document the decision to use full population review explicitly rather than applying a reduced sample to a thin population.
Tolerable deviation and why it is a judgment
There is no universal tolerable deviation rate you can import from a framework and apply without analysis. Under AS 2315.34, the tolerable rate of deviation is the maximum rate the auditor can accept while still relying on the control as planned — it is an auditing concept tied to the planned level of reliance, not simply a reflection of organizational risk appetite. In internal assurance work, the analogous concept is the maximum failure rate compatible with concluding the control is operating effectively. The rate represents how many control failures, expressed as a proportion of the population, can occur before that conclusion is no longer supportable.
That threshold depends on the severity of what the control is preventing and what level of assurance the engagement is designed to provide. Management's risk appetite and the organizational risk register are appropriate inputs to internal assurance planning. They do not determine an external auditor's threshold, and that distinction should be maintained when borrowing audit methodology for security control testing.
NIST SP 800-53A Rev. 5 approaches this through the attributes of depth and coverage applied across examine, interview, and test methods — recognizing that what constitutes adequate assessment depends on what's at stake and what the control is asserting (Source: NIST SP 800-53A Rev. 5, https://csrc.nist.gov/publications/detail/sp/800-53a/rev-5/final). A higher-risk control warrants a lower tolerable deviation rate, which drives sample size up.
Compensating controls can inform this judgment, but with a constraint: the existence of another detective control does not by itself justify accepting a higher failure rate in the primary control. The intended reliance on the compensating control and evidence of its effectiveness need to be established before using it to adjust the tolerable deviation threshold. Document the reasoning and get stakeholder agreement before testing begins, not after a deviation surfaces.
When the population cannot be enumerated
Some populations resist complete enumeration. Event-driven controls — triggered by user behavior, system events, or exception conditions — may not have a clean population list unless the triggering system logs reliably and completely. If you can't establish the population with confidence, you can't establish what fraction of it your sample represents.
The honest output in this situation is a scope limitation, not a larger sample. Documenting that the population could not be fully enumerated due to logging gaps, system architecture, or control design limitations is a more defensible position than implying statistical coverage you cannot support.
If a scope limitation is required, document what source was used to define the population, what gaps were identified, what steps were taken to validate completeness, and what assurance the sample does and does not provide. An auditor can work with a disclosed limitation. An auditor who discovers an undisclosed one after the fact will question the entire engagement.
What a sample size cannot fix
A sample of the right size, drawn correctly from a complete population, still can't compensate for testing the wrong thing. If the control objective is that privileged access is reviewed and revoked when no longer needed, a sample of review completion records doesn't test whether revocation actually occurred. The evidence type and the assertion it supports must align before sample size has any meaning.
This maps to the appropriateness side of the sufficiency/appropriateness distinction. Getting the quantity right while the quality is wrong is a common pattern in programs that have systematized evidence collection without revisiting whether the evidence speaks to the control.
What the results actually support
A correctly sized sample drawn from a complete population supports a conclusion about the defined population, subject to sampling risk — it does not establish that every untested item performed as expected. Understanding what the results do and don't prove is part of what makes a sampling plan defensible.
When deviations are found, they must be evaluated against the planned tolerance accounting for sampling risk. AS 2315.41–.43 provides the relevant framework: the auditor considers whether the observed deviation rate, projected to the population and adjusted for sampling risk, exceeds the tolerable rate. If it does, the planned reliance on the control is not supported. If it doesn't, the conclusion is that the control appears to be operating effectively — not that no failures exist in the untested population.
In practice this means you must document what deviation rate was observed, project that rate to the population, consider the upper bound of the estimated population deviation rate given the sampling risk you accepted, and compare that to your tolerable deviation threshold. If the upper bound exceeds the threshold, the sample does not support the planned conclusion regardless of whether any individual exception seems minor.
A worked example
The following is illustrative. It does not prescribe specific counts or rates for any control type — these are engagement judgments.
Control: Monthly privileged access review. Scope: 240 accounts reviewed across 12 months. Assertion: Each account included in the review population was evaluated and dispositioned (retained or revoked) within the defined timeframe.
Sampling unit: Individual account review decisions within each monthly cycle.
Population: 240 account-month instances across the review period.
Inputs to size decision:
- Tolerable deviation rate: 5% — the engagement team determined that a failure rate above this level would be inconsistent with planned reliance on the control.
- Expected deviation rate: 2% — based on prior period results showing two exceptions in approximately 100 items tested.
- Acceptable risk of assessing control risk too low: 10% — moderate reliance planned, consistent with the broader engagement risk assessment.
Derived sample size: Using standard attribute sampling tables for these inputs, the required sample is approximately 77 items. (This is illustrative; refer to the applicable sampling table or software for your specific inputs.)
Selection: Items selected by randomly generating account-month combinations from a complete population export. The tester controlled selection; the control owner provided the export and returned the identified records without substitution.
Results: 3 deviations observed in 77 items tested. Observed rate: 3.9%. Projected to the population at the upper bound of sampling risk (at the 90% confidence level), the estimated population deviation rate is approximately 9% — above the 5% tolerable threshold.
Conclusion: The sample does not support the planned reliance on the control. The three observed exceptions are valid findings, but the conclusion is not simply that three things went wrong — it is that the evidence does not support treating the control as operating effectively at the planned tolerance level. Document the observed rate, the projected upper bound, and the comparison to the tolerable threshold. The control owner's agreement or disagreement with individual exceptions does not change what the sample supports.
The defensibility test
Before signing off on a sampling plan, apply four questions. If any of them require a guess, the plan isn't ready.
Can you show how the sampling unit and population were defined and validated? If the population was taken from a system export without verifying completeness, document that limitation. If the sampling unit is ambiguous relative to the control assertion, resolve that before selection begins.
Can you trace each input to the size decision? Tolerable deviation, expected deviation, and acceptable risk of wrong conclusion should each have a documented basis — not a convention inherited from the prior engagement. Note whether the approach is statistical or non-statistical and what that means for the conclusions the results can support.
Can you demonstrate that selection was tester-controlled and free of owner discretion? Preserve the selection method and criteria, not just the selected items. If the owner extracted records based on tester-defined identifiers, document the validation step confirming what was returned matched what was requested.
Can you state what the sample does and doesn't prove? A sample that covers the period adequately supports a conclusion about the population subject to sampling risk — it does not prove the control has no failures. Document the observed deviation rate, the projected population rate, and how that compares to the tolerable threshold. The conclusion drawn must be consistent with that comparison.
These four questions surface in nearly every audit challenge to sampling methodology. A team that can answer all four with documentation — before the auditor asks — shifts the conversation back
