Why Logs Fail During Incidents
Eight conditions cause cloud logs to fail as investigation evidence, each common in environments that assume adequate logging coverage:Logging absent in peripheral accounts and new regions. Cloud estates grow through account provisioning. Organizations without centralized governance frameworks — such as AWS Organizations, Azure Management Groups, GCP Organizations, Control Tower, Landing Zones, or enforced Infrastructure-as-Code provisioning — are at elevated risk of inconsistent logging coverage, because new accounts do not automatically inherit logging configuration. An attacker who pivots to a recently created, ungoverned account may find no audit trail. The diagnostic question: can you confirm that CloudTrail is enabled in every account created in the last 90 days?Retention shorter than investigation requirements. Log retention periods should be calibrated to regulatory obligations, forensic requirements, and operational risk — not set to defaults and left unchanged. Where retention windows fall short of compliance requirements or realistic dwell time estimates, attackers operate in periods with no recoverable evidence.Logs stored in the compromised account. If audit logs export to an S3 bucket or storage account in the same environment that was compromised, an attacker with sufficient permissions can modify or delete evidence before investigators reach it.Modern protection mechanisms — immutable storage, S3 Object Lock or equivalent WORM capabilities, cross-account log archives, and dedicated security accounts — directly address this exposure. What changes this: logs exported to a security-controlled destination outside the accounts they cover, with write protections the compromised account cannot override.Data-plane logging absent. Control-plane logs capture API calls; data-plane logs capture what was accessed. An S3 bucket access log, Azure Storage blob access record, or GCP Data Access Audit Log is the only way to determine whether a public resource was accessed. Without data-plane logging, investigators know a bucket was exposed but not whether it was read.Identity events lack session context. A CloudTrail entry shows which role was used; it does not by itself show which role assumption chain produced that session or whether the original credential was federated, service-account-based, or stolen. Reconstructing the full identity thread increasingly requires correlation across multiple sources: cloud audit logs, Identity Provider (IdP) federation records, Privileged Identity Management (PIM) systems, Identity Governance and Administration (IGA) platforms, and Cloud Infrastructure Entitlement Management (CIEM) solutions. Cloud audit logs are one input into identity investigation — not the complete picture.Logs not queryable under pressure. Logs distributed across accounts or regions without centralized query capability require investigators to manually extract, transform, and search during an active incident. Modern practice extends beyond log aggregation: SIEM platforms, security data lakes, and cloud-native security analytics tools normalize, correlate, enrich, and prioritize events across providers. Investigation readiness depends as much on these downstream capabilities as on collection itself. When they are absent, investigation speed collapses even when the underlying evidence exists.Evidence scope narrower than the attack surface. Cloud incident investigations increasingly require evidence beyond control-plane and data-plane logs. Endpoint telemetry, Kubernetes audit logs, container runtime logs, DNS logs, network flow telemetry, and SaaS audit logs are all relevant sources depending on the environment. Multi-cloud estates and cross-tenant architectures introduce additional complexity: API version changes, inconsistent logging configurations across providers, and cross-account visibility gaps each create blind spots that investigators encounter under time pressure.Pre-approved containment absent. Logs that produce attacker activity reconstruction are only useful if the organization can act on the reconstruction. An investigation that identifies compromised identity but cannot revoke it — because the authority to do so requires approval — produces scope knowledge without containment capability.What the Gap Enables
When logging-readiness gaps exist, attackers gain five specific capabilities:Extended dwell time. Actions that fall outside retention windows or occur in unlogged accounts allow persistence beyond organizational estimates. NSA and CISA joint guidance on cloud security specifically identifies logging gaps — including insufficient log retention, logs not centralized, and telemetry not covering all accounts and regions — as conditions that impair incident response capability and that adversaries actively exploit to reduce detection probability.Scope concealment. Actions in unlogged accounts, services, or regions remain invisible to investigators, enabling resource access without appearing in investigation timelines. The attacker operates in authenticated spaces with no evidence trail.Evidence destruction. An attacker with permissions to modify logging configuration can delete CloudTrail trails, remove log archives, or change export rules before investigation reaches the relevant time window. Where logs are not protected by immutable storage or isolated security accounts, the control plane becomes a weapon against investigation capability.Investigation delay as access time. When logs cannot reconstruct scope, investigators require longer to determine containment boundaries. Each day of investigation delay provides the attacker additional time in the environment.Disclosure uncertainty. When data access scope cannot be confirmed from logs, notification obligations under GDPR, HIPAA, and SEC disclosure rules require assuming potential exposure of all data the attacker could have reached — expanding notification scope beyond what evidence supports.What This Looks Like at Incident Time
When the logging-readiness gap hits during active cloud incidents, investigators encounter specific failure modes:Log retention that ends before the estimated compromise date — not because the compromise was recent, but because default settings expired the evidence. Accounts with no CloudTrail activity that are not dormant — logging was never configured for accounts provisioned outside the centralized governance framework after initial setup.Identity events showing role assumptions without the session context needed to reconstruct the full chain — investigators can see that a role was used but cannot determine how the attacker obtained the authority to assume it without correlating across IdP, PIM, and cloud audit log sources. Data-plane logs absent for storage resources most likely accessed — S3 bucket permissions suggest data exposure, but access logs cannot confirm whether reads occurred.Evidence that logging configuration was modified after initial compromise — the attacker used control-plane permissions to reduce investigation capability before investigators arrived. Query interfaces that require manual correlation across multiple log sources and providers — no centralized search capability when containment decisions need answers in hours, not days.The investigation produces partial timelines with gaps that prevent scope determination. Containment decisions require assumptions about attacker reach rather than evidence-based boundaries.Decision Checklist
Six diagnostic questions testable now:-
Account and region coverage: Is CloudTrail (or equivalent) enabled in every cloud account and every region — including accounts created in the last 90 days? If account provisioning is not governed by a centralized framework, how is logging configuration verified for new accounts?
-
Evidence preservation: Are logs exported to a security-controlled destination outside the accounts they cover — protected by immutable storage, Object Lock or WORM capabilities, or a dedicated security account the compromised account cannot modify or delete?
-
Retention adequacy: Does log retention satisfy applicable regulatory and compliance requirements, and does it support investigation of realistic attacker dwell times across all accounts, subscriptions, and projects?
-
Data-plane visibility: Are data-plane access logs enabled for storage services and other resources containing sensitive data?
-
Investigation capability: If your highest-permission cloud identity were compromised today, can you produce a complete reconstruction of every action it has taken in the last 30 days — across all accounts, regions, services, and relevant evidence sources including endpoint telemetry, Kubernetes logs, and SaaS audit logs — within the time containment decisions require? Does your SIEM, data lake, or security analytics platform support that reconstruction without manual extraction?
-
Readiness validation: Has your organization conducted a tabletop exercise or log reconstruction exercise in the last 12 months that tested whether investigation capability holds under simulated incident conditions? Many organizations discover logging gaps in simulation rather than during actual incidents — which is the preferable order.
Sources
- NSA/CISA Joint Cybersecurity Advisory AA23-278A, https://www.cisa.gov/news-events/cybersecurity-advisories/aa23-278a