Risk advisory: image scans do not prove runtime safety

Risk statement

Organizations with comprehensive image scanning, CWPP deployment, and CVE prioritization workflows still carry concentrated runtime risk. The gap: vulnerability scans confirm pre-execution state but cannot govern runtime behavior. A scanned, patched, approved image can execute with unsafe privileges, connect to unauthorized services, abuse its workload identity, and behave differently in production than in test environments — without any of those behaviors appearing in scan results.

The diagnostic question: if a container in your production environment were compromised right now — assuming a known-good image, zero CVEs, and clean vulnerability scan — would your program detect the compromise, and how long would it take? If the answer depends on the container exhibiting a previously known-bad behavior pattern, the program detects after the fact. If the answer requires manual reconstruction from scattered logs, the program has a runtime visibility gap.

Why runtime risk persists

Six operational gaps create persistent runtime risk in organizations with active workload security programs:

Gap 1 — Image scans confirm pre-execution state only.
Vulnerability scanning identifies what CVEs existed in the image at build time but does not govern runtime behavior. A vulnerability-free image can run with root privileges, make unauthorized API calls, or establish connections to services outside its documented function.

Gap 2 — Runtime baselines are absent or stale.
Workloads execute without documented expectations of normal process, network, and API call behavior. Without baselines, anomaly detection cannot distinguish expected behavior from deviation. Note that behavioral baselines alone are insufficient in microservice architectures: the volume of legitimate inter-service variation generates alert fatigue that degrades detection effectiveness. Deterministic enforcement controls — blocking unauthorized system calls, enforcing read-only root filesystems, and stripping shell binaries through distroless container images — must accompany or, in high-sensitivity workloads, replace passive baseline monitoring.

Gap 3 — Workload identity is not mapped to runtime behavior.
Service accounts and instance roles attached to workloads receive permission scope reviews but no monitoring for whether actual API call patterns during execution match the workload's function. An overprivileged workload identity that is never used may not surface as risk; the same identity suddenly making API calls it has never made before should trigger detection.

Gap 4 — East-west communication is not governed.
Service-to-service connections receive authorization through network policies but no baselining for expected call patterns, volumes, or targets. A compromised workload pivoting through authorized connections produces no detection signal.

Gap 5 — Serverless and ephemeral workloads are treated as configuration objects.
Functions and short-lived containers generate no persistent telemetry if runtime logging is not explicitly configured, creating execution windows with no evidence of what ran. Critically, user-space log collection from within the container is insufficient: if the container terminates or an attacker achieves a container escape before logs are flushed, that telemetry is lost. The architectural requirement is streaming kernel and container lifecycle events out-of-band to a write-once, immutable sink before the pod terminates — ensuring the evidence record is independent of workload availability.

Gap 6 — Container privilege and capability scope is not continuously validated.
Containers that deployed with unsafe privileges — root, privileged mode, excessive Linux capabilities — remain in production environments after initial misconfiguration findings are closed without confirmed remediation.

The instrumentation requirement

The advisory identifies visibility and monitoring as program requirements throughout. The technical mechanism matters: polling container logs or deploying user-space agents is insufficient and is bypassed by container escapes, because a process that has exited the container namespace is no longer observable through container-layer instrumentation.

Effective runtime visibility requires instrumentation at the Linux kernel level using eBPF (Extended Berkeley Packet Filter). eBPF-based tools — including Tetragon, Falco with eBPF drivers, and Cilium for network enforcement — attach to kernel execution paths and observe system calls, process execution, network connections, and file access before those events reach user space. This means:

Programs that rely solely on container log aggregation or agent-based monitoring should treat kernel-level eBPF instrumentation as a gap, not an enhancement option.

What the gap enables

Runtime control gaps enable four specific attacker capabilities:

Workload compromise without detection.
An attacker operating through a workload that has no runtime baseline, no behavior monitoring, and no process execution alerting produces no signals in detection systems. The compromise persists until secondary indicators surface or manual investigation discovers it.

Privilege escalation from workload to cloud control plane.
A compromised workload with an overprivileged service account can make cloud API calls that extend compromise from the workload layer to cloud account administration. This transition is a runtime event that neither image scanning nor posture tools surface.

Lateral movement through authorized connections.
A compromised workload pivoting through legitimate service-to-service connections extends compromise without triggering network perimeter detection. East-west movement remains invisible when communication governance is absent.

Incident scope uncertainty.
NSA and CISA joint guidance identifies that cloud environment breaches consistently involve workload-level compromise paths including abuse of workload identities, container escape, and lateral movement through service-to-service connections — establishing that runtime behavior governance gaps, not only vulnerability presence, are among the most persistently exploited conditions in cloud incidents (Source: NSA/CISA Joint Cybersecurity Advisory AA23-278A, https://www.cisa.gov/news-events/cybersecurity-advisories/aa23-278a).

What this looks like at incident time

When runtime control gaps are present during an active incident, investigators encounter:

A compromised workload with no runtime baseline, making it impossible to determine whether observed behavior represents compromise or normal operation. Teams cannot distinguish malicious activity from legitimate workload function.

Workload identity API calls to cloud services that cannot be attributed to legitimate application function. The investigation must assume worst-case privilege abuse because no baseline exists for comparison.

East-west connections to downstream services that the investigation cannot scope. Without communication baselines, teams cannot determine which services the compromised workload legitimately accessed versus which represent lateral movement.

An execution window for ephemeral workloads or deleted containers where no telemetry was captured — a direct consequence of relying on in-container log collection rather than out-of-band kernel event streaming. Investigators face evidence gaps for the entire compromise period.

IBM Cost of a Data Breach Report 2024 identifies that breach identification and containment time is directly correlated with total breach cost — establishing that runtime visibility gaps that delay workload incident scope determination are direct drivers of breach cost, not only technical program failures (Source: IBM Cost of a Data Breach Report 2024, https://www.ibm.com/reports/data-breach).

Each gap extends the investigation timeline and forces worst-case assumptions for containment decisions.

Decision checklist

Five diagnostic questions testable now:

Can the program enumerate every running workload across all accounts and regions and confirm that runtime behavior monitoring covers each?
Include containers deployed in the last 24 hours. Confirm that monitoring operates at the kernel level, not solely through container log aggregation.
Gap indicators: workloads discovered during incident response that were not in the monitoring inventory; monitoring coverage that does not survive a container escape.

Do runtime controls exist for each workload type that enforce expected behavior — not only detect deviation from it?
This includes blocking unauthorized system calls, enforcing read-only root filesystems, and using distroless images to eliminate shell access. Passive anomaly detection alone is insufficient where alert volume from microservice architectures prevents timely triage.
Gap indicators: anomaly detection with no enforcement layer; alert queues that are routinely deprioritized due to volume.

Are workload identities mapped to the workloads that use them, with monitoring for API call patterns that deviate from those workloads' function?
This extends beyond permission scope reviews at provisioning time.
Gap indicators: workload identity API calls that cannot be attributed to specific workload functions.

Is east-west service-to-service communication baselined so that a workload establishing a new connection to a downstream service triggers a detection event?
Gap indicators: lateral movement discovered through manual investigation rather than automated detection.

If a workload were compromised today and then deleted or recycled, would the program have captured sufficient runtime evidence to reconstruct what happened during the compromise window?
Evidence must include process execution, network connections, and API calls streamed out-of-band to an immutable sink before workload termination — not collected from the container after the fact.
Gap indicators: incident investigations that rely on secondary indicators because primary workload telemetry was not captured before the container exited.

Sources