FFmpeg vulnerability ‘PixelSmash’ could enable RCE via video file

A heap buffer overflow in FFmpeg, dubbed “PixelSmash,” could be exploited to cause crashes or achieve remote code execution (RCE) in specific cases, JFrog’s Vulnerability Research Team reported Monday.

The vulnerability, tracked as CVE-2026-8461, with a CVSS score of 8.8, lies in the MagicYUV decoder component of FFmpeg. MagicYUV is FFmpeg’s default decoder for AVI, MKV and MOV media files.

The JFrog researchers found that an attacker could trigger a heap buffer overflow by exploiting a mismatch between how FFmpeg’s frame allocator and MagicYUV’s decoder determine chroma plane heights.

In the YUV420P pixel format, chroma planes (which determine color information) have half the vertical resolution of luma planes (which determine brightness), so the allocator determines the chroma buffer by halving the frame height, JFrog explained. When the frame height is odd, the halved value is rounded up.

The decoder follows a similar process but encodes video in horizontal slices and reads each slice height from an attacker-controlled bitstream. For example, in a file where the frame height is 32 and the chroma buffer is 16, and there are two slices with slice heights of 31 and 1, the decoder will round up and write an extra chroma row per slice, causing an out-of-bounds (OOB) write to occur for the last slice. The length of the OOB write depends on the frame width.

In FFmpeg’s heap layout, this OOB write could allow an attacker to overwrite the AVBuffer struct with their own content. This struct contains a function pointer that can be overwritten to point to the address for system() as well as pointer arguments that can be overwritten to point to the address of an attacker-controlled command. The command itself was written within 88 bytes of unused space at OOB offset 0.

The researchers demonstrated that the PixelSmash exploit could be used to achieve RCE in a Jellyfin media server, but only when Address Space Layout Randomization (ASLR) was disabled. ASLR is enabled by default in Linux, but the researchers argued an attacker could potentially defeat ASLR by chaining PixelSmash with a memory disclosure flaw that reveals the necessary addresses, noting “this chaining requires additional research and has not been demonstrated.”

The exploit uses an “auto-calibration pipeline” that leverages the GNU Project Debugger (GPD) to precisely determine the exact heap state prior to triggering the OOB write, ensuring attacker bytes are written to the correct offsets to overwrite AVBuffer while ensuring the glibc malloc chunk headers, glibc top chunk size and tchache entry in the OOB region are preserved to prevent crashing.

The researchers successfully executed a reverse shell running as the Jellyfin service user by uploading a crafted 50 KB AVI file and triggering a Jellyfin library scan. They further demonstrated that a similar attack could be achieved on the Nextcloud self-hosted cloud storage platform by simply uploading the crafted AVI file and ensuring the file is visible in the folder view. Nextcloud includes an optional Movie preview feature that invokes FFmpeg for thumbnail generation, making this a potential zero-click attack.

Even where RCE is not achievable, the heap buffer overflow flaw can be used for denial-of-service (DoS) attacks, as triggering the overflow leads to a crash. The widespread use of FFmpeg for media processing makes the potential blast radius of this exploit substantial, the researchers said.

Services that use FFmpeg are recommended to upgrade to version 8.1.2, which contains the fix for CVE-2026-8461. As a workaround, users can rebuild FFmpeg with the MagicYUV decoder removed or apply a minimal patch, provided by JFrog, that rejects the malformed slice_height values that trigger the OOB write.