Unbounded memory growth in libvmaf 3.2.0 when the decoder outpaces feature extraction (regression from 3.1.0)

[csk61-gh]

Under ffmpeg's libvmaf filter, libvmaf 3.2.0 accepts pictures faster than it scores them and queues the surplus without bound, growing until the OS kills the process. libvmaf 3.1.0, with the same ffmpeg binary, command, and input, holds flat memory indefinitely.

This only manifests when the decoder can outrun feature extraction — here, two 4K HEVC 10-bit streams via hardware decode (~167 fps each) feeding libvmaf (~21 fps).

Environment:

• macOS 26.6.2, Apple M1 Max (10 cores), 64 GB RAM
• ffmpeg 8.1.2 (Homebrew), --enable-libvmaf --enable-videotoolbox --enable-gpl --enable-version3
• libvmaf 3.2.0 (fails) / 3.1.0 (works) — Homebrew bottles, swapped via DYLD_LIBRARY_PATH
• Input: two 3840×2160 yuv420p10le HEVC files, 24 fps, 2:36:22, 225,177 frames

Reproduce:

ffmpeg -hwaccel videotoolbox -i reference.mkv -hwaccel videotoolbox -i distorted.mkv \
  -lavfi "[0:v][1:v]libvmaf=n_threads=10:shortest=1" -an -sn -f null -

Monitor with footprint -p <pid> — not ps -o rss, which falls as pages swap out and hides the growth.

Result:
3.2.0 - 26GB at 20s, 131GB at ~8 min, then SIGKILL by jetsam. ~44fps vmaf speed
3.1.0 - flat 2.1GB for 3 minutes, swap unmoved, ~21fps vmaf speed

Growth is ~0.27 GB/s, consistent with (decode rate − scoring rate) × 24.9 MB per 4K 10-bit frame. Failure is deterministic: two independent full runs died at frame 9,916 and frame 10,065.

The reported fps is itself diagnostic. 3.2.0's ~44 fps counts pictures accepted into the queue, not scored. 3.1.0's ~21 fps is the true rate, and matches a full run that completed on 3.1.0 in 3h38m (0.717x).

As a control:
the same framesync path with 'ssim' is fine.

ffmpeg -hwaccel videotoolbox -i reference.mkv -hwaccel videotoolbox -i distorted.mkv \
  -lavfi "[0:v][1:v]ssim" -an -sn -f null -

Flat at 918 MB for 3 minutes at 113 fps — a 54 fps producer/consumer gap absorbed with no growth. So ffmpeg's framesync layer applies backpressure correctly; the difference is libvmaf.

Decode alone is also clean: one input to -f null holds 304 MB at 274 fps; two inputs hold 590 MB at 335 fps combined.

n_threads makes it worse, not better

n_threads=1 reached 72 GB in 20 seconds (vs 26 GB at n_threads=10), which is consistent with n_threads setting the drain rate of an unbounded queue rather than its capacity.

Scores are unchanged

Over 1,440 frames with vmaf_4k_v0.6.1: mean 98.419877 (3.1.0) vs 98.419869 (3.2.0); min 91.551662 vs 91.551591; harmonic mean 98.380873 vs 98.380864. Differences ~1e-5 — this is purely a memory/flow-control regression.

Ruled out

• PTS mismatch / frame-count disparity. Both inputs start at PTS 0 with identical timebases. Growth is identical when timestamps are regenerated from frame index (settb=AVTB,setpts=N/(24*TB) on both), and occurs at frame ~10,000 of 225,000 — nowhere near EOF.
• shortest=1 — removing it changes nothing (27 GB in 20 s).
• -thread_queue_size (8 and 32), -extra_hw_frames (2 and 8) — no effect.
• format=yuv420p before the filter — halves frame size, growth continues (21 GB in 20 s).
• Software decode — narrows the gap so growth is slower, but it continues.
• ffmpeg version — 8.1.1 fails identically when linked against libvmaf 3.2.0, which is how we isolated libvmaf as the variable.

Possibly related in the 3.2.0 release notes

VMAF_BATCH_THREADING / VMAF_PICTURE_POOL threading modes, and USE_DIRECT_READ ("eliminate intermediate buffer and memcpy"). I bisected the version but did not read the diff, so I have not identified the specific change.

vmaf_read_pictures() is documented as queueing pictures and taking ownership, so queueing is by design — the question is whether that call still applies pressure when the queue is saturated.

[lusoris]

There is already a post-3.2.0 upstream fix directly relevant to this: 8fc71e3, “fix thread pool queue depth to restore backpressure”. It is present in current master (f85a853692a8c730d0270cd733c8bb30b5b93b7c), but absent from the v3.2.0 tag (3f9e02af258a5c0e30124fc585a3c3af90126dee).

The change makes vmaf_thread_pool_enqueue() wait when the queued-job count reaches n_threads, and wakes producers when workers take jobs. In the release version, that enqueue path has no corresponding queue-capacity wait. This matches the producer/consumer mechanism you isolated, including why reducing n_threads does not bound the release build’s memory.

A useful next comparison is your same FFmpeg/input setup linked against current master, or 3.2.0 with just that commit applied. I verified the source/tag distinction; I have not rerun your macOS/VideoToolbox workload and cannot claim its measured memory growth is resolved on that setup.

[csk61-gh]

Thank you for your incredibly fast response, it's greatly appreciated!

That was the right pointer, and I can confirm it fixes the issue on this workload.

I built the v3.2.0 tag with only 8fc71e3 cherry-picked (the second option you suggested), so the
only difference from the release is that one commit.

Build

base:     3f9e02af258a5c0e30124fc585a3c3af90126dee   (v3.2.0)
applied:  8fc71e3006f0b21e8e31d6e5d1b904332149ad9e   (cherry-picks cleanly, 19 insertions in
                                                      libvmaf/src/thread_pool.c)
config:   meson setup build libvmaf --buildtype=release --libdir=lib --wrap-mode=nofallback

That matches the Homebrew formula's build arguments, and xxd was present so the built-in models are
compiled in. The resulting dylib was swapped in via the same soname, so ffmpeg is otherwise untouched.

Environment

• MacBook Pro, Apple M1 Max, 10 cores, 64 GB, macOS 26.6.2 (25G83)
• ffmpeg 8.1.2 (Homebrew), --enable-libvmaf --enable-videotoolbox
• Content: two 3840×2160 yuv420p10le HEVC files, 24 fps, 225,177 frames (2:36:22)
• -hwaccel videotoolbox on both inputs, n_threads=10, shortest=1, model vmaf_4k_v0.6.1

Memory

build	footprint	outcome
v3.2.0 (stock)	12 GB @ 10 s → 24 GB @ 20 s	unbounded; killed at my 20 GB test cap
v3.2.0 + 8fc71e3	flat 4.62 GB	bounded

The patched build has now been running 18+ minutes (~30,000 frames) inside the real application,
holding 4.62 GB with zero drift — the same figure to two decimals across more than 70 consecutive
15-second samples, with swap unmoved.

For reference, the stock build previously died deterministically at ~8 min / frame ~10,000 on this
pair (two independent runs: 8m34s at frame 9,916, and 8m11s at frame 10,065), reaching ~131 GB with
~63 GB of swap in use before jetsam killed it. The patched build is now ~2.5× past that point with no
growth at all.

Scores are unaffected

Over a 480-frame window, every per-frame VMAF value and every pooled metric is bit-identical
between stock and patched. The commit is purely a concurrency fix with no numerical impact.

One incidental finding that may be useful

The stock build's reported fps was misleading. On this pair, stock v3.2.0 reports 39–42 fps while
the patched build reports 28–31. The stock figure appears to count frames accepted into the unbounded
queue rather than frames actually scored — consistent with the missing backpressure. The 28–31 fps is
the honest throughput, and it is still a solid improvement over 3.1.0, which measures ~21 fps on a short
run of this pair and averaged ~17 fps (0.717x) over a completed full-length pass. So the performance
work in 3.2.0 looks real; it was just masked by the bug and simultaneously overstated by the readout.

Also worth flagging for anyone else reproducing this: ps -o rss gives the wrong answer. RSS
falls as the backlog is paged out, which reads like the problem resolving itself while the process is
actually being driven toward jetsam. footprint -p <pid> (what Activity Monitor's Memory column shows)
tracked it correctly.

Scope of what I verified

I tested v3.2.0 + that single commit, not current master, so I can't speak to the other 12 commits since
the tag. This is one platform (macOS/arm64, VideoToolbox decode) and one content type. I did not attempt
to characterise behaviour at other n_threads values beyond confirming the original report that
n_threads=1 made the unpatched build worse — which is consistent with the mechanism, since without
a depth cap n_threads is only the drain rate.

Note on releases

Since 8fc71e3 landed 11 days after the v3.2.0 tag and isn't in any released tag, everyone installing
libvmaf 3.2.0 from a package manager currently gets the unbounded version. On this machine Homebrew
installed 3.2.0 on 2026-07-11 — ten days after the fix already existed in master. If a 3.2.1 is
plausible, it would save others the bisect. Happy to test a release candidate on this setup if that's
useful.

[yohimik]

I also isolated the backpressure behavior with a small pthread probe compiled against the two versions of
libvmaf/src/thread_pool.c. It creates one worker, holds that worker inside the first job, and has another thread try
to enqueue 1,000 no-op jobs:

source	additional jobs accepted after 300 ms while the worker is held
v3.2.0	1,000
8fc71e3	1

Both variants drained all 1,000 jobs after the held worker resumed. This does not reproduce the full FFmpeg,
VideoToolbox, or memory workload; the reporter's A/B run already supplies that evidence. It independently exercises
the queue-capacity mechanism and may be useful as a regression test: while all workers are held, enqueue should stop
advancing once queued depth reaches n_threads, then resume when a worker removes a job.

[kylophone]

Thanks, I just cut v3.2.1 and the fix (8fc71e3) is in.