Advice needed for implementing custom filters / blurs to video streams

[optical]

Hi! We're trying to implement our own version of blurring for live streams & recordings. At it's core, we'd like to train and run our own model for blurring sensitive regions in video feeds. These regions will be dynamic & will move around with the objects being detected - so we'd like to implement a detection pass, render a blur over that detected region and have that served up. The core issue being a synchronous pass/pipeline of an image -> compositing -> rendering of the feed.

I've tried to implement this by consuming a VDO stream alongside axoverlay2. For each frame from the VDO Stream, I draw onto the overlay (setup with axo_props_set_timesynced=true). However I've noticed that the overlay is always a few frames behind, despite this process running comfortably at 25fps - there is no dropping of frames.

What I've noticed is that the frame I get from my VDO stream's timestamp (via vdo_frame_get_timestamp) is always ~250+ms later than clock_gettime. So I suspect that when I call axo_buffer_set_timestamp_to_submit and then axo_submit_buffer - it's already too late as the source frame has gone out the door, so to speak. This results in the overlaid region visibly lagging behind the rest of the image - ie the moving object appears to enter the region late (partially disappears), then then remains in it longer when exiting (see it twice)

Essentially, I feel like I am either missing a knob to adjust or have fundamentally approached this issue of being able to alter a frame before it's baked into the stream the wrong way.

For reference, I have mostly been testing this on a Q1728 on 12.10.

Any help is much appreciated!

[Cacsjep]

@optical
I observed the same issue while trying to implement a live filter. A proper implementation would need a way to modify the pixels and return the processed frame to VDO, or feed it into another supported output pipeline. However, VDO appears to be consumer-only, and I haven’t found a way to submit modified frames. Unfortunately, the resulting delay makes unsuitable for fast-moving scenes.