Feature/egress h264 passthrough - #1295
Conversation
Adds an enable_video_passthrough deployment flag (default false). When set, track composite and participant egress with HLS segment output no longer force VideoDecoding/VideoEncoding at request time: the request becomes a pass-through candidate if it subscribes to at most one video track and carries no video encoding options (preset or advanced). Any file, stream, or image output, and any input codec that does not match the output codec at subscription time, falls back to the default decode+encode pipeline with an informative log. With the flag off, behavior is byte-for-byte identical to before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
In pass-through mode there is no local encoder, so the GstForceKeyUnit events emitted by splitmuxsink (send-keyframe-requests=true) at segment boundaries died unanswered. The keyframe probe now installs an upstream event probe on the parser src pad when video decoding is disabled and bridges those events to the appwriter's existing PLI path, reusing its 1s throttle rather than adding a second one. Both the event receipt and the actual PLI send are logged with nanosecond timestamps so the egress->SFU->publisher keyframe delay can be measured in staging. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|
Without a decoder, the input-selector + videotestsrc fallback does not exist in pass-through mode: a muted or disconnected publisher stops the buffer flow and splitmuxsink stalls, freezing the HLS playlist and the viewer's player. The selector machinery is now also built in pass-through mode, operating on the encoded H.264 stream, with the videotestsrc replaced by a freeze injector: an auxiliary appsrc that caches the last keyframe seen on the real track and re-pushes it (2fps, do-timestamp PTS) while the fallback pad is selected. Since every injected frame is an IDR, splitmuxsink keeps cutting segments at the nominal duration while frozen. Switching uses the existing mute/unmute/removed callbacks - the appwriter's inactivity path (jitter buffer latency) already acts as the sample watchdog, and unmute re-selects the track pad, dropping delta frames until the PLI-requested keyframe arrives. A secondary stall detector (2x segment duration) logs if data stops without any mute event firing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
#EXTINF entries already carry the measured duration of each closed fragment, so variable-length segments were mostly handled. But EXT-X-TARGETDURATION was declared as the nominal segment duration, and in pass-through mode real segments regularly exceed it (keyframe boundaries depend on the publisher GOP + PLI round trip), violating the HLS spec requirement that no segment's rounded duration exceed the declared target. Pass-through playlists now declare 2x the nominal duration, and any segment that still exceeds the declared target logs a warning. Audited remaining SegmentDuration uses: splitmuxsink max-size-time (split threshold, intentional), x264 vbv-buf-capacity (encoder path only), and upload queue sizing - none assume fixed real durations. Player note documented in README: window sizing by segment count (hls.js liveSyncDurationCount) sees slightly variable latency. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Failure-focused staging scenarios for the pass-through feature: without a decoder there is no loss concealment, so a corrupted bitstream reaches h264parse/mpegtsmux directly and must degrade gracefully. - publish.sh: synthetic OBS-like publisher (pre-encoded H.264, fixed 2s GOP + Opus) over WHIP bypass, with pause/resume/stop for mute and disconnect scenarios - egress.sh: participant egress with HLS output; --transcode adds a preset to force the decode+encode fallback on the same deployment - netem.sh: port-scoped packet loss on loopback (publisher<->ingress and SFU<->egress legs) - measure-cpu.sh: container + per-handler CPU sampling to CSV for the 30-minute pass-through vs transcode comparison - check-avsync.sh: per-segment audio/video first-PTS offset (transcoded audio vs pass-through video drift detection) README maps each script to the five scenarios and their pass criteria. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
biglittlebigben
left a comment
biglittlebigben
left a comment
There was a problem hiding this comment.
Thanks for the PR. We tried to implement passthrough video export in the past but ended up going away from it because without transcoding it is simply impossible to ensure that the stream abides to the required parameter:
- No way to split the stream in segments at the required boundary (other than best effort). The HLS spec particularly forbids any segment longer than the target duration https://datatracker.ietf.org/doc/html/rfc8216#section-4.3.3.1.
- Dealing with track muting properly is not possible. Your patch repeats the last I frame, which would yield a very awkward stream in most cases.
Thanks for taking the time to review this — you're right that a general-purpose passthrough can't guarantee segment boundaries, and that's exactly why this is behind an opt-in config flag rather than a default. But I'd like to push back on two specifics. On the TARGETDURATION requirement: the RFC text you quoted constrains each segment's EXTINF duration, rounded to the nearest integer, against the declared target duration — not against the requested segment_duration. Nothing in the spec requires declaring the nominal segment duration as the target. This PR declares 2× the nominal duration in pass-through mode (segments.go), keeps the measured durations in #EXTINF (already the existing behavior), and logs a warning if a segment ever exceeds the declared target. Deriving TARGETDURATION from actual max segment duration rather than the requested one is what ffmpeg's HLS muxer and Shaka Packager do as well. Note the current transcode path also produces variable-length segments — splitmuxsink only cuts at keyframes even with a local encoder; the difference is overshoot magnitude (~1 frame vs one GOP + PLI RTT), not kind. Where your concern is fully valid: if the publisher never delivers a keyframe within the headroom (ignores PLI, sustained loss), the segment exceeds the declared target and nothing can enforce it — that's a real, unbounded tail for arbitrary publishers. That's the case I'd scope this feature away from: it's intended for deployments that control the publisher (fixed GOP ≤ segment_duration), which is where the CPU win matters most. Happy to make that constraint louder in docs, or fail back to transcode at runtime when the observed GOP exceeds the headroom. On the freeze fallback: the alternative in the current decoded path is videotestsrc pattern=black — a black screen. Repeating the last IDR gives a frozen frame, which is what most conferencing UIs show on mute, and it's a valid stream that keeps segment cuts exact (every injected frame is a keyframe). The genuinely rough edge is the unmute transition (delta frames dropped until the PLI keyframe arrives, so freeze→jump). If the freeze behavior itself is the sticking point, I'm happy to make the fallback configurable. Would you be open to this landing as an explicitly opt-in mode with documented publisher requirements, or is the position that passthrough won't be accepted upstream in any form? If the latter, no hard feelings — I'll maintain it as a fork and would just appreciate a pointer to the earlier attempt you mentioned, to learn from what broke. |
3 participants
No description provided.