Add configurable affinity_mode (spread) for egress pod selection [port #1209] - #1
Open
Harishkrishna17 wants to merge 5 commits into
Open
Add configurable affinity_mode (spread) for egress pod selection [port #1209]#1Harishkrishna17 wants to merge 5 commits into
Harishkrishna17 wants to merge 5 commits into
Conversation
The current StartEgressAffinity always scores idle pods at 0.5 and busy
pods at 1.0. Combined with MaximumAffinity=1 in the psrpc client, this
means the first pod to accept any job wins all subsequent jobs until its
CPU budget is exhausted — a staircase pattern rather than even spread.
The existing code comment acknowledges this is intentional for mixed
fleets ("avoids having many instances with one track request each, taking
availability from room composite"). However for a TrackEgress-only fleet
the packing strategy provides no benefit and causes sequential scale-out
delays.
This commit adds a configurable affinity_mode field to ServiceConfig:
pack (default) — current behaviour, unchanged
spread — CPU-proportional scoring; idle pods score 1.0 and
win immediately via MaximumAffinity short-circuit,
busy pods score proportionally so the least-loaded
pod wins after ShortCircuitTimeout. Best for
single-type (TrackEgress-only) fleets.
type_aware — RoomComposite/Web requests prefer idle pods (1.0
idle / 0.5 busy); Track/Participant requests spread
by CPU load. Best of both worlds for mixed fleets.
Default is "pack" so all existing deployments are unaffected.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
(cherry picked from commit e3ab476)
…e_aware modes Addresses both root causes of the 24/51-job-on-one-pod skew observed in the 2026-05-07 load test: Cause A (strict > tie-break): psrpc's ShortCircuitTimeout means the first replier wins when all idle pods return the same score. Fixed by subtracting rand.Float32()*0.001 jitter so idle pods produce distinct scores, making the strict-> comparison effectively random among equally-idle peers. Cause B (m.requests.Inc lag): StartEgressAffinity is called before StartEgress, so the winning pod's m.requests counter stays 0 across an entire 200ms burst window and all callers see score 1.0. Fixed by a pendingClaims atomic.Int32 that increments at affinity time and decrements at StartEgress accept (consumePendingClaim). A 2s self-decay timer guards against claims that are never fulfilled. A CAS loop in consumePendingClaim ensures exactly one decrement fires per increment even when StartEgress and the timer race. New monitor helper AvailableCPUFractionWithPending deducts pendingSlots*TrackCpuCost from the available budget so the score decreases with each in-flight claim. Image: asia-south1-docker.pkg.dev/avian-pulsar-430509-f6/r41-livekit/egress:v1.12.0-r41.2 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> (cherry picked from commit ded32ad)
…eout In spread mode, an idle pod computed AvailableCPUFraction = 2.4/4.0 = 0.6. Since psrpc's MaximumAffinity is 1.0, the ShortCircuitTimeout (500ms fast path) never fired — every dispatch waited the full AffinityTimeout even when idle pods were ready. An idle pod (activeRequests=0, pendingClaims<=1) now returns 1.0 plus tiny jitter, triggering the 500ms short-circuit. Busy pods still return a proportional fraction. Also fix jitter direction bug in type_aware heavy-request path: 1.0 - jitter landed below MaximumAffinity; changed to 1.0 + jitter. (cherry picked from commit d2fc658)
When all egress pods are simultaneously over their CPU budget (Chrome cold-start storm), every pod returns -1 and the dispatcher gets zero bids. The job is permanently dropped. New config fields soft_reject_floor (float, default 0) and max_active_requests (int, default 0) allow a pod to return a small positive score instead of -1 when CanAcceptRequest is false but the pod is below its design capacity. The dispatcher can then select this pod as a last resort instead of dropping the job. Guarded by MaxActiveRequests so genuinely full pods still hard-reject. Also add unit tests for softRejectScore helper. (cherry picked from commit acf8a3d)
type_aware mode is not used in production (affinity_mode: spread in both config blocks). Reverts idle-1.0 and jitter-direction changes in that branch to keep the diff minimal and production-relevant only. build-push-ar.yml removed — images are built and pushed manually. (cherry picked from commit 3c41258)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports upstream livekit#1209 (
affinity_mode) into the Talview fork — the spread scorer validated live on VR prod (thev1.13.0-tv1image, SRE-4121).Why
For a track-only egress fleet, upstream's default
consolidate/packscorer packs many light track recorders onto one pod → CPU saturation → dropped session callbacks (VR saw 9–12 recorders on a 4-vCPU pod).affinity_mode: spreadscores by available-CPU ratio and distributes evenly.Config
Commits (cherry-picked from livekit#1209,
-xtrailer preserved)Add configurable affinity_mode for egress pod selectionfix(spread): pending-counter + jitter to fix burst skewfix(spread): idle pod returns 1.0 affinity to trigger ShortCircuitTimeoutfeat: add soft_reject_floor to prevent all-pods-silent dropsrevert: remove unused type_aware changes and CI workflowDeliberately excluded
Upstream livekit#1209 also carries a later
feat: add Redis heartbeat for CapacityManager integrationcommit — that is a separate concern (CapacityManager), not the spread fix, and was not part of what we validated in prod. Left out to keep this PR focused on spread. Raise it separately if the fork wants CapacityManager.Verification
Cherry-picks applied cleanly onto
main.pkg/config,pkg/server,pkg/statscompile clean. Fullgo testneeds gstreamer/cgo (CI/Docker only) — not run locally. This is the livekit#1209 half of the two spread options; livekit#1248 (load_distribution) is the sibling PR.