live runners: orchestrator keeps selling paid reservations for a runner slot stuck with a stale session

[rickstaa]

Summary

An orchestrator running live runners can get into a state where its single runner slot is permanently held by a stale session, yet it keeps accepting paid reservations for that slot indefinitely. Every reservation is paid through the remote signer at reserve time, then fails at POST .../app/stream with 409 runner already has an active session. Observed against a third-party orchestrator on Arbitrum mainnet during sustained load testing: 559 paid-but-undeliverable reservations in one hour, while three other orchestrators in the same run served 386/386 sessions cleanly.

Environment

• Client: livepeer-python-gateway ja/live-runner @ 89ac0f7, load test looping reserve_session -> POST /stream -> drain -> stop_runner_session
• App: livepeer-example/flux-klein (persistent mode, capacity 1)
• Payments: remote signer (-remoteSigner, live per-second pricing), Arbitrum mainnet
• Orchestrator: third party, recent build (advertises the new price schema from runner: Add fixed pricing #3999), exact version unknown

Timeline / two related failure modes

Run 1 (30 min): the first session's stop failed with HTTP 404; body='runner not found', stranding the session on the capacity-1 runner. Every subsequent attempt for the rest of the run did: reserve OK -> signer payment created -> POST /stream -> 409 runner already has an active session. 284 attempts, 0 successes, 165 payments signed to the orch's recipient.

Operator restarted the runner (new runner_id), discovery showed capacity_used: 0, capacity_available: 1.

Run 2 (60 min): zero stop failures from the client this time, yet the fresh runner 409'd from the very first stream attempt to the last: 559 attempts, 0 successes, every one preceded by a successful paid reservation. So the runner's internal "active session" state diverged from the orchestrator's capacity accounting without any client-side misbehavior, and the orchestrator sold that dead slot for a full hour.

Healthy control group in the same runs: three other orchestrators (including one running stock v0.9.0 images) at 100% success across 386 sessions, so client-side stop handling is not the trigger.

Client-side evidence

reserve/NoRunnerAvailableError: All runners failed (1 tried):
  https://<orch>/apps/runner_dschhn5u/session/159bc96c/app/stream:
  HTTP 409; body='runner already has an active session'

(x559, each after a successful /generate-live-payment on the signer)

Run 1 leak precursor:

stop/LivepeerGatewayError: HTTP empty POST error: HTTP 404; body='runner not found'

Expected behavior / suggestions

1. Reservation should verify the slot is actually deliverable before charging: if the runner reports an active session for its only slot, the reserve should fail (402/503), not collect a payment for a stream that can only 409.
2. Stale session TTL / reaping: a runner session with no traffic (no stream started, or stream gone) should be reaped after a timeout so a leaked or desynced session cannot hold a slot forever. The load-test docs already call this out ("no idle TTL").
3. On runner not found at stop time, the orchestrator should release its own capacity accounting for that session rather than leaving it to diverge.

Happy to provide full load-test logs (951-session run summary, per-session verbose log) on request.

[linear-code]

ENG-208

[rickstaa]

8-hour overnight soak (5 orchestrators, 8113 session attempts) adds harder evidence and a second flavor of the same gap:

• Healthy control group: 2875/2875 sessions OK (100%) across three orchestrators, zero leaks all night.
• The affected orchestrator (operator fixed the original 409 wedge mid-evening) now fails one stage deeper: reserve accepted -> payment signed -> POST /stream accepted -> publish fails with "Trickle publisher channel does not exist". 1586 consecutive failures over 8 hours, and the orchestrator collected 1587 payments (one per reserve, roughly every 18s, all night).

So the payment-before-deliverability gap is not specific to the session-gate 409: any orchestrator that accepts a reservation it cannot service (wedged slot, missing trickle channel, broken internal routing) collects payments indefinitely. Suggests the fix should gate payment on the stream/channel actually being established, or refund/void the payment when the session dies before first byte.

[j0sh]

held by a stale session, yet it keeps accepting paid reservations for that slot indefinitely

Does this "stale session" count as used capacity? Or does the orchestrator see the capacity as free when the runner itself is actually busy?

[rickstaa]

Update from a 15.75h continuous-stream soak (5 orchestrators, funded held sessions via the session-payments SDK loop): the wedge now reproduces organically and locally, no operator intervention involved.

Organic sequence, observed twice in one run:

1. A funded held stream runs for hours (11.4h on the affected third-party orch; 15.4h on our own local stack) and then dies naturally (connection break / trickle channel loss; in the local case the runner container was later OOM-killed, exit 137).
2. The client's stop returns HTTP 404 runner not found.
3. The orchestrator's slot stays capacity_used indefinitely; on the third-party orch, 1,058 subsequent paid reservations were accepted and charged against the dead slot before we stopped counting. Mid-run snapshot of the local stack showed the exact desync: orch /discovery reports used: 1 while the runner's own /stats reports "session": null.

Reproduction recipe (local, stock images): hold a funded continuous session against a capacity-1 runner, then SIGKILL the runner container mid-session. The orch retains the session, stop 404s, the slot never frees, and paid reserves keep being accepted. We have full orch + runner logs from both occurrences if useful.

The durability context makes the priority point: streams survive 11-16 hours reliably — the failure mode of long-lived paid sessions is exclusively the aftermath accounting, not the streaming.

[rickstaa]

Additional data from a third long soak (intended 48h, ran 23h on 4 orchestrators with held funded streams before the host rebooted):

Wedges reproduce from client-side crashes too. The previous soak's process was killed by a host reboot (no stops sent). Both flux-klein runners on one orchestrator came back as phantom-wedged: every reserve on both slots answered 503 no capacity available for runner for the remaining 21 hours. So the trigger set is now: natural stream death (this thread, earlier), runner container OOM (earlier), and ungraceful client death (this run) — all leave slots permanently held.

Scale of the paid-retry churn on a double-wedged orch: ~90-110 signed payments per 5 minutes from one client retrying at ~15s intervals, ~2.1e11 wei per attempt (the upfront session credit, about 4 seconds of runtime at the advertised 5.15e10 wei/s price, non-refundable per the known prepay semantics). Over the 21h window that is thousands of paid reservations and roughly $10/day in expected ticket value for zero delivered frames — noting the inverted incentive as severity context for prioritizing the stale-session reaping here.

Healthy-lane contrast: the two non-wedged orchestrators in the same run each held single funded streams ~13h straight (after a couple of early sub-30-minute drops), with per-second top-ups never missing. The streaming and payment paths remain solid; slot lifecycle after abnormal termination is the one broken piece.