Bound Seqera executor scheduler requests with a request timeout

[pditommaso]

Closes #7435.

Draft: blocked on a sched-client release. This bumps the pin from 0.73.2 to 0.78.0, which is not published yet — see the dependency section below.

What this fixes

The Task monitor thread can block indefinitely inside SchedClient.describeTask() on a stalled scheduler response. Only the connect phase was bounded (10s, set inside sched-client); nothing bounded the wait for the response body once the connection was established. Because TaskPollingMonitor polls all tasks from that single thread, one hung poll stalls checkAllTasks() for the entire session — no task in the run advances.

What changes

A new config option, applied to every request the executor's client issues:

seqera.executor.requestTimeout = '45 sec'   // default

It bounds a single attempt, not the whole call. A timed-out read raises HttpTimeoutException, which extends IOException — exactly what seqera.executor.retryPolicy (default 10 attempts, 450ms initial delay, exponential to 90s) exists to absorb. So a slow poll is retried rather than propagated into checkAllTasks(), where it would fail the task instead of being picked up on the next 10s cycle.

0 sec restores the previous unbounded behaviour.

Why task submission is not at risk

POST /v1a1/compute/tasks is not idempotent — the scheduler mints fresh task ids per delivery and has no idempotency key — so a re-sent batch submit would create a duplicate set of tasks whose ids SeqeraBatchSubmitter.flushBatch() never learns, and which nothing polls or cancels. sched-client 0.78.0 pins every non-idempotent request to a single attempt for exactly this reason (seqeralabs/sched#1033), with the server-side fix tracked in seqeralabs/sched#1031. A bounded submit therefore fails cleanly through onBatchSubmitFailure() rather than duplicating work.

One consequence worth knowing: a submit that fails with connection refused — a draining scheduler pod during a rolling deploy — is no longer retried either, because the client cannot distinguish "nothing was delivered" from "delivered and processed". That is lifted once the submit endpoints accept an idempotency key.

Dependency

sched-client 0.73.2 → 0.78.0, required for correctness rather than for the API: requestTimeout exists in 0.73.2, but setting it there silently forced maxAttempts(1) and discarded retryPolicy for every call, so a single timed-out poll would fail the task — the opposite of what this PR is for. The decoupling landed in seqeralabs/sched#937.

Note this plugin overrides lib-httpx to 2.4.0 while sched pins 2.2.0; the client behaviour relied on here (default retry condition throwable instanceof IOException, and the per-request timeout being re-applied on each attempt because the same HttpRequest is re-sent) is identical in both.

Ready to un-draft as soon as 0.78.0 is on the Seqera Maven repo.

Tests

ExecutorOptsTest — the 45s default and an explicit value round-trip through the config scope.
SeqeraExecutorTest — an explicit timeout reaches SchedClientConfig, and 0 sec maps to null (unbounded).

:plugins:nf-seqera:test for both classes — 52 tests, 0 failures.

🤖 Generated with Claude Code

[netlify]

✅ Deploy Preview for nextflow-docs canceled.

Name	Link
🔨 Latest commit	c557e93
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs/deploys/6a79e7961c0747000814a1c0