Fix run hanging forever when a shutdown callback blocks during abort

[pditommaso]

Closes #7444

Problem

Session.abort() invoked shutdown0() before force-terminating the execution barriers:

shutdown0()                          // runs shutdown callbacks + notifyFlowComplete()
notifyError(null)
logObserver?.forceTermination()
executorFactory?.signalExecutors()
processesBarrier.forceTermination()  // the thing `main` is waiting on

Meanwhile main parks in Session.await() on processesBarrier.awaitCompletion(), which has no timeout. The only code that can release it sits behind the callback drain, so any blocking shutdown callback deadlocks the whole run — no Execution complete -- Goodbye, and no terminal status reported to any observer, since notifyFlowComplete() is inside the blocked shutdown0() too. The reporter observed a head job stuck like this for 2h38m.

A second unbounded wait exists in SimpleAgent.getResult(), which awaits a CountDownLatch that only the agent's own runner thread can count down. That runner also exits its loop on InterruptedException, after which every subsequent getValue() hangs forever. And there is a guaranteed self-deadlock: WorkflowStatsObserver builds its agent as .onError { err -> session.abort(err) }, so the runner thread itself can enter abort() → invokeOnComplete() → getWorkflowStats() → getValue(), enqueueing an event that only that same (no longer polling) thread could serve.

Changes

Session.abort() — force-terminate the barriers before running the shutdown callbacks. The relative order of shutdown0() → notifyError() → logObserver.forceTermination() is preserved, so observer event ordering is unchanged.

Session.shutdown0() — releasing the barrier early lets main race ahead into destroy(). With the #7349 CAS guard alone, the second caller returns immediately, which would mean destroy() closing the cache and stopping plugins while the callbacks are still running. Instead the second caller now awaits the in-flight run on a CountDownLatch — restoring exactly the ordering that exists today — bounded at 5 min, after which it logs a warning and proceeds. Normal aborts behave identically; only the pathological case degrades, from "hangs forever" to "warns and exits".

One knock-on: on the abort path the pool managers in destroy() now shut down concurrently with the callbacks rather than after. shutdownOrAbort(aborted=true) is shutdownNow() — non-blocking, and those pools are killed regardless on an abort.

SimpleAgent — bound getResult() at 1 min, returning the current state with a warning on expiry; short-circuit getValue()/getQuickValue() when the caller is the runner thread, fixing the self-deadlock properly rather than merely bounding it; name the runner thread agent-<StateType> so it is identifiable in a thread dump.

Testing

Two regression tests added. SessionTest > should release the await barrier when a shutdown callback is blocking was verified to fail against the original abort() ordering and pass with the fix. Full :nextflow:test suite passes.

Out of scope

Two other anomalies visible in the reporter's thread table are left alone: the AWSBatch-executor threads still submitting jobs 8s after the abort, and the Task submitter thread's unconditional while(true).

🤖 Generated with Claude Code

[netlify]

✅ Deploy Preview for nextflow-docs canceled.

Name	Link
🔨 Latest commit	654f764
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs/deploys/6a7a2bbebb7ddd0008929a47