Async execution swallows every pod failure; error_policy is accepted and never read

[brian-arnold]

Async execution swallows every pod failure; error_policy is accepted and never read

Affects: HEAD (f73f539f), and every rev we've run. Not a regression — it has always behaved this way.

Summary

A pod that fails under AsyncPipelineOrchestrator is recorded as FAILED in _status, logged at debug, and then execution continues. The run completes and reports success. There is no way for a caller to opt out of this: the async path has no error-policy parameter at all, and the one the orchestrator advertises is dead code.

We lost ~1 TB/probe of hot-tier reclamation for months to this. Two terminal nodes failed on every single run, every run reported success, and the failure was visible only to someone who thought to query _status directly.

Two separate defects

1. AsyncPipelineOrchestrator.error_policy is accepted, stored, and never used

# src/orcapod/pipeline/async_orchestrator.py:53
error_policy: Literal["continue", "fail_fast"] = "continue",
# :56
self._error_policy = error_policy   # never read again

self._error_policy has no other reference in the package. SyncPipelineOrchestrator does forward it (sync_orchestrator.py:94 → execute(error_policy=self._error_policy)), so the two orchestrators silently disagree on a documented parameter.

Passing error_policy="fail_fast" to the async orchestrator today does nothing at all, with no warning.

2. The async pod path has no error policy to forward to

Even once (1) is plumbed through, there is nothing on the receiving end. FunctionPod.async_execute catches unconditionally:

# src/orcapod/core/function_pod.py:558-565
except Exception as exc:
    logger.debug("Data processing failed, skipping: %s", exc, exc_info=True)
    obs.on_data_crash(pod_label, tag, data, exc)

Compare FunctionJobNode.execute (function_node.py:1287), which does honour fail_fast. The sync path can stop on failure; the async path cannot.

logger.debug also means that at any normal log level the only trace of a failed pod is the _status row.

Why this is worth fixing rather than working around

Callers can (and we now do) query _status for FAILED rows after run() and set their own exit code. But that is a workaround for the general case: the orchestrator knows a node failed, and every caller has to independently rediscover it. It also cannot express "stop now" — by the time the caller can look, the rest of the run has already happened.

The specific hazard is that a terminal node failing is invisible by construction. A mid-graph failure at least starves its downstream nodes of data, so something looks wrong. A leaf that fails produces no signal at all.

Suggested fix

1. Add error_policy to FunctionPod.async_execute and re-raise on fail_fast — mirroring FunctionJobNode.execute.
2. Forward self._error_policy from AsyncPipelineOrchestrator through async_execute, matching the sync orchestrator.
3. Raise the swallow-path log from debug to warning. A failed pod is not a debug-level event under either policy.

Optionally: have OrchestratorResult carry the set of failed (node, tag) pairs, so continue callers can act on failures without querying _status. That would let a caller distinguish "ran to completion" from "ran to completion, having failed 40 nodes."

Repro

Any pod that raises, run through AsyncPipelineOrchestrator, with error_policy="fail_fast". run() returns normally; _status shows FAILED.

---

Filed from downstream NPIPE-211 (orcapod-spikesorting).