CachedFunctionPod cache hit ignores whether referenced output files still exist (dangling refs after intermediate deletion)

[brian-arnold]

Summary

CachedFunctionPod treats a cache hit as valid based solely on the input-data content hash, without checking that the output artifacts the cached record references (op.File / op.Directory paths) still exist on disk. When those artifacts are deleted out-of-band (e.g. a scratch/hot-tier reclaim, TTL cleanup, or a manual rm), the pod still returns the stale record instead of recomputing. Downstream pods then receive a dangling path reference and fail at runtime with a confusing "file/namespace not found" error, when the correct behavior would be to invalidate the hit and recompute.

The root reason: what flows between pods is a path reference (an op.File/engm:// string), not file bytes. Its content hash is stable whether or not the file exists, so deleting an intermediate is invisible to the lineage — nothing changes, so nothing triggers a rerun.

Where

orcapod/core/result_cache.py — CachedFunctionPod.lookup(). It matches on INPUT_DATA_HASH_COL and returns the stored output Data directly:

constraints = {constants.INPUT_DATA_HASH_COL: input_data.content_hash().to_prefixed_digest()}
result_table = self._result_database.get_records_with_column_value(...)
if result_table is None or result_table.num_rows == 0:
    return None
...
return Data(result_table, record_uuid=record_uuid,
            meta_info={self.RESULT_COMPUTED_FLAG: False})

There is no check that the paths inside result_table (for op.File/op.Directory columns) still resolve to existing artifacts.

Minimal repro

# Pod A writes a file and returns a REFERENCE to it
@function_pod
def preprocess(raw: Directory) -> {"ap_cache": File}:
    write_cache(cache_path)          # produces /cache/x
    return {"ap_cache": cache_path}

# Pod B consumes that reference
@function_pod
def motion(ap_cache: File) -> {"corrected": File}:
    rec = load(ap_cache)             # opens /cache/x

# Run 1: A computes, writes /cache/x, caches {input_hash -> ap_cache=/cache/x}
#        (B may or may not have completed)
# --- operator deletes /cache/x  (disk reclaim / TTL / manual rm) ---
# Run 2: A CACHE-HITS on input_hash -> returns /cache/x  (file NOT regenerated, no existence check)
#        B is a miss -> runs -> load('/cache/x') -> FileNotFoundError

Expected: In Run 2, A's cache hit is invalidated because /cache/x no longer exists → A recomputes → B succeeds.
Actual: A returns the dangling reference → B fails at runtime.

Real-world impact

We hit this in the spike-sorting pipeline. A hot-tier cache reclaim deleted the intermediate caches of 23 partially-completed probes (those that had passed preprocess_ap but not yet reached the terminal node). On the next run, preprocess_ap cache-hit and never regenerated the deleted binaries, so correct_motion/sort failed on missing inputs. There is currently no way to force just those probes to recompute short of DB surgery — the cache has no notion that the outputs are gone.

Proposed fix

Add an opt-in existence check to lookup():

• New flag, e.g. verify_output_exists: bool on NodeConfig / CachedFunctionPod (default False to preserve current performance).
• When enabled, after selecting the matching record and before returning it:
  1. Determine the output columns typed as op.File / op.Directory from the pod's output schema.
  2. For each referenced path, check existence via UPath(path).exists() — protocol-agnostic (works for local, engm://, s3://, …), no backend-specific code in core.
  3. If any referenced artifact is missing, log at INFO and return None (treat as a cache miss → recompute).

Sketch:

if self._verify_output_exists:
    for path in _file_reference_paths(result_table, self._output_schema):
        if not UPath(path).exists():
            logger.info("Cache hit for %s references missing artifact %s; recomputing.",
                        constraints[constants.INPUT_DATA_HASH_COL], path)
            return None

Design notes / trade-offs

• Perf: one exists() stat per file-reference column per cache hit. Existence-only (not re-hashing). For multi-root engm:// namespaces this is a probe per root until found — hence opt-in, and worth documenting.
• Partial outputs: if any referenced artifact is missing, invalidate the whole record (a pod can't partially recompute its outputs).
• Scope: this is existence-only. A file that still exists but was modified out-of-band is not detected — that's a separate (content-integrity) concern and intentionally out of scope here.
• Inline outputs: non-file columns are skipped.

Alternatives considered

• A runner-level skip_cache_lookup for a whole run — exists as a kwarg on process() but is not plumbed through executors, and it's coarse (forces recompute of everything in scope, including probes whose outputs are intact).
• Manual DB row deletion — no public per-record invalidation API; error-prone.

The existence-check approach is the smallest change that makes the cache correct in the presence of out-of-band artifact deletion, and it self-heals reclaims automatically.

[brian-arnold]

Tracked internally: NPIPE-203 — https://linear.app/metamorphic/issue/NPIPE-203/orcapod-cache-should-verify-referenced-output-files-exist-before