[YouWanHee]

Problem

validate_lineage checks a completed cycle's required artifact roles by scanning artifacts[] once per revision (a next(a for a in artifacts if ...) inside the loop over artifact_revisions). The cost grows with the square of the number of files a cycle holds.

Measured on unmodified main c0119a79a with a synthetic completed cycle (one revision per artifact):

artifacts	validate_lineage
2,500	0.23 s
5,000	0.90 s
10,000	3.59 s
20,000	15.86 s

How it showed up

A code stage kept its gate evidence under the cycle's artifacts/evidence/gates/: 1.43 million files (wav, jsonl and json; 77 GB). The sealed manifest grew to 1.45 GB with about 700k artifacts, so by the growth above one validate_lineage call on it takes roughly five hours.

• After the owner exited, its dispatch-reap-watch drained the process group and then reached this check through the runtime settlement (materialize_after_terminal_close → settle_owner_completion → sealed-cycle verification). With every watched process already gone, the watcher stayed at 100% CPU for 25 hours (RSS up to 15.7 GB) until it was stopped by hand.
• The read-only dispatch_terminal_commit.py inspect for the same attempt goes through the same verification. Stack samples put it in validate_lineage at the per-revision next(...) scan.

Fix

Build one artifact_id -> role table first, keeping the first row per id as the scan did, and look each revision up in it. The check is now linear: 20,000 artifacts validate in 0.11 s and 320,000 in 3.2 s. Before the change, the old and new functions returned the same violations on 600 randomized documents, including repeated artifact ids, revisions whose artifact is missing, and both first_close values.

Tests

TestCompletedCycleRoles in utilities/artifact_manifest.test.py:

• Roles that have a revision complete the cycle.
• A repeated artifact id keeps its first row's role, which pins the old next(...) behavior.
• Reads of artifact rows stay bounded. For 2,002 artifacts the per-revision scan reads 2,019,017 rows (this test fails on main); the table reads 16,016, or 8 per row, against a limit of 20 per row.

Checks run on this branch (1e2e28812), on a host whose harness Python is conda CPython 3.13.11:

• utilities/artifact_manifest.test.py: 103 tests pass on CPython 3.13.11 and on the system CPython 3.12.3.
• The 18 suites that import artifact_manifest, run with tools/run-tests.py on this branch and then on unmodified main, one tree at a time: 17 pass on both. runtime_observation_cpu.test.py::test_isolated_production_observer_workload fails the same way on both trees (timed out waiting for all watchers pending on actual registered work after native idle), so it is a host issue and not caused by this change.
• The adaptation-boundary job's checks all pass: generate.py --check, render-landing.py --check, check-adaptation-boundary.sh, check-model-config.py, check-unit-config.py and check-scope-placeholders.py. So do check_destructive_calls.py tools/install and check-installed-layout-trigger.py (fired=none).
• The install-lifecycle suites: 14 of 15 pass. projection-completeness.test.sh fails on this host only because PyYAML is installed in the user site and the fixture's private HOME hides it (host.pyyaml). It fails the same way on unmodified main.

Not in this PR

The manifest still lists every file. A large directory becomes a single descriptor only when it is named raw-results, so a stage that writes many files anywhere else still gets a very large manifest; here that was 1.45 GB plus a snapshot of the same size. Even with this fix, loading and digesting a manifest of that size takes minutes and several GB of memory on each pass. Whether a large subtree should be bundled automatically is a separate design question, left for a follow-up.

🤖 Generated with Claude Code