[longquanzheng]

Summary

• Preserve all 500 apply seeds × 1,000 commands and 250 engine seeds × 300 simulated seconds per ref. Split them into five apply shards (100 seeds each) and three engine shards (83, 83, 84). Every shard retains coverage assertions and adds all regression cases.
• Optimize test builds at level 1 while retaining debug assertions and overflow checks. Give compilation its own 25-minute limit, separate from the 50/55-minute simulation limits; reserve 90 minutes for setup, both phases, and artifact uploads.
• Stream successful seeds and command output live while retaining the full artifact log. Report wall time, base seed, and last completed seed; distinguish timeout, invariant failure, command failure, interruption, missing seed progress, and seeded-acceptance failure. An invariant failure remains a failure if shrinking later times out; legacy ambiguous baseline failures never count as timeouts. Build outputs and summary lines are separate from simulation results.
• Give head and the latest release separate random bases in disjoint seed ranges. Remove read-only regular-CI cache restores that cannot match the optimized Cargo profile. Run the Python runner integration tests in their own PR workflow and required check gate, independently of the nightly resolver.
• Use the workflow commit's harness for both refs. For older releases, add only the progress print to the successful test-loop branch before building; fail explicitly on unfamiliar layouts. Production code and deterministic trajectories are unchanged.

Why

Issue #925 reports nightly workloads exhausting their command limits. In run 40667618822, compilation consumed 8m58s/8m06s of the apply jobs' 50-minute allowance and 7m35s of main's 55-minute engine allowance. All three then exited 124 without a panic or assertion.

The workload had insufficient headroom for unoptimized simulation code, growing invariant checks, and seed/runner variability. The green October 25 manual run, 40231084500, used only 100 apply seeds and 64 engine seeds; it did not validate the scheduled 500/250 budgets. Standard engine on the unchanged v0.1.9 tag timed out on October 27 but completed its test in 2,158.58s on October 28. Apply main took 2,341.31s on October 25 and 2,448.05s on October 26, before both refs timed out on October 28. This does not establish one sudden deterministic code regression.

The #908 full-suite timing increase mentioned in #911 also compares different test inventories: #908 added the Step failure profile and pinned-command test, whereas nightly selects one test with --exact. Local sampling of the nightly apply workload showed broad unoptimized row parsing, iterator, copying, and protobuf work rather than a new single blocking operation. An equivalent 100-seed apply run measured 220.27s unoptimized and 42.67s optimized; these exploratory runs overlapped other local work, so the ratio is indicative rather than an isolated benchmark. Sharding plus optimized builds addresses the actual cost without removing checks or shortening seed trajectories.

Testing

Rust 1.97.1; local simulation runs use CARGO_PROFILE_DEV_OPT_LEVEL=1 CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=4.

Nightly workload	Parameters	Test time	Observed command wall time
Apply representative shard	base 2618013579; 100 seeds; 1,000 commands	42.67s	70.3s, including 27.6s rebuild
Standard engine representative shard	base 2990327231; 84 seeds; 300 seconds	114.80s	141.9s, including build-lock/rebuild time
Redeploy full nightly job	base 2752189851; 48 seeds; 300 seconds	104.63s	106.99s
Step failure full nightly job	base 646969045; 48 seeds; 300 seconds	90.57s	91.68s

All four retained and passed their coverage assertions. These are laptop measurements, not hosted-runner timing guarantees; the workflow dispatch must verify actual runner headroom.

• cargo test --locked -p dexzd-simulation: passed (17 library tests, 6 apply tests, 1 RNG contract, 5 engine tests), including pinned regression digests and repeated-run determinism.
• make server-zd-simulation-runner-test: 20 CLI integration tests passed after review fixes, covering every result classification, real baseline failure versus anchored acceptance timeout, invariant failure followed by timeout, child exit 124 versus supervisor timeout, signal-killed children, SIGINT/SIGTERM after build success, zero tests/missing progress, build-only output keys, live output before child exit, descendant termination and output draining, nested-session cleanup, intentional mutant failures, release instrumentation idempotence/rejection, and exact seed partitioning.
• Short apply shard through the revised runner: base 2618013579, 64 random seeds × 1,000 commands plus two regression seeds; all coverage assertions passed, 137,823 total commands, 29.12s test time and 29.7s command wall time. Build-only runner completed in 0.8s using the existing local optimized target cache.
• Progress instrumentation checked against actual server-zd/v0.1.9 test sources: passed.
• Fresh optimized seeded-acceptance build in an empty target directory, with four build jobs: 268.4s, inside the separate 1,500s build limit.
• cargo clippy --locked -p dexzd-simulation --all-targets -- -D warnings, cargo fmt --all --check, make rust-rules-check, make copyright-check, workflow actionlint, and git diff --check: passed.
• CARGO_PROFILE_DEV_OPT_LEVEL=1 CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=4 bash script/server-zd-seeded-bugs.sh: passed in 653.01s wall; all 7 seeded bugs detected, 0 gaps, all unpatched suites passed.
• No Docker Compose or cloud services used. Hosted validation completed with zero timeouts; see the results below.

Hosted workflow results before review fixes

The revised classification, PR CI, separate build outputs, and disjoint target ranges still need a new hosted nightly dispatch on the updated branch. The earlier run below demonstrates cold-build headroom and the preserved full workload; it predates the review fixes.

Default-input run 40727840173 completed on commit bbd3fba: 20 test jobs passed, two apply shards found the existing Attribute Sync bug #1012, and zero jobs timed out. The resolver also passed in 51 seconds. The overall workflow is red solely because of those two real invariant failures.

Among simulation jobs, the longest separate build phase was 740.4s against its 1,500s limit, and the longest simulation was 327.9s against its 3,300s limit. Successful apply shards completed 102 seeds each (100 random plus two regressions); engine shards completed 87/87/88 seeds (83/83/84 random plus four regressions each), preserving the full 250-seed random budget per ref. Both profile types completed all 48 seeds per ref. Both seeded-acceptance jobs detected all seven mutants with zero gaps.

Both failing apply shards found seed 4200525811 at command index 755: apply-error: apply invariant violated: synced Attribute "orders/2" has no live value. Each logged Result: invariant failure, base seed 4200525742, and last completed seed 4200525810; they exited 101 rather than 124. Branch simulation time was 50.4s; release simulation time was 98.6s. The failure artifacts retain the reproduction and shrinking output. This production bug is intentionally left to #1012.

Ref	Job	Result	Build seconds	Run seconds	Job wall time
branch	Apply simulation / shard 0	success	483.0	63.9	9m 50s
branch	Apply simulation / shard 1	success	702.8	116.3	14m 16s
branch	Apply simulation / shard 2	success	463.2	73.6	9m 43s
branch	Apply simulation / shard 3	success	451.8	60.9	9m 13s
branch	Apply simulation / shard 4	invariant failure	420.0	50.4	8m 55s
branch	Engine simulation / shard 0	success	691.6	327.9	17m 31s
branch	Engine simulation / shard 1	success	686.3	322.3	17m 26s
branch	Engine simulation / shard 2	success	670.9	321.3	17m 15s
branch	Engine simulation Step failure profile	success	592.2	185.1	13m 34s
branch	Engine simulation redeploy profile	success	677.3	182.4	14m 56s
branch	Seeded bug acceptance	success	—	1325.4	22m 42s
v0.1.9	Apply simulation / shard 0	success	452.7	62.4	9m 21s
v0.1.9	Apply simulation / shard 1	success	656.1	120.5	13m 33s
v0.1.9	Apply simulation / shard 2	success	625.5	103.0	12m 54s
v0.1.9	Apply simulation / shard 3	success	717.6	117.8	14m 43s
v0.1.9	Apply simulation / shard 4	invariant failure	723.2	98.6	14m 34s
v0.1.9	Engine simulation / shard 0	success	570.8	317.6	15m 32s
v0.1.9	Engine simulation / shard 1	success	675.4	322.9	17m 12s
v0.1.9	Engine simulation / shard 2	success	689.0	325.5	17m 48s
v0.1.9	Engine simulation Step failure profile	success	740.4	180.6	15m 58s
v0.1.9	Engine simulation redeploy profile	success	538.4	143.8	12m 25s
v0.1.9	Seeded bug acceptance	success	—	1039.7	17m 58s

Seeded-acceptance run seconds include its fresh baseline build, per-mutant builds, and all acceptance suites. Job wall times also include setup and uploads.

Existing invariant failures found during investigation

These remain outside this timeout fix. Reproductions use current origin/main production and simulation logic, with only the new progress output in tests.

Historical run / job	Reproduction	Finding on current main
40297458739 / Apply latest-release	seed 161503589; 312 commands	Still fails at index 311: apply-error: apply invariant violated: synced Attribute "orders/1" has no live value
40478868158 / Apply main	seed 879068896; 266 commands	Still fails at index 265: apply-error: apply invariant violated: synced Attribute "status" has no live value
40177164849 / Engine latest-release	seed 1798558341; 300 simulated seconds	Passes now; historical failure at 249927000 µs was node-cannot-recover: 40 starts failed, ending in a transient object-storage get failure (the recovery retry fix in #944 is already on main)

Fixes #925

🤖 Generated with Claude Code