fix(fastsac): preserve final observations and stabilize actor

[wlgys8]

Summary

• Correct timeout bootstrap in the default single-collector async FastSAC path using pre-reset final observations rather than the next episode's reset observations.
• Publish owned, sparse final observations through ArrayEnvState.final_obs; expose non-consuming adapter access. No RNG backup/restore: noisy final-observation capture consumes the shared random stream normally.
• Use minimum-over-critic-heads Q aggregation in the actor objective, with an explanatory comment about optimistic-head errors and the underestimation tradeoff. Keep Gaussian means unclamped, matching the successful A4 ablation.
• Add lifecycle, ownership, RNG, adapter and replay/queue contract tests; document remaining transport risks and experimental limitations.

Validation

• Relevant local regression suite before clamp removal: 81 passed.
• After clamp removal: actor mean/gradient regression tests (tanh and non-tanh), collector and learner tests: 17 passed.
• Training-runtime final-observation and ArrayEnv tests: 16 passed.
• Formatting/lint/license hooks: prek run --all-files passed.
• Both ablations passed 1,500-iteration, 20-step-episode smoke tests, with nonzero final-observation events, then completed 200,000 iterations with exit code 0.
• CI's seven learner-mock failures are fixed by initializing the replay pointer and disabled final-event state in fixtures, without production fallbacks. All builtin fast tests: 94 passed.
• Broad local fast selection: 1,728 passed, 3 skipped, with two local environment failures (stale Unitree entry-point metadata and missing tomlkit); the original CI failures are no longer reproduced. Updated CI checks are pending.

Completed ablation experiments

Task: g1-wbt-dance; 2,048 environments; seed 1; single async collector; strict UTD approximately 4; gradient clipping disabled. CPU environment rollout uses Numba TBB; actor inference and learning use separate GPUs for the two concurrent runs.

Definitions:

• F1: actor Q aggregation mean → min.
• F2: hard clamp on pre-tanh actor mean.
• F3: timeout bootstrap from the true pre-reset final observation.

Both new runs use the same Env refactor baseline. A1 disables F1/F2; A4 disables F2 only. The PR now adopts A4's algorithm settings: Q min + final-observation bootstrap, without a hard mean clamp.

Configuration	Final return	Mean return after 10k	Fraction below 30 after 10k	Deep drops after 10k (>60%)
Historical baseline	45.52	40.99	8.84%	3
Historical F1	43.70	43.11	4.63%	2
Historical F1+F2	47.63	43.56	3.68%	1
Historical F1+F2+F3	49.62	46.22	2.89%	1, curriculum-concentration signature
A1: F3 only, Q mean, no clamp	48.71	44.50	5.94%	3
A4: F1+F3, Q min, no clamp	50.40	47.29	1.37%	0

Returns are training log-window statistics, not a separate held-out policy evaluation. Deep drops are grouped relative to a rolling 20-point median baseline; counts describe numerical depth, not a definitive causal classification.

A1 troughs occurred at approximately 78.9k, 91.4k and 119.2k iterations (returns -1.54, 2.60 and -2.51); recovery to 90% of the pre-drop baseline took approximately 8.3k, 2.0k and 7.3k iterations. Its mature-period Q peak was 17.35 and minimum episode length 19.45. A4 had a Q peak of 10.94 and minimum episode length 382.30. A1's deep drops did not require positive entropy, so a positive-entropy-only pathology detector would miss them.

Interpretation and limitations

• F3 alone did not prevent severe drops in this run. A4 supports retaining Q-min aggregation: +2.79 mature-period mean return, 4.58 percentage points fewer low-return windows, and no deep drops.
• A4 demonstrates that F1+F3 can complete this 200k run without a mean clamp. This does not establish that removing the clamp universally improves performance; the PR removes the clamp based on this evidence and avoids its zero-gradient region. A same-baseline clamp on/off experiment and additional seeds remain recommended for broader validation.
• Q min is a conservative SAC actor aggregation choice; the inspected Holosoma FastSAC implementation uses Q mean. Mean aggregation is not asserted to be a universal implementation bug.
• Historical runs differ in Env/RNG implementation and resource conditions, so comparisons against them are not strict single-variable causal tests.
• The recorded 200k runs used RNG backup/restore during capture. The current simplified implementation no longer restores RNG, so these runs are not an exact-seed validation of the latest random trajectory. After simplification, ArrayEnv/Manager/backend/final-observation regression tests: 52 passed; full 200k was not rerun.
• Only one seed was run. The two new runs shared CPU/memory resources without CPU/NUMA isolation. Async timing is not fully controlled.
• Mature-period window-average throughput was about 67.6k / 67.4k environment steps/s per run. A4's final window reached about 96.8k after A1 finished; this is not a controlled single-task performance benchmark.
• Final-observation timing is per capture invocation, whereas Env-step timing is per step; dividing these means does not give amortized overhead.

Remaining risks / scope

The independent final-event queue is removed. Sparse final payloads are attached to transition-ring slots and share publication/acknowledgement; collector owns one pending completed step on backpressure. Learner ingests transition + final rows together, including during warmup, without stamp joins or replay private-field access. Replay reclaims rows by live transition references and grows geometrically rather than assuming episode-length spacing.

Refactor validation: 105 builtin tests passed, including spawned producer/slow-consumer ownership, both wrap boundaries, sparse/main-ring backpressure, no duplicate step on retries, CPU/CUDA warmup ingest, replay growth/reuse and n-step bootstrap. Host and CUDA-IPC short-episode training both completed 300 iterations (64 envs, 20-step limit, learning_starts=100) with nonzero final-observation diagnostics. Hooks passed. Full 200k training was not rerun after transport refactoring.

Remaining scope: sync/multi-collector final-observation support is not implemented. Sparse replay can reach dense worst-case memory under every-step truncation; rare growth synchronizes CUDA. Diagnostic clipping limitations and single-seed causal limitations remain documented.

Details: final-observation contract, slot payload transport.

Public text omits internal addresses, accounts and absolute server/workspace paths.

[Copilot]

🔵 Needs a closer look

It makes a broad, high-risk change spanning an RL algorithm semantics shift (Q-min aggregation), a new cross-process sparse-payload transport, and replay-buffer/env-contract changes that warrant human validation.

1 open finding

• Docstring incorrectly says final observation restores RNG state · New

What changed in this PR

This PR hardens the FastSAC trainer in two ways. First, it introduces a backend-neutral final-observation contract: ArrayEnv captures pre-reset observations of time-limit-truncated rows (ArrayEnvState.final_obs, implemented by ManagerEnv), which the async single-collector path threads through a new sparse FinalObsStorage transport and replay-buffer terminal-row storage so timeout bootstrapping uses the true episode-ending observation instead of the next episode's reset observation. Second, it changes the actor objective to use min-over-critic-heads Q aggregation (keeping Gaussian means unclamped), and replaces the per-call metrics dict / CUDA-graph _own machinery with a WindowMetrics accumulator drained at logging boundaries.

Changes:

• Add FinalObs/_capture_final_obs env contract and wire sparse final-observation payloads through ring transport, replay buffer, collector, and learner (async single-collector + multi-shard merge).
• Refactor FastSAC env wrappers to return FastSacStepResult; make FastSacAgent.update() return None and accumulate WindowMetrics instead of a metrics dict.
• Switch actor Q aggregation from mean to min across critic heads; rename ring cursors to *_completed_count/*_issued_count; add design docs and extensive regression tests.

File	Description
motrix_env_core/​src/​.../​array/​env.py	Adds FinalObs, _capture_final_obs hook, and pre-reset capture in step lifecycle
motrix_env_core/​src/​.../​numba/​manager/​env.py	ManagerEnv final-obs capture via dedicated observe slot
motrix_env_core/​src/​.../​numba/​manager/​observations.py	ObsTerm docstring update (contains RNG-restore inaccuracy)
motrix_rl_builtin/​.../​fastsac/​wrap*.py	FastSacStepResult dataclass; np/torch wrappers return it; adds max_episode_steps
motrix_rl_builtin/​.../​fastsac/​buffer.py	Sparse terminal-row storage + final-obs bootstrap in sampling
motrix_rl_builtin/​.../​fastsac/​async_impl/​transport/​ring.py, ipc_ring.py	FinalObsStorage/FinalObsPayload; cursor renames; row-capacity backpressure
motrix_rl_builtin/​.../​fastsac/​async_impl/​collector.py, learner.py, worker.py, train.py	Plumb final payloads through collect/drain/ingest and store allocation
motrix_rl_builtin/​.../​fastsac/​agent.py, window_metrics.py	Q-min aggregation; remove _own/metrics dict; WindowMetrics
motrix_rl_builtin/​.../​fastsac/​sync/​train.py	Consume step result and window metrics
motrix_rl_builtin/​tests/​*, motrix_env_core/​tests/​*	New/updated regression tests for final-obs, transport, buffer, actor
wiki/​design/​*.md	New final-observation design doc + transport payload section

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.