chore(m11): offload-audit GPU memory probes for vast smoke runs - #35
Closed
zhenyulincs wants to merge 1 commit into
Closed
chore(m11): offload-audit GPU memory probes for vast smoke runs#35zhenyulincs wants to merge 1 commit into
zhenyulincs wants to merge 1 commit into
Conversation
Diagnostic instrumentation used to root-cause the ~12 GB whole-GPU residual during the M11 offload audit on 4x RTX 5090 (cu130): - memory_utils: report physical GPU id alongside the CVD-local index; add log_nontorch probe (whole-GPU used vs torch reserved delta) to attribute non-torch allocations phase by phase - megatron actor: log_nontorch around reload_process_groups and build_cpu_bucket_cache - megatron model: log_nontorch around forward_backward and optimizer.step in train_one_step - sglang_engine: _log_whole_gpu (nvidia-smi, PID-namespace-proof) before/after release/resume_memory_occupation; print instead of logger because engine actor processes have no logging handler - sglang.patch: strip a trailing space on a context line so the patch applies cleanly on the vast cu130 image Probe-only change: no behavior difference beyond added logging.
Author
|
Closing — wrong content: this PR carried the audit instrumentation probes, but the intended contribution is the vast-compute sync/debug script toolkit, which will be PR'd to rlops/rlix instead. The probe diff remains available locally (stash) if ever needed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Probe-only instrumentation from the M11 offload audit (4x RTX 5090, cu130 vast instance) — the diagnostics used to root-cause the ~12 GB whole-GPU residual (torch-hook tms pauses only the init-time 2.76 GB; optimizer state never offloads) that led to rlops/rlix#17 and rlops/rlix#31.
available_memory()now also reports the physical GPU id (CVD slice lookup) so per-GPU logs are unambiguous when several actors each see their GPU ascuda:0; newlog_nontorch(label)probe logswhole_used / torch reserved / non_torchso deltas between consecutive probes attribute non-torch allocations (NCCL, context, driver pools) phase by phase.log_nontorcharoundreload_process_groups(wake path) andbuild_cpu_bucket_cache.log_nontorcharoundforward_backwardandoptimizer.stepintrain_one_step._log_whole_gpu(label)— whole-GPU used vianvidia-smifor the engine's visible GPUs, before/afterrelease_memory_occupation/resume_memory_occupation. Deliberately whole-GPU (per-process attribution breaks under some container PID namespaces) and deliberatelyprint(engine actor processes have no logging handler; Ray forwards stdout unconditionally).What this is NOT
run_sync_sessionsupersession guard observed during the same audit is not included here — that fix is fix(miles): supersession guard in run_sync_session — sync latest when requested step is stale #34 (approved separately).Validation
python -m py_compileon all four touched Python files.