Skip to content

chore(m11): offload-audit GPU memory probes for vast smoke runs - #35

Closed
zhenyulincs wants to merge 1 commit into
zhenyu/m11-mvp-testfrom
zhenyu/m11-offload-audit-probes
Closed

chore(m11): offload-audit GPU memory probes for vast smoke runs#35
zhenyulincs wants to merge 1 commit into
zhenyu/m11-mvp-testfrom
zhenyu/m11-offload-audit-probes

Conversation

@zhenyulincs

Copy link
Copy Markdown

What

Probe-only instrumentation from the M11 offload audit (4x RTX 5090, cu130 vast instance) — the diagnostics used to root-cause the ~12 GB whole-GPU residual (torch-hook tms pauses only the init-time 2.76 GB; optimizer state never offloads) that led to rlops/rlix#17 and rlops/rlix#31.

  • memory_utils: available_memory() now also reports the physical GPU id (CVD slice lookup) so per-GPU logs are unambiguous when several actors each see their GPU as cuda:0; new log_nontorch(label) probe logs whole_used / torch reserved / non_torch so deltas between consecutive probes attribute non-torch allocations (NCCL, context, driver pools) phase by phase.
  • megatron actor: log_nontorch around reload_process_groups (wake path) and build_cpu_bucket_cache.
  • megatron model: log_nontorch around forward_backward and optimizer.step in train_one_step.
  • sglang_engine: _log_whole_gpu(label) — whole-GPU used via nvidia-smi for the engine's visible GPUs, before/after release_memory_occupation / resume_memory_occupation. Deliberately whole-GPU (per-process attribution breaks under some container PID namespaces) and deliberately print (engine actor processes have no logging handler; Ray forwards stdout unconditionally).
  • docker/patch/latest/sglang.patch: strip a trailing space on one context line — without it the patch fails to apply on the vast cu130 image.

What this is NOT

Validation

Diagnostic instrumentation used to root-cause the ~12 GB whole-GPU
residual during the M11 offload audit on 4x RTX 5090 (cu130):

- memory_utils: report physical GPU id alongside the CVD-local index;
  add log_nontorch probe (whole-GPU used vs torch reserved delta) to
  attribute non-torch allocations phase by phase
- megatron actor: log_nontorch around reload_process_groups and
  build_cpu_bucket_cache
- megatron model: log_nontorch around forward_backward and
  optimizer.step in train_one_step
- sglang_engine: _log_whole_gpu (nvidia-smi, PID-namespace-proof)
  before/after release/resume_memory_occupation; print instead of
  logger because engine actor processes have no logging handler
- sglang.patch: strip a trailing space on a context line so the patch
  applies cleanly on the vast cu130 image

Probe-only change: no behavior difference beyond added logging.
@zhenyulincs

Copy link
Copy Markdown
Author

Closing — wrong content: this PR carried the audit instrumentation probes, but the intended contribution is the vast-compute sync/debug script toolkit, which will be PR'd to rlops/rlix instead. The probe diff remains available locally (stash) if ever needed.

@zhenyulincs zhenyulincs closed this Aug 3, 2026
@zhenyulincs
zhenyulincs deleted the zhenyu/m11-offload-audit-probes branch August 3, 2026 06:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant