Skip to content

fix(bypass): honest CUDA-graph handling in the TraceLens-free trace analysis - #1047

Closed
zengleixin-amd wants to merge 8 commits into
mainfrom
feature/leixin/bypass-cudagraph-idle
Closed

fix(bypass): honest CUDA-graph handling in the TraceLens-free trace analysis#1047
zengleixin-amd wants to merge 8 commits into
mainfrom
feature/leixin/bypass-cudagraph-idle

Conversation

@zengleixin-amd

@zengleixin-amd zengleixin-amd commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Problem

On the bypass (TraceLens-free) trace-analysis route (HYPERLOOM_TRACE_ANALYSIS_ROUTE=bypass), CUDA-graph replay traces derailed the optimization loop. Root cause: the reader ignored cuda_runtime event names, so it never saw the graph-replay launches. Under graph replay the profiler under-records device kernels, so:

  • idle formula broke: busy = union(recorded kernels), total = event span → busy 63ms / total 7242ms → idle 99.1% on a near-saturated GPU.
  • attribution broke: graph-replayed kernels have no cpu_op link → ~91% of kernel time unlinked (no op names/shapes/sources).
  • no integrity check: the spurious 99% idle tripped the shared high-idle gate, cleared every kernel candidate, and emitted high_gpu_idle_pct — steering the run to "tune params, not kernels."

Approach: honest degradation, not reconstruction

The upstream trace genuinely lacks the data (graph replays carry no per-replay device/op info). It cannot be reconstructed downstream, so this change detects the incomplete trace and marks the affected metrics unreliable — it never fabricates values or adds fallbacks.

What changed (5 files, agents/kernel/tools/)

  • Detect graph replays by name in _bypass_trace_reader → attribution.graph_launch_count (canonical _GRAPH_LAUNCH_RE, reused by _trace_shape_manifest).
  • Don't derail: when graph replays are present and idle is high, skip the candidate-suppression gate and emit bypass_cudagraph_unreliable_idle instead of the misleading high_gpu_idle_pct (the shared _idle_gate / TraceLens route is untouched).
  • Honest degradation of idle (high idle, or degenerate empty timeline, under replay) and attribution (coverage below the shared floor under replay): idle_reliable / attribution_reliable / data_reliability_reason propagated through result (+ timeline/attribution sub-blocks), summary.json, trace_input_manifest.json, and analysis.md (idle/busy rendered —; accurate op-attribution coverage kept + flagged; a DATA RELIABILITY note). xDiT diffusion sidecar nulls busy/idle when unreliable.
  • --capture-folder: made its no-op explicit — it is not consumed by the core analysis (only the opt-in shape manifest) and does not reconstruct anything → bypass_capture_folder_not_used_for_core warning when provided.

Deliberate: idle_reliable keys off the idle threshold (under-recording manifests as high idle), so a plausible low idle from real recorded kernels stays trustworthy.

Verification

Before/after on an incident-shaped trace (128 replays, ~4410 kernels, ~91% unlinked, idle 99.1%), run on a real Linux MI300X pod:

before after
graph_launch_count missing 128
hot kernels 0 (cleared) 2 (retained)
routing warning high_gpu_idle_pct bypass_cudagraph_unreliable_idle + bypass_cudagraph_replay
idle in report 99.11% (authoritative) — + DATA RELIABILITY note; idle_reliable=False

Tests: bypass suite green locally and on the Linux pod (185 passed / 21 skipped on pod). Adds regression tests for detection, non-suppression, idle/attribution degradation, degenerate empty timeline, healthy-low-idle-cudagraph, and the capture-folder signal.

Not done (deliberate)

Accurate idle values and recovered op-attribution are not reconstructed — the data is absent upstream; reconstructing it would mean fabricating numbers. The true fix for that belongs in the upstream trace producer.

🤖 Generated with Claude Code

Zeng and others added 7 commits July 29, 2026 17:46
Zeng @claude
fix(bypass): detect CUDA-graph replay so idle gate cannot derail the …
9407d67 
…loop

The TraceLens-free bypass trace reader ignored cuda_runtime event names, so
it never saw CUDA-graph replay launches (cudaGraphLaunch/hipGraphLaunch).
Under graph replay the profiler under-records device kernels, so the union of
recorded kernel intervals covers only a fraction of the real GPU-busy time and
idle_pct reads spuriously high (~99%) even on a near-saturated GPU. The shared
high-idle gate then suppressed every hot-kernel candidate and emitted the
route-to-params high_gpu_idle_pct warning, steering the run away from kernel
optimization on a false "GPU is idle" signal.

Stop-the-bleed minimal set (does not fix idle numerics or replay attribution):

- A: _bypass_trace_reader counts graph-replay launches by name on cuda_runtime
  events and exposes attribution.graph_launch_count. The launch-name regex is
  now the reader's canonical _GRAPH_LAUNCH_RE; _trace_shape_manifest reuses it
  instead of keeping a duplicate.
- B: bypass_trace_analysis high-idle gate carve-out: when graph replays are
  detected the idle_pct is untrustworthy, so it keeps the candidate lists and
  emits bypass_cudagraph_unreliable_idle instead of the misleading
  high_gpu_idle_pct. The shared _idle_gate (also used by the TraceLens route)
  is untouched.
- C: _emit_quality_warnings surfaces bypass_cudagraph_replay whenever replays
  are detected (attribution/idle unreliable).

Adds a reproduction test asserting candidates are retained and the right
warnings fire on a cudagraph high-idle trace.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Zeng @claude
fix(bypass): address code-review findings on cudagraph detection
a338faa 
- Emit bypass_cudagraph_replay unconditionally (moved out of the
  kernels-gated _emit_quality_warnings), so a graph-replay trace with zero
  recorded device kernels — the most severe under-recording case — still
  surfaces the replay signal. Compute graph_launch_count once and reuse it in
  the idle-gate carve-out. Adds a zero-kernel regression test.
- Reduce _GRAPH_LAUNCH_RE to `graphlaunch|graph_launch` (the hip/cuda
  alternatives were dead: `graphlaunch` already matches as a substring).
- Restore the missing blank line before _trace_shape_manifest.classify_op
  (PEP8 E302).

Not changed (deferred, out of stop-the-bleed scope): the idle-gate carve-out
still keys off graph_launch_count>0 within the high-idle branch (a genuinely
host-idle cudagraph workload is not routed to params — needs replay-aware busy
time, D/E); count_graph_replays still scans device kernels (pre-existing).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Zeng @claude
fix(bypass): honest degradation for incomplete CUDA-graph traces (D/E)
4f36b78 
When the upstream trace is an incomplete CUDA-graph capture, the GPU-time and
op-attribution data simply are not present. Rather than reconstruct absent data
(impossible) or fabricate fallbacks, the bypass analysis now DETECTS the
incomplete trace and marks the affected metrics unreliable, propagating that so
nothing downstream treats a spurious idle_pct / zero attribution as authoritative.

- Reliability predicates (reuse existing signals, no new thresholds):
  * idle_reliable=False when idle_pct>gate AND graph replays present
    (a high idle reading under replay is an under-recording artifact).
  * attribution_reliable=False when graph replays present AND attributed_pct
    below the shared op-attribution floor. The floor is now a single
    _bypass_corr_warn_pct() used by both this gate and the
    bypass_low_op_correlation warning so they can never disagree.
  * data_reliability_reason="cudagraph_incomplete_trace" when either is false.
- Propagate idle_reliable / attribution_reliable / data_reliability_reason into
  the result (and result.timeline), summary.json, and trace_input_manifest.json.
- analysis.md: when unreliable, the idle/busy/op-attribution cells render as the
  em-dash placeholder (not a bogus number) and a DATA RELIABILITY provenance
  note is emitted. This also keeps any workload-summary parser from reading a
  spurious idle_pct downstream.

No data reconstruction, no capture-folder wiring, no fallback fabrication. The
idle-gate carve-out is unchanged in behavior (now expressed via idle_reliable).

Adds tests: idle marked unreliable + degraded rendering; attribution unreliable
on a zero-recorded-kernel replay trace; and a regression lock that a normal
(non-cudagraph) trace stays fully reliable with a numeric idle.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Zeng @claude
fix(bypass): make cudagraph_replay warning consistent with reliabilit…
981bae3 
…y flags

The unconditional bypass_cudagraph_replay message asserted idle_pct/attribution
"are unreliable" whenever replays were detected, which contradicts idle_reliable
=True computed for a cudagraph trace whose idle is below the gate. Reword it to
state only the mechanism (under-recording, no cpu_op link) and defer the
trustworthiness verdict to the idle_reliable / attribution_reliable flags, so
the general replay signal never contradicts the first-class reliability flags.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Zeng @claude
fix(bypass): resolve self-review findings on reliability degradation
7235ba4 
Adversarial self-review (correct base) surfaced consistency/coverage gaps in the
honest-degradation change. Fixes:

- Attribution coverage is no longer nulled in the Executive Summary. The low
  coverage % is accurate and is itself the evidence that op mapping is poor;
  nulling it hid that and contradicted the Appendix (which still printed the raw
  figure). Now the number is kept and the attribution_reliable flag +
  bypass_low_op_correlation warning + a reworded DATA RELIABILITY note carry the
  "op names/shapes unresolved" signal. Resolves the Exec-vs-Appendix contradiction.
- xDiT diffusion sidecar no longer derives gpu_busy_ratio from a spurious
  busy/idle: when idle is unreliable the timeline handed to build_report_from_bypass
  has busy_pct/idle_pct nulled (diffusion_roofline treats a missing busy_pct as
  gpu_busy_ratio=None, not a fabricated value).
- attribution_reliable is mirrored into result["attribution"] (symmetric with the
  idle_reliable mirror into result["timeline"]) so a consumer reading the
  sub-block directly sees the untrusted marker.
- The DATA RELIABILITY provenance note now describes each metric accurately
  (idle shown as '-'; attribution coverage shown but op mapping unresolved).

Tests: attribution-only degradation (idle kept numeric, coverage kept numeric +
flagged, mirror asserted); and a healthy low-idle cudagraph trace stays fully
reliable — locking the deliberate threshold semantics (CUDA graph alone does not
mark idle unreliable; only a high idle does).

Not changed (deliberate, per decision): idle_reliable keys off the idle
threshold, not off graph presence — under-recording manifests as high idle, so a
sub-threshold cudagraph idle is treated as trustworthy.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Zeng @claude
fix(bypass): honest degradation for --capture-folder (point #4)
4515dc0 
--capture-folder was a silent no-op for the core analysis: it is only consumed
by the opt-in TraceShapeManifest and never feeds idle/attribution/candidates,
so an operator could believe passing it fixed the CUDA-graph data gaps when it
did not. Rather than wire it up (there is no valid upstream capture data to
reconstruct from — the deliberate "no reconstruction" stance), make the no-op
explicit: when --capture-folder is provided, emit a
bypass_capture_folder_not_used_for_core health warning stating it is not used by
the core analysis (only the opt-in manifest) and does not reconstruct idle or
op-attribution. A run without the flag emits nothing new.

Adds a test for both the flag-present warning and the flag-absent silence.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Zeng @claude
fix(bypass): flag idle unreliable on degenerate empty CUDA-graph time…
0a27e31 
…line

The zero-recorded-kernel case (graph replays present, no device kernels) is the
most severe under-recording: total_time_ms computes to 0, so idle_pct fell out
as 0.0, which is below the gate threshold — leaving idle_reliable=True and the
report showing "GPU Idle % = 0.00%" as trustworthy even though nothing was
recorded. Extend idle_reliable to also trip when, under graph replay, the
timeline is degenerate (total_time_ms missing or <= 0). A plausible low/moderate
idle computed from real recorded kernels stays trustworthy (the deliberate
threshold semantics), so this only catches the empty-timeline artifact.

Adds the idle_reliable=False assertion to the zero-kernel regression test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@zengleixin-amd
zengleixin-amd requested a review from a team as a code owner July 30, 2026 02:15
@zengleixin-amd zengleixin-amd added the skip-e2e-test It's a PR that doesn't need to be e2e tested label Jul 30, 2026
Zeng @claude
ci: re-trigger checks (e2e skipped via skip-e2e-test label)
814cffc 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

skip-e2e-test It's a PR that doesn't need to be e2e tested

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@zengleixin-amd