Skip to content

Keep the batching rewrites out of the post-optimization cleanup - #3227

Closed
AsafManela wants to merge 1 commit into
EnzymeAD:mainfrom
AsafManela:fix-cleanup-pipeline-hang
Closed

Keep the batching rewrites out of the post-optimization cleanup#3227
AsafManela wants to merge 1 commit into
EnzymeAD:mainfrom
AsafManela:fix-cleanup-pipeline-hang

Conversation

@AsafManela

Copy link
Copy Markdown

Fixes #3225.

compile_mlir! finishes the :all pipeline with a cleanup stage that re-runs the whole
pattern set with transpose/reshape propagation reversed (down, up, down). In that
configuration three EnzymeXLA patterns rewrite each other in a cycle:

start:                            convert(concat(reshape(a), reshape(b)))
convert_concat                 -> concat(convert(reshape(a)), convert(reshape(b)))
elementwise_reshape_like       -> concat(reshape(convert(a)), reshape(convert(b)))
concat_insert_dim_elementwise  -> convert(concat(reshape(a), reshape(b)))   == start

and each trip through the batching rewrite adds another unbatched wrapper function, so the
greedy driver never reaches a fixed point and the module grows without bound. Any graph with
fp32 master weights and bf16 compute hits this -- a rotary embedding's concat(reshape(...))
feeding a bf16 cast is enough -- and @compile then runs for hours with no output.
Individual primitives compile in seconds; two or more transformer layers hang. The issue has
the full bisect and the gdb trace.

Batching has already been applied by the main pipeline before this stage, so run the cleanup
with the concat/slice-to-batch groups disabled.

Results

  • Depth-2 bf16 Enzyme gradient of a small GPT: compiles under :all in 96.5 s, was

    45 min with no output.

  • Depth-4 hand-written backward pass (no enzyme.autodiff in the module -- this is not an
    Enzyme problem): 7.5 s, was >25 min. Compiled-vs-eager bf16 loss difference 9.5e-7.
  • The pass-pipeline reproducer from the issue returns in 0.14 s.

Tests

test/core/optimization_passes.jl: the cleanup pipeline strings must not contain the
batching patterns while the main pipeline still does, and the minimal
convert(concat(reshape)) module must survive down, up, down (10/10 locally). Note that
before this change the second testset does not fail, it hangs.

Relationship to the Enzyme-JAX fix

The rewrite cycle itself is an Enzyme-JAX bug, filed and fixed separately
(EnzymeAD/Enzyme-JAX#2983, EnzymeAD/Enzyme-JAX#2988: ConvertConcat declines the batchable
shape). This change is still worth having on its own -- it works on today's jll, and it also
covers anyone passing reshape_propagate=:down to the main pipeline, where the same three
patterns meet.

Verified on Julia 1.12.7 / Reactant_jll v0.0.405, CPU backend and CUDA (H100).

🤖 Generated with Claude Code

https://claude.ai/code/session_012k6KZhhCNSCvZFKzErHaHH

`compile_mlir!` finishes the `:all` pipeline with a cleanup stage that re-runs
the whole pattern set with transpose/reshape propagation reversed (down, up,
down). In that configuration three EnzymeXLA patterns rewrite each other in a
cycle on `convert(concat(reshape(a), reshape(b)))`:

    convert_concat                -> concat(convert(reshape(x_i)))
    elementwise_reshape_like      -> concat(reshape(convert(x_i)))
    concat_insert_dim_elementwise -> convert(concat(reshape(x_i)))   (start)

and each trip through the batching rewrite adds another unbatched wrapper
function, so the greedy driver never reaches a fixed point. Any graph with
fp32 master weights and bf16 compute hits this -- a rotary embedding's
`concat(reshape(...))` feeding a bf16 cast is enough -- and `@compile` then
runs for hours with no output. Individual primitives compile in seconds; two
or more transformer layers hang.

Batching is already applied by the main pipeline, so run the cleanup stage
with the concat/slice-to-batch groups disabled. A depth-2 bf16 Enzyme gradient
that did not finish in 45 minutes now compiles in 96 s, and a depth-4
hand-written backward pass in 7.5 s (was >25 min), with unchanged results.

The rewrite cycle itself is an Enzyme-JAX bug and is fixed there separately
(EnzymeAD/Enzyme-JAX: ConvertConcat declines the batchable shape); this keeps
the default pipeline out of it, and also covers `reshape_propagate=:down`
users on an older jll.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@AsafManela

Copy link
Copy Markdown
Author

Closing this: I built the fix and it turns out this PR is unnecessary and not free.

The root cause is the Enzyme-JAX rewrite cycle in #3225 / EnzymeAD/Enzyme-JAX#2983. I have
now compiled that fix (EnzymeAD/Enzyme-JAX#2988) into a JLL and measured it against an
otherwise identical build. On a real optimization-stage module that hangs today, with the
full pattern set -- i.e. with the batching rewrites this PR disables still enabled:

without the Enzyme-JAX fix   HANG (killed at 300 s)
with the Enzyme-JAX fix      OK, 0.7 s

So once the cycle is fixed at its source, nothing needs to be excluded from the
post-optimization cleanup. This PR would permanently disable the concat/slice-to-batch
rewrites in that stage for every user, to work around a bug that no longer exists. Its
secondary justification -- that it also covers reshape_propagate=:down -- is moot for
the same reason: the cycle cannot form once ConvertConcat declines the batchable shape.

The measurements quoted in the original description stand and now belong to the
Enzyme-JAX fix instead. Two independent real-model confirmations, both bf16/fp8 training
graphs with a hand-written backward on a single H100:

  • depth 24, bf16: killed after 12 min without the fix, 164 s with it
  • depth 24, fp8: killed after 23 min without the fix, 215 s with it
  • depth 4, fp8: the stalling stage alone, >300 s vs 0.4 s

One piece worth keeping: the regression test. Happy to open a small test-only PR adding
test/core/optimization_passes.jl -- the convert(concat(reshape)) module driven through
the real cleanup pipeline, asserting it terminates -- once the JLL carries the Enzyme-JAX
fix. Say the word and I'll put that up.

@AsafManela AsafManela closed this Aug 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

:all pipeline never terminates on bf16-compute graphs: a three-pattern rewrite cycle in the post-optimization cleanup

1 participant