Keep the batching rewrites out of the post-optimization cleanup - #3227
Keep the batching rewrites out of the post-optimization cleanup#3227AsafManela wants to merge 1 commit into
Conversation
`compile_mlir!` finishes the `:all` pipeline with a cleanup stage that re-runs
the whole pattern set with transpose/reshape propagation reversed (down, up,
down). In that configuration three EnzymeXLA patterns rewrite each other in a
cycle on `convert(concat(reshape(a), reshape(b)))`:
convert_concat -> concat(convert(reshape(x_i)))
elementwise_reshape_like -> concat(reshape(convert(x_i)))
concat_insert_dim_elementwise -> convert(concat(reshape(x_i))) (start)
and each trip through the batching rewrite adds another unbatched wrapper
function, so the greedy driver never reaches a fixed point. Any graph with
fp32 master weights and bf16 compute hits this -- a rotary embedding's
`concat(reshape(...))` feeding a bf16 cast is enough -- and `@compile` then
runs for hours with no output. Individual primitives compile in seconds; two
or more transformer layers hang.
Batching is already applied by the main pipeline, so run the cleanup stage
with the concat/slice-to-batch groups disabled. A depth-2 bf16 Enzyme gradient
that did not finish in 45 minutes now compiles in 96 s, and a depth-4
hand-written backward pass in 7.5 s (was >25 min), with unchanged results.
The rewrite cycle itself is an Enzyme-JAX bug and is fixed there separately
(EnzymeAD/Enzyme-JAX: ConvertConcat declines the batchable shape); this keeps
the default pipeline out of it, and also covers `reshape_propagate=:down`
users on an older jll.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Closing this: I built the fix and it turns out this PR is unnecessary and not free. The root cause is the Enzyme-JAX rewrite cycle in #3225 / EnzymeAD/Enzyme-JAX#2983. I have So once the cycle is fixed at its source, nothing needs to be excluded from the The measurements quoted in the original description stand and now belong to the
One piece worth keeping: the regression test. Happy to open a small test-only PR adding |
Fixes #3225.
compile_mlir!finishes the:allpipeline with a cleanup stage that re-runs the wholepattern set with transpose/reshape propagation reversed (
down,up,down). In thatconfiguration three EnzymeXLA patterns rewrite each other in a cycle:
and each trip through the batching rewrite adds another unbatched wrapper function, so the
greedy driver never reaches a fixed point and the module grows without bound. Any graph with
fp32 master weights and bf16 compute hits this -- a rotary embedding's
concat(reshape(...))feeding a bf16 cast is enough -- and
@compilethen runs for hours with no output.Individual primitives compile in seconds; two or more transformer layers hang. The issue has
the full bisect and the gdb trace.
Batching has already been applied by the main pipeline before this stage, so run the cleanup
with the concat/slice-to-batch groups disabled.
Results
:allin 96.5 s, wasenzyme.autodiffin the module -- this is not anEnzyme problem): 7.5 s, was >25 min. Compiled-vs-eager bf16 loss difference 9.5e-7.
Tests
test/core/optimization_passes.jl: the cleanup pipeline strings must not contain thebatching patterns while the main pipeline still does, and the minimal
convert(concat(reshape))module must survivedown, up, down(10/10 locally). Note thatbefore this change the second testset does not fail, it hangs.
Relationship to the Enzyme-JAX fix
The rewrite cycle itself is an Enzyme-JAX bug, filed and fixed separately
(EnzymeAD/Enzyme-JAX#2983, EnzymeAD/Enzyme-JAX#2988:
ConvertConcatdeclines the batchableshape). This change is still worth having on its own -- it works on today's jll, and it also
covers anyone passing
reshape_propagate=:downto the main pipeline, where the same threepatterns meet.
Verified on Julia 1.12.7 /
Reactant_jllv0.0.405, CPU backend and CUDA (H100).🤖 Generated with Claude Code
https://claude.ai/code/session_012k6KZhhCNSCvZFKzErHaHH