Skip to content

transformers: bound an attention mask by its known And factor - #3012

Merged
kali merged 1 commit into
mainfrom
fix/pulse-attn-length-mask
Oct 13, 2026
Merged

kali merged 1 commit into
mainfrom
fix/pulse-attn-length-mask

Conversation

@kali

@kali kali commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator

A band mask ANDed with a padding mask built from a runtime length has no uniform_tdim, so ScaledMaskedSoftmax planted no ROI on the scores and the rel-pos attention could not be blockified for pulsing. input_roi now uses an upper bound of the mask: its uniform_tdim, or the known factors of an And, followed through axis ops and casts.

A band mask ANDed with a padding mask built from a runtime length has no
uniform_tdim, so ScaledMaskedSoftmax planted no ROI on the scores and the
rel-pos attention could not be blockified for pulsing. input_roi now uses an
upper bound of the mask: its uniform_tdim, or the known factors of an And,
followed through axis ops and casts.
@github-actions

Copy link
Copy Markdown

⚠️ Bench vs main — no speed regressions · 13 secondary regression(s)

Reference: 2026-10-13 morning nightly run (0d old) · full report → run

Speed — evaltime · prefill · decode

no inference-speed regressions

⚠️ 13 secondary regression(s)
Δ metric device main → PR
⚠️ +25.9% nemotron_3_5_asr_streaming_0_6b_f32f32_encoder_1s
load · cpu
apple-m1-max 3.44 s → 4.32 s
⚠️ +25.5% nemotron_3_5_asr_streaming_0_6b_f32f32_encoder_1s
load · metal
apple-m1-max 3.44 s → 4.32 s
⚠️ +25.0% nemotron_3_5_asr_streaming_0_6b_f32f32_encoder_1s
load · cpu_mt
apple-m1-max 3.45 s → 4.32 s
⚠️ +24.0% nemotron_3_5_asr_streaming_0_6b_f32f32_encoder_1s
load+optimize · metal
apple-m1-max 3.63 s → 4.5 s
⚠️ +23.1% nemotron_3_5_asr_streaming_0_6b_f32f32_encoder_1s
load+optimize · cpu
apple-m1-max 3.87 s → 4.76 s
⚠️ +23.0% nemotron_3_5_asr_streaming_0_6b_f32f32_encoder_1s
load · cuda
i9-11900kb_rtx-4060 4.73 s → 5.82 s
⚠️ +22.2% nemotron_3_5_asr_streaming_0_6b_f32f32_encoder_1s
load+optimize · cpu_mt
apple-m1-max 3.89 s → 4.75 s
⚠️ +19.9% nemotron_3_5_asr_streaming_0_6b_f32f32_encoder_1s
load+optimize · cuda
i9-11900kb_rtx-4060 5.47 s → 6.57 s
⚠️ +11.1% hey_snips_v1
load · 400ms
cortex-a7 63 ms → 70 ms
⚠️ +6.0% en_tdnn_lstm_bn_q7
RSS @ load · pulse_240ms
cortex-a53 25.1 MB → 26.7 MB
⚠️ +5.3% en_tdnn_8M
RSS @ ready · 2600ms
cortex-a55 63.8 MB → 67.2 MB
⚠️ +5.3% en_tdnn_8M
load · pulse_180ms
cortex-a55 982 ms → 1.03 s
⚠️ +5.3% en_tdnn_8M
load+optimize · pulse_180ms
cortex-a55 1.12 s → 1.18 s

Bench numbers come from shared CI hardware and do produce false positives: a flagged
row is not on its own evidence of a real regression, and a clean report is not proof
there is none. Contributors do not need to chase what is reported here — leave the
reading of it to the maintainers, who will say if something needs action.

@kali
kali merged commit 7e4fb46 into main Oct 13, 2026
78 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant