Skip to content

feat(speechlm2): add DFlash and DFlash2 training for SALM - #16113

Draft
Slyne wants to merge 7 commits into
NVIDIA-NeMo:mainfrom
Slyne:slyne/feat/salm-dflash
Draft

feat(speechlm2): add DFlash and DFlash2 training for SALM#16113
Slyne wants to merge 7 commits into
NVIDIA-NeMo:mainfrom
Slyne:slyne/feat/salm-dflash

Conversation

@Slyne

@Slyne Slyne commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add SALMDFlashModule to train an Automodel DFlash or DFlash2 draft against a frozen, audio-conditioned SALM target
  • preserve SALM audio-placeholder expansion and export draft-only checkpoints
  • select the implementation with dflash.variant and ship DFlash2 defaults matching the Automodel model recipe
  • bound anchor allocation for both variants; DFlash supports fused CE, while DFlash2 uses dense logits for its path selector
  • keep distributed skips rank-symmetric, normalize backbone and selector losses independently across the full DP group, and synchronize expert-parallel target execution
  • match serving hidden-state semantics for final-layer target taps
  • document both variants and add focused unit/regression coverage

Stacked dependency

This draft is stacked on NVIDIA-NeMo/Automodel#3673. Until that PR merges, nemo-automodel is pinned immutably to the implementation commit 426711b8336d563c37c7ec4a9b97b1bb6dd86aca on the author fork.

Before marking this PR ready, replace the fork pin with the upstream Automodel merge commit.

Scope and limitations

  • DFlash and DFlash2 training currently support BSHD layout with packed sequences disabled
  • SALM draft training requires tensor, pipeline, and context parallel sizes of 1; the frozen target may still use expert parallelism/HSDP
  • DFlash2 requires use_fused_linear_ce=false because its selector consumes full draft logits
  • public NVFP4 inference checkpoints cannot directly initialize the BF16 linear draft used for training
  • a GPU training smoke test remains required before this draft is marked ready

Validation

  • 59 focused SpeechLM2 tests passed
  • 47 focused Automodel DFlash2 tests passed
  • Black, Ruff, uv lock --check, and git diff --check passed
  • independent review agent: no actionable findings
  • Claude Code Opus with maximum reasoning, repeated after fixes: no actionable findings

@copy-pr-bot

copy-pr-bot Bot commented Aug 18, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.


salm_dflash._synchronize_ep_group_before_target_forward(None)
salm_dflash._synchronize_ep_group_before_target_forward(
_FakeMoEMesh(SimpleNamespace(size=lambda: 1, get_group=lambda: object()))
Slyne added 5 commits August 25, 2026 14:01
Signed-off-by: SlyneD <slyned@nvidia.com>
Signed-off-by: SlyneD <slyned@nvidia.com>
Signed-off-by: SlyneD <slyned@nvidia.com>
Signed-off-by: SlyneD <slyned@nvidia.com>
Signed-off-by: SlyneD <slyned@nvidia.com>
@Slyne
Slyne force-pushed the slyne/feat/salm-dflash branch from 1eee2ed to 54e087b Compare August 25, 2026 21:35
Signed-off-by: SlyneD <slyned@nvidia.com>
@Slyne Slyne changed the title feat(speechlm2): add DFlash training for SALM feat(speechlm2): add DFlash and DFlash2 training for SALM Aug 26, 2026
Signed-off-by: SlyneD <slyned@nvidia.com>
@Slyne
Slyne force-pushed the slyne/feat/salm-dflash branch from a9ced5e to 0cf301f Compare August 26, 2026 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants