Skip to content

fix(asr): derive speech token compression ratio from encoder config - #420

Open
Urvi Tyagi (Urvity03) wants to merge 1 commit into
microsoft:mainfrom
Urvity03:fix/332-long-audio-sequence-limit
Open

fix(asr): derive speech token compression ratio from encoder config#420
Urvi Tyagi (Urvity03) wants to merge 1 commit into
microsoft:mainfrom
Urvity03:fix/332-long-audio-sequence-limit

Conversation

@Urvity03

Copy link
Copy Markdown

Summary

This PR addresses Issue #332 by deriving the ASR speech token compression ratio from the model's encoder configuration (acoustic_tokenizer_config.encoder_ratios) instead of relying solely on preprocessor_config.json.

When available, the processor now computes the effective compression ratio directly from the encoder configuration while preserving the previous behavior as a fallback for checkpoints that do not expose encoder ratios.

Motivation

Issue #332 reports that long audio can generate approximately 153k speech placeholder tokens, exceeding the model context limit (131072) and eventually leading to a tensor dimension mismatch during ASR inference.

From investigating the processor and model configuration, the placeholder count can become inconsistent if speech_tok_compress_ratio differs from the encoder's effective stride.

Deriving the ratio from the encoder configuration keeps placeholder generation aligned with the encoder output while maintaining backward compatibility through the existing fallback behavior.

Changes

  • Derive speech_tok_compress_ratio from acoustic_tokenizer_config.encoder_ratios when available.
  • Preserve the previous fallback behavior if encoder ratios are unavailable.
  • Add a regression test covering compression-ratio resolution.

Validation

  • Added a regression test (tests/test_vibevoice_asr_processor.py).
  • Regression test passes locally.
  • Verified that the derived compression ratio matches the encoder stride defined by the model configuration.

@Urvity03

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant