fix(asr): derive speech token compression ratio from encoder config - #420
Open
Urvi Tyagi (Urvity03) wants to merge 1 commit into
Open
fix(asr): derive speech token compression ratio from encoder config#420Urvi Tyagi (Urvity03) wants to merge 1 commit into
Urvi Tyagi (Urvity03) wants to merge 1 commit into
Conversation
Author
|
@microsoft-github-policy-service agree |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR addresses Issue #332 by deriving the ASR speech token compression ratio from the model's encoder configuration (
acoustic_tokenizer_config.encoder_ratios) instead of relying solely onpreprocessor_config.json.When available, the processor now computes the effective compression ratio directly from the encoder configuration while preserving the previous behavior as a fallback for checkpoints that do not expose encoder ratios.
Motivation
Issue #332 reports that long audio can generate approximately 153k speech placeholder tokens, exceeding the model context limit (131072) and eventually leading to a tensor dimension mismatch during ASR inference.
From investigating the processor and model configuration, the placeholder count can become inconsistent if
speech_tok_compress_ratiodiffers from the encoder's effective stride.Deriving the ratio from the encoder configuration keeps placeholder generation aligned with the encoder output while maintaining backward compatibility through the existing fallback behavior.
Changes
speech_tok_compress_ratiofromacoustic_tokenizer_config.encoder_ratioswhen available.Validation
tests/test_vibevoice_asr_processor.py).