Skip to content

Latest commit

 

History

History
317 lines (250 loc) · 17.5 KB

omnivoice.md

File metadata and controls

317 lines (250 loc) · 17.5 KB

OmniVoice

OmniVoice supports multilingual TTS, voice cloning, voice design, non-verbal tag tokens, long-form chunking, and chunked pseudo-streaming output.

Route Task Mode Inputs
Auto voice tts offline, streaming --text
Voice clone tts, clon offline, streaming --text, --voice-ref, optional --reference-text
Voice design tts, vdes offline, streaming --text, --instruct

Model Layout

Use a local OmniVoice model package:

models/OmniVoice/

The package should contain the OmniVoice model weights and the audio-tokenizer files expected by model_specs/omnivoice.json. GGUF packages can also be used when they embed the package spec.

Offline CLI

Voice clone:

audiocpp_cli --task tts --family omnivoice --model /path/to/OmniVoice --backend cuda --text "Hello from OmniVoice." --voice-ref assets/resources/b.wav --reference-text "Some call me nature. Others call me Mother Nature. I've been here for over 4.5 billion years. 22,500 times longer than you." --out out.wav

Voice design:

audiocpp_cli --task tts --family omnivoice --model /path/to/OmniVoice --backend cuda --text "Hello from OmniVoice." --instruct "female, young adult, moderate pitch" --out out.wav

Auto voice:

audiocpp_cli --task tts --family omnivoice --model /path/to/OmniVoice --backend cuda --text "Hello from OmniVoice." --out out.wav

clon and vdes are aliases to the same synthesis implementation. Route selection does not add or remove input requirements.

Streaming CLI

OmniVoice upstream Python does not expose model-native streaming. audio.cpp exposes chunked pseudo streaming: the session emits one audio event per generated text chunk, then returns a merged final WAV.

audiocpp_cli --task tts --mode streaming --family omnivoice --model /path/to/OmniVoice --backend cuda --text "Hello from OmniVoice. This text can be split into chunk events." --voice-ref assets/resources/b.wav --reference-text "Some call me nature. Others call me Mother Nature. I've been here for over 4.5 billion years. 22,500 times longer than you." --text-chunk-size 160 --out stream.wav --out-dir stream_chunks

--out writes the final merged WAV. --out-dir writes emitted chunk WAVs such as chunk_0.wav, chunk_1.wav, and so on.

Server Streaming

Configure OmniVoice with mode: "streaming":

{
  "host": "127.0.0.1",
  "port": 8080,
  "backend": "cuda",
  "device": 0,
  "threads": 8,
  "models": [
    {
      "id": "omnivoice-stream",
      "family": "omnivoice",
      "path": "/path/to/OmniVoice",
      "task": "tts",
      "mode": "streaming"
    }
  ]
}

SSE request:

curl -N http://127.0.0.1:8080/v1/audio/speech \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -d '{
    "model": "omnivoice-stream",
    "input": "Hello from OmniVoice streaming.",
    "response_format": "pcm",
    "stream_format": "sse",
    "voice": "assets/resources/b.wav",
    "reference_text": "Some call me nature. Others call me Mother Nature. I have been here for over four billion years.",
    "options": {
      "text_chunk_size": 160,
      "text_chunk_mode": "tag_aware"
    }
  }'

The SSE stream emits speech.audio.delta events followed by speech.audio.done.

Common Options (use directly)

Option Values Default Meaning
--voice-ref WAV path not set Reference speaker audio for voice cloning.
--reference-text text empty string Transcript for the reference audio.
--instruct text empty string Voice-design instruction.
--language language hint auto Optional language hint.
--text-chunk-size integer chars disabled Enables framework text chunking and controls pseudo-streaming chunk size.
--text-chunk-mode default, tag_aware, japanese, endline tag_aware Framework text chunking mode when --text-chunk-size is set.
--num-inference-steps integer 32 Decoder diffusion steps.
--guidance-scale float 2.0 Decoder CFG strength.
--seed unsigned 32-bit integer random session RNG Generation seed; omitted requests continue the session RNG state.

The corresponding request-option names are reference_text, instruct, language, text_chunk_size, text_chunk_mode, num_inference_steps, guidance_scale, and seed.

Request Options (use with --request-option)

Option Values Default Meaning
speed float 1.0 Speech speed multiplier.
duration_sec seconds estimated Optional target output duration.
shift float 0.1 Diffusion timestep schedule shift.
denoise bool true Add the denoise control token when reference audio is present.
preprocess_prompt bool true Preprocess reference audio before tokenization.
postprocess_output bool true Apply output postprocessing and trimming.
layer_penalty_factor float 5.0 Layer-order penalty for masked audio-codebook position selection.
position_temperature float 5.0 Gumbel temperature for masked-position selection.
class_temperature nonnegative float 0.0 Audio-token class sampling temperature; 0 selects classes greedily.
audio_chunk_duration_sec seconds 15.0 Model-side automatic chunk duration when framework chunking is not explicitly enabled.
audio_chunk_threshold_sec seconds 30.0 Estimated audio length threshold before model-side chunking is used.

Legacy request names duration, t_shift, audio_chunk_duration, and audio_chunk_threshold remain accepted, respectively. Use one spelling per option. Both spellings work with existing GGUFs; no reconversion is required.

Session Options (use with --session-option)

Arena and weight-context sizes are in MiB.

Option Values Default Meaning
omnivoice.mem_saver bool false Release staged runtime graphs after request phases; default false. Cannot be combined with perf_mode=flash_attention.
omnivoice.perf_mode off, flash_attention off Generator performance mode; default off keeps the exact-safe attention path. flash_attention requires mem_saver=false.
omnivoice.packed_cfg bool true Give every guidance (unconditional) sequence only its target frames instead of the full prompt length, and pack batched requests without padding. false restores the padded layout. Ignored with mem_saver=true, whose layerwise graph always uses the padded layout.
omnivoice.device_scoring bool true For greedy class decoding (class_temperature=0), pick each step's best tokens on the backend and read back only those instead of the full logits. Ignored with mem_saver=true and for sampled class decoding.
omnivoice.packed_projections bool true Load the q/k/v projections and the gate/up projections as one packed weight each, so every layer runs one GEMM per group (its shared input is converted to f16 once instead of three and two times) and one fused SwiGLU kernel. false keeps separate projections.
omnivoice.weight_type native, f32, f16, bf16, q8_0 native Generator weight storage fallback.
omnivoice.generator_weight_type native, f32, f16, bf16, q8_0 inherits weight_type Explicit generator weight storage override.
omnivoice.audio_tokenizer_weight_type native, f32, f16, bf16, q8_0 native Runtime storage type for audio tokenizer weights.
omnivoice.audio_tokenizer_codebook_weight_type native, f32, f16, bf16 f32 Storage type for the audio tokenizer codebooks, codebook score matrix, semantic feature projection, and fc2. These ignore audio_tokenizer_weight_type. native keeps the packaged type and is rejected when that type is quantized, as in the Q8_0 GGUF.
omnivoice.audio_tokenizer_graph_arena_mb positive integer 128 Audio tokenizer graph arena.
omnivoice.generator_prefill_graph_arena_mb positive integer 256 Generator prefill graph arena.
omnivoice.generator_decode_graph_arena_mb positive integer 256 Generator decode graph arena.
omnivoice.audio_tokenizer_weight_context_mb positive integer 128 Audio tokenizer weight tensor context.
omnivoice.generator_weight_context_mb positive integer 256 Generator weight tensor context.
omnivoice.exec_plan JSON object {} Component execution plans. The only component is generator; see the RKNN section.

omnivoice.perf_mode=flash_attention is only available on the normal graph path and cannot be combined with omnivoice.mem_saver=true.

omnivoice.packed_cfg, omnivoice.device_scoring and omnivoice.packed_projections change only floating-point rounding:

  • With f32 generator weights the first-step logits of the packed layout match the padded one to within 0.002 (mean).
  • Device scoring picks the same tokens as host scoring.
  • Packed projections are bit-identical with f32 generator weights on the standard path; with f16 weights the larger GEMMs can round differently.

To reproduce the output of earlier versions exactly, set all three to false. With mem_saver=true, packed_cfg and device_scoring are ignored, so only packed_projections=false is needed there.

perf_mode=flash_attention passes the 8 K/V heads to the kernel as they are (grouped-query attention) instead of repeating them to 16 heads.

Rockchip NPU (RKNN) generator

The generator is the expensive half of this family: the diffusion-LM step graph runs once per inference step (num_inference_steps, default 32) over the whole sequence, and on an RK3588 it is ~99% of synthesis. omnivoice.exec_plan voices that one stage through librknnrt and leaves the text frontend, the audio tokenizer decode and the postprocessing on the host.

RKNN has no runtime graph builder, so each engine is compiled offline for one fixed total-token length (the conditional row: style + text + reference + target frames). Build them on an x86_64 host from the same checkpoint the GGUF was converted from:

python3 tools/rknn/prepare_omnivoice_rknn_generator.py \
    --model-dir /path/to/OmniVoice \
    --seq-len 256 --seq-len 384 --seq-len 512 --seq-len 768 --seq-len 1024 \
    --output-dir models/OmniVoice-RKNN

Then point the session at the ladder (order does not matter; the smallest engine that fits is used, and a request longer than the largest engine runs the CPU graph). The ladder travels as the generator component's main artifact, comma-separated - see component execution plans:

audiocpp_cli --task tts --family omnivoice --model models/OmniVoice-GGUF/omnivoice-q8_0.gguf \
    --backend cpu --session-option 'omnivoice.exec_plan={"generator":{"impl":"rknn","artifacts":{"main":"models/OmniVoice-RKNN/omnivoice_generator_seq256_fp16.rknn,models/OmniVoice-RKNN/omnivoice_generator_seq1024_fp16.rknn"}}}' \
    --text "..." --out out.wav

Build with -DENGINE_ENABLE_RKNN=ON (plus ENGINE_RKNN_LIBRARY / ENGINE_RKNN_INCLUDE_DIR pointing at the board's librknnrt); a plan selecting impl: rknn throws at load in a build without it rather than falling back silently. Each engine carries the fp16 weight set and asks the driver for its whole residency at init, so one rung is resident at a time (AUDIOCPP_RKNN_APERTURE_MIB). Requests pad up to the bucket; the padding is exact (measured 398-401 dB padded-vs-exact on the fp32 graph), and the step costs about S^2 in attention, so the bucket ladder is a time trade.

A ladder built with the producer's --front-k 28 (omnivoice_generator_front28_seq*.rknn) is the front-shard contract: the engine runs the embeddings and all 28 backbone layers on the NPU and returns the raw hidden state, and the CFG tail - final RMS norm, the 8 audio heads, the guidance combine - runs on the host in fp32. This is the mode that makes --guidance-scale > 0 usable on the NPU: the full ladder's in-graph fp16 heads put the unconditional logits at 26.9 dB SNR (past the decoding knife-edge, so guidance renders noise), while the same fp16 backbone leaves the hidden state at 67 dB and the fp32 tail returns the guided logits to ~84 dB. It costs about 0.4 s/step of CPU tail time on top of the ~8.4 s/step NPU pass, is validated at guidance 2.0 (board gate whisper-clean, same reference and seed), and runs at batch 1 only (a batched shard request falls back to the CPU graph). Length accounting includes the reference audio: a 14 s reference alone costs about 393 prompt tokens, so the short rungs (seq256/384) only engage with a short reference clip; anything above a rung's capacity runs the CPU graph.

Native Batching

Offline OmniVoice sessions implement native batching (IBatchedOfflineVoiceTaskSession::run_batch), so the server's /v1/tasks/batch runs several requests in one generator pass per decoding step instead of one after another. The model must be configured with mode=offline.

  • Requests may differ in text, length, reference voice, guidance scale and step count. They are packed without padding and ordered by length internally; results keep the request order.
  • Each request has its own seeded RNG: with seed set, a request in a batch follows the same sampling path as run() with that seed. Batched kernels can still round differently, so audio is identical to run() only when the kernels happen to match (most requests with f32 weights, many with f16).
  • Requests above audio_chunk_threshold_sec are split into chunks as in run(); chunk k of every request is batched together.
  • Audio decoding and post-processing still run per request.

RTX 4090, generator_weight_type=f16, audio_tokenizer_weight_type=f16, 16 steps, voice clone (tests/omnivoice/omnivoice_batch_test):

Workload Before (packed_cfg=false, device_scoring=false) Default perf_mode=flash_attention
1 request, 9 words 0.281 s 0.147 s 0.138 s
8 x run(), 9 words 2.228 s 1.180 s 1.096 s
run_batch() of 8, 9 words 1.995 s 0.995 s 0.824 s
8 x run(), 5-20 words 2.822 s 1.569 s 1.362 s
run_batch() of 8, 5-20 words 4.327 s 1.442 s 1.226 s

Tesla T4, same settings, total GPU kernel time (nsys) before -> after the un-repeated flash K/V heads and packed_projections:

Workload Before After
1 request, 20 words, perf_mode=flash_attention 2.02 s 1.77 s
run_batch() of 8, 5-20 words, standard attention 12.73 s 12.17 s

The T4 is already saturated by one request, so run_batch() is not faster than sequential run() there.

The two weight_type options quantize SafeTensors weights at load time, which is not the same as loading a prebuilt Q8 GGUF package. On CUDA, omnivoice.generator_weight_type=f16 measured 1.26x faster than native, while runtime omnivoice.audio_tokenizer_weight_type=q8_0 produced unusable audio without being faster. See the OmniVoice weight-type benchmark.

For throughput on an RTX 4090 with several instances and batch sizes, in f16 and bf16, compared with the Python OmniVoice package (with and without FlashInfer), see the OmniVoice concurrency benchmark.

VoiceTut-TTS (Egyptian Arabic)

VoiceTut-TTS is an OmniVoice fine-tune for Egyptian Arabic and Arabic-English code-switching. It has the same architecture and tensor layout as OmniVoice and reuses the OmniVoice audio tokenizer unchanged, so it runs through the omnivoice family with every route and option above. It was trained with the arz (Egyptian Arabic) language id, so pass --language arz. Model and integration by @Mohammedaly22.

Package Format Notes
voicetut_tts_q8_0 GGUF Q8_0 Standalone GGUF with the audio tokenizer embedded.
voicetut_tts_f16 GGUF F16 Closest to the original weights.

Voice clone with Egyptian Arabic and English code-switching:

audiocpp_model_manager install voicetut_tts_q8_0 --models-root models
audiocpp_cli --task tts --family omnivoice --model models/VoiceTut-TTS-GGUF --backend cuda --language arz --text "عندي meeting بكرة الصبح، فهحاول أخلص الشغل بدري النهارده." --voice-ref speaker.wav --reference-text "<transcript of speaker.wav>" --out out.wav

The upstream repository ships 17 reference speakers in reference_speakers/ with transcripts in references.json. Copy them into a server voice_dir (one <Name>.wav per speaker plus a prompt_text file with <Name>|<transcript> lines) to use them as named voices, for example "voice": "Mohamed". See the server voice library.

To build the GGUF packages, place the VoiceTut weights and the OmniVoice audio_tokenizer/ directory in one model root, then convert:

audiocpp_gguf --input weights=models/VoiceTut-TTS/model.safetensors   --input audio_tokenizer_weights=models/VoiceTut-TTS/audio_tokenizer/model.safetensors   --root models/VoiceTut-TTS --output models/VoiceTut-TTS-GGUF/voicetut-tts-q8_0.gguf   --family omnivoice --type q8_0 --overwrite

Tags

Non-verbal tags are written directly in --text. Supported spellings include:

[laughter] [sigh] [confirmation-en] [question-en] [question-ah] [question-oh]
[question-ei] [question-yi] [surprise-ah] [surprise-oh] [surprise-wa]
[surprise-yo] [dissatisfaction-hnn]