OmniVoice supports multilingual TTS, voice cloning, voice design, non-verbal tag tokens, long-form chunking, and chunked pseudo-streaming output.
| Route | Task | Mode | Inputs |
|---|---|---|---|
| Auto voice | tts |
offline, streaming |
--text |
| Voice clone | tts, clon |
offline, streaming |
--text, --voice-ref, optional --reference-text |
| Voice design | tts, vdes |
offline, streaming |
--text, --instruct |
Use a local OmniVoice model package:
models/OmniVoice/
The package should contain the OmniVoice model weights and the audio-tokenizer files expected by model_specs/omnivoice.json. GGUF packages can also be used when they embed the package spec.
Voice clone:
audiocpp_cli --task tts --family omnivoice --model /path/to/OmniVoice --backend cuda --text "Hello from OmniVoice." --voice-ref assets/resources/b.wav --reference-text "Some call me nature. Others call me Mother Nature. I've been here for over 4.5 billion years. 22,500 times longer than you." --out out.wavVoice design:
audiocpp_cli --task tts --family omnivoice --model /path/to/OmniVoice --backend cuda --text "Hello from OmniVoice." --instruct "female, young adult, moderate pitch" --out out.wavAuto voice:
audiocpp_cli --task tts --family omnivoice --model /path/to/OmniVoice --backend cuda --text "Hello from OmniVoice." --out out.wavclon and vdes are aliases to the same synthesis implementation. Route
selection does not add or remove input requirements.
OmniVoice upstream Python does not expose model-native streaming. audio.cpp exposes chunked pseudo streaming: the session emits one audio event per generated text chunk, then returns a merged final WAV.
audiocpp_cli --task tts --mode streaming --family omnivoice --model /path/to/OmniVoice --backend cuda --text "Hello from OmniVoice. This text can be split into chunk events." --voice-ref assets/resources/b.wav --reference-text "Some call me nature. Others call me Mother Nature. I've been here for over 4.5 billion years. 22,500 times longer than you." --text-chunk-size 160 --out stream.wav --out-dir stream_chunks--out writes the final merged WAV. --out-dir writes emitted chunk WAVs such as chunk_0.wav, chunk_1.wav, and so on.
Configure OmniVoice with mode: "streaming":
{
"host": "127.0.0.1",
"port": 8080,
"backend": "cuda",
"device": 0,
"threads": 8,
"models": [
{
"id": "omnivoice-stream",
"family": "omnivoice",
"path": "/path/to/OmniVoice",
"task": "tts",
"mode": "streaming"
}
]
}SSE request:
curl -N http://127.0.0.1:8080/v1/audio/speech \
-H "Content-Type: application/json" \
-H "Accept: text/event-stream" \
-d '{
"model": "omnivoice-stream",
"input": "Hello from OmniVoice streaming.",
"response_format": "pcm",
"stream_format": "sse",
"voice": "assets/resources/b.wav",
"reference_text": "Some call me nature. Others call me Mother Nature. I have been here for over four billion years.",
"options": {
"text_chunk_size": 160,
"text_chunk_mode": "tag_aware"
}
}'The SSE stream emits speech.audio.delta events followed by speech.audio.done.
| Option | Values | Default | Meaning |
|---|---|---|---|
--voice-ref |
WAV path | not set | Reference speaker audio for voice cloning. |
--reference-text |
text | empty string | Transcript for the reference audio. |
--instruct |
text | empty string | Voice-design instruction. |
--language |
language hint | auto | Optional language hint. |
--text-chunk-size |
integer chars | disabled | Enables framework text chunking and controls pseudo-streaming chunk size. |
--text-chunk-mode |
default, tag_aware, japanese, endline |
tag_aware |
Framework text chunking mode when --text-chunk-size is set. |
--num-inference-steps |
integer | 32 |
Decoder diffusion steps. |
--guidance-scale |
float | 2.0 |
Decoder CFG strength. |
--seed |
unsigned 32-bit integer | random session RNG | Generation seed; omitted requests continue the session RNG state. |
The corresponding request-option names are reference_text, instruct,
language, text_chunk_size, text_chunk_mode, num_inference_steps,
guidance_scale, and seed.
| Option | Values | Default | Meaning |
|---|---|---|---|
speed |
float | 1.0 |
Speech speed multiplier. |
duration_sec |
seconds | estimated | Optional target output duration. |
shift |
float | 0.1 |
Diffusion timestep schedule shift. |
denoise |
bool | true |
Add the denoise control token when reference audio is present. |
preprocess_prompt |
bool | true |
Preprocess reference audio before tokenization. |
postprocess_output |
bool | true |
Apply output postprocessing and trimming. |
layer_penalty_factor |
float | 5.0 |
Layer-order penalty for masked audio-codebook position selection. |
position_temperature |
float | 5.0 |
Gumbel temperature for masked-position selection. |
class_temperature |
nonnegative float | 0.0 |
Audio-token class sampling temperature; 0 selects classes greedily. |
audio_chunk_duration_sec |
seconds | 15.0 |
Model-side automatic chunk duration when framework chunking is not explicitly enabled. |
audio_chunk_threshold_sec |
seconds | 30.0 |
Estimated audio length threshold before model-side chunking is used. |
Legacy request names duration, t_shift, audio_chunk_duration, and
audio_chunk_threshold remain accepted, respectively. Use one spelling per
option. Both spellings work with existing GGUFs; no reconversion is required.
Arena and weight-context sizes are in MiB.
| Option | Values | Default | Meaning |
|---|---|---|---|
omnivoice.mem_saver |
bool | false |
Release staged runtime graphs after request phases; default false. Cannot be combined with perf_mode=flash_attention. |
omnivoice.perf_mode |
off, flash_attention |
off |
Generator performance mode; default off keeps the exact-safe attention path. flash_attention requires mem_saver=false. |
omnivoice.packed_cfg |
bool | true |
Give every guidance (unconditional) sequence only its target frames instead of the full prompt length, and pack batched requests without padding. false restores the padded layout. Ignored with mem_saver=true, whose layerwise graph always uses the padded layout. |
omnivoice.device_scoring |
bool | true |
For greedy class decoding (class_temperature=0), pick each step's best tokens on the backend and read back only those instead of the full logits. Ignored with mem_saver=true and for sampled class decoding. |
omnivoice.packed_projections |
bool | true |
Load the q/k/v projections and the gate/up projections as one packed weight each, so every layer runs one GEMM per group (its shared input is converted to f16 once instead of three and two times) and one fused SwiGLU kernel. false keeps separate projections. |
omnivoice.weight_type |
native, f32, f16, bf16, q8_0 |
native |
Generator weight storage fallback. |
omnivoice.generator_weight_type |
native, f32, f16, bf16, q8_0 |
inherits weight_type |
Explicit generator weight storage override. |
omnivoice.audio_tokenizer_weight_type |
native, f32, f16, bf16, q8_0 |
native |
Runtime storage type for audio tokenizer weights. |
omnivoice.audio_tokenizer_codebook_weight_type |
native, f32, f16, bf16 |
f32 |
Storage type for the audio tokenizer codebooks, codebook score matrix, semantic feature projection, and fc2. These ignore audio_tokenizer_weight_type. native keeps the packaged type and is rejected when that type is quantized, as in the Q8_0 GGUF. |
omnivoice.audio_tokenizer_graph_arena_mb |
positive integer | 128 |
Audio tokenizer graph arena. |
omnivoice.generator_prefill_graph_arena_mb |
positive integer | 256 |
Generator prefill graph arena. |
omnivoice.generator_decode_graph_arena_mb |
positive integer | 256 |
Generator decode graph arena. |
omnivoice.audio_tokenizer_weight_context_mb |
positive integer | 128 |
Audio tokenizer weight tensor context. |
omnivoice.generator_weight_context_mb |
positive integer | 256 |
Generator weight tensor context. |
omnivoice.exec_plan |
JSON object | {} |
Component execution plans. The only component is generator; see the RKNN section. |
omnivoice.perf_mode=flash_attention is only available on the normal graph path and cannot be combined with omnivoice.mem_saver=true.
omnivoice.packed_cfg, omnivoice.device_scoring and
omnivoice.packed_projections change only floating-point rounding:
- With f32 generator weights the first-step logits of the packed layout match the padded one to within 0.002 (mean).
- Device scoring picks the same tokens as host scoring.
- Packed projections are bit-identical with f32 generator weights on the standard path; with f16 weights the larger GEMMs can round differently.
To reproduce the output of earlier versions exactly, set all three to
false. With mem_saver=true, packed_cfg and device_scoring are ignored,
so only packed_projections=false is needed there.
perf_mode=flash_attention passes the 8 K/V heads to the kernel as they are
(grouped-query attention) instead of repeating them to 16 heads.
The generator is the expensive half of this family: the diffusion-LM step graph runs once per
inference step (num_inference_steps, default 32) over the whole sequence, and on an RK3588 it
is ~99% of synthesis. omnivoice.exec_plan voices that one stage through librknnrt and leaves
the text frontend, the audio tokenizer decode and the postprocessing on the host.
RKNN has no runtime graph builder, so each engine is compiled offline for one fixed total-token length (the conditional row: style + text + reference + target frames). Build them on an x86_64 host from the same checkpoint the GGUF was converted from:
python3 tools/rknn/prepare_omnivoice_rknn_generator.py \
--model-dir /path/to/OmniVoice \
--seq-len 256 --seq-len 384 --seq-len 512 --seq-len 768 --seq-len 1024 \
--output-dir models/OmniVoice-RKNNThen point the session at the ladder (order does not matter; the smallest engine that fits is
used, and a request longer than the largest engine runs the CPU graph). The ladder travels as
the generator component's main artifact, comma-separated - see
component execution plans:
audiocpp_cli --task tts --family omnivoice --model models/OmniVoice-GGUF/omnivoice-q8_0.gguf \
--backend cpu --session-option 'omnivoice.exec_plan={"generator":{"impl":"rknn","artifacts":{"main":"models/OmniVoice-RKNN/omnivoice_generator_seq256_fp16.rknn,models/OmniVoice-RKNN/omnivoice_generator_seq1024_fp16.rknn"}}}' \
--text "..." --out out.wavBuild with -DENGINE_ENABLE_RKNN=ON (plus ENGINE_RKNN_LIBRARY / ENGINE_RKNN_INCLUDE_DIR
pointing at the board's librknnrt); a plan selecting impl: rknn throws at load in a build
without it rather than falling back silently. Each engine carries the fp16 weight set and asks
the driver for its whole residency at init, so one rung is resident at a time
(AUDIOCPP_RKNN_APERTURE_MIB). Requests pad up to the bucket; the padding is exact (measured
398-401 dB padded-vs-exact on the fp32 graph), and the step costs about S^2 in attention, so
the bucket ladder is a time trade.
A ladder built with the producer's --front-k 28 (omnivoice_generator_front28_seq*.rknn) is
the front-shard contract: the engine runs the embeddings and all 28 backbone layers on the NPU
and returns the raw hidden state, and the CFG tail - final RMS norm, the 8 audio heads, the
guidance combine - runs on the host in fp32. This is the mode that makes
--guidance-scale > 0 usable on the NPU: the full ladder's in-graph fp16 heads put the
unconditional logits at 26.9 dB SNR (past the decoding knife-edge, so guidance renders noise),
while the same fp16 backbone leaves the hidden state at 67 dB and the fp32 tail returns the
guided logits to ~84 dB. It costs about 0.4 s/step of CPU tail time on top of the ~8.4 s/step
NPU pass, is validated at guidance 2.0 (board gate whisper-clean, same reference and seed), and
runs at batch 1 only (a batched shard request falls back to the CPU graph). Length accounting
includes the reference audio: a 14 s reference alone costs about 393 prompt tokens, so the
short rungs (seq256/384) only engage with a short reference clip; anything above a rung's
capacity runs the CPU graph.
Offline OmniVoice sessions implement native batching
(IBatchedOfflineVoiceTaskSession::run_batch), so the server's
/v1/tasks/batch runs several
requests in one generator pass per decoding step instead of one after
another. The model must be configured with mode=offline.
- Requests may differ in text, length, reference voice, guidance scale and step count. They are packed without padding and ordered by length internally; results keep the request order.
- Each request has its own seeded RNG: with
seedset, a request in a batch follows the same sampling path asrun()with that seed. Batched kernels can still round differently, so audio is identical torun()only when the kernels happen to match (most requests with f32 weights, many with f16). - Requests above
audio_chunk_threshold_secare split into chunks as inrun(); chunk k of every request is batched together. - Audio decoding and post-processing still run per request.
RTX 4090, generator_weight_type=f16, audio_tokenizer_weight_type=f16, 16
steps, voice clone (tests/omnivoice/omnivoice_batch_test):
| Workload | Before (packed_cfg=false, device_scoring=false) |
Default | perf_mode=flash_attention |
|---|---|---|---|
| 1 request, 9 words | 0.281 s | 0.147 s | 0.138 s |
8 x run(), 9 words |
2.228 s | 1.180 s | 1.096 s |
run_batch() of 8, 9 words |
1.995 s | 0.995 s | 0.824 s |
8 x run(), 5-20 words |
2.822 s | 1.569 s | 1.362 s |
run_batch() of 8, 5-20 words |
4.327 s | 1.442 s | 1.226 s |
Tesla T4, same settings, total GPU kernel time (nsys) before -> after the
un-repeated flash K/V heads and packed_projections:
| Workload | Before | After |
|---|---|---|
1 request, 20 words, perf_mode=flash_attention |
2.02 s | 1.77 s |
run_batch() of 8, 5-20 words, standard attention |
12.73 s | 12.17 s |
The T4 is already saturated by one request, so run_batch() is not faster
than sequential run() there.
The two weight_type options quantize SafeTensors weights at load time, which is not the same as loading a prebuilt Q8 GGUF package. On CUDA, omnivoice.generator_weight_type=f16 measured 1.26x faster than native, while runtime omnivoice.audio_tokenizer_weight_type=q8_0 produced unusable audio without being faster. See the OmniVoice weight-type benchmark.
For throughput on an RTX 4090 with several instances and batch sizes, in f16 and bf16, compared with the Python OmniVoice package (with and without FlashInfer), see the OmniVoice concurrency benchmark.
VoiceTut-TTS is an
OmniVoice fine-tune for Egyptian Arabic and Arabic-English code-switching. It
has the same architecture and tensor layout as OmniVoice and reuses the
OmniVoice audio tokenizer unchanged, so it runs through the omnivoice family
with every route and option above. It was trained with the arz (Egyptian
Arabic) language id, so pass --language arz. Model and integration by
@Mohammedaly22.
| Package | Format | Notes |
|---|---|---|
voicetut_tts_q8_0 |
GGUF Q8_0 | Standalone GGUF with the audio tokenizer embedded. |
voicetut_tts_f16 |
GGUF F16 | Closest to the original weights. |
Voice clone with Egyptian Arabic and English code-switching:
audiocpp_model_manager install voicetut_tts_q8_0 --models-root models
audiocpp_cli --task tts --family omnivoice --model models/VoiceTut-TTS-GGUF --backend cuda --language arz --text "عندي meeting بكرة الصبح، فهحاول أخلص الشغل بدري النهارده." --voice-ref speaker.wav --reference-text "<transcript of speaker.wav>" --out out.wavThe upstream repository ships 17 reference speakers in reference_speakers/
with transcripts in references.json. Copy them into a server voice_dir
(one <Name>.wav per speaker plus a prompt_text file with
<Name>|<transcript> lines) to use them as named voices, for example
"voice": "Mohamed". See the server voice library.
To build the GGUF packages, place the VoiceTut weights and the OmniVoice
audio_tokenizer/ directory in one model root, then convert:
audiocpp_gguf --input weights=models/VoiceTut-TTS/model.safetensors --input audio_tokenizer_weights=models/VoiceTut-TTS/audio_tokenizer/model.safetensors --root models/VoiceTut-TTS --output models/VoiceTut-TTS-GGUF/voicetut-tts-q8_0.gguf --family omnivoice --type q8_0 --overwriteNon-verbal tags are written directly in --text. Supported spellings include:
[laughter] [sigh] [confirmation-en] [question-en] [question-ah] [question-oh]
[question-ei] [question-yi] [surprise-ah] [surprise-oh] [surprise-wa]
[surprise-yo] [dissatisfaction-hnn]