Skip to content

Add native NEUTRINO v3 ONNX renderer - #2322

Open
xiawow wants to merge 1 commit into
openutau:masterfrom
xiawow:neutrino-v3-pr
Open

Add native NEUTRINO v3 ONNX renderer#2322
xiawow wants to merge 1 commit into
openutau:masterfrom
xiawow:neutrino-v3-pr

Conversation

@xiawow

@xiawow xiawow commented Aug 25, 2026

Copy link
Copy Markdown

Summary

This PR adds native, in-process support for NEUTRINO v3 singers to OpenUtau.

Previous PRs (#2136 #2232 ) have explored NEUTRINO support using the official NEUTRINO executable and a background server. This PR addresses the same use case with a different architecture: the NEUTRINO v3 models run directly in-process through OpenUtau's existing ONNX Runtime integration.

OpenUtau loads the voicebank models directly through its existing ONNX Runtime integration. It does not require the official NEUTRINO executable, a background server, a subprocess wrapper, or a Python environment. t.bin provides phoneme timing, s.bin generates the acoustic features, and v.bin generates the waveform. p.bin is used when loading natural pitch and when applying StyleShift.

Main features

  • The phonemizer runs t.bin for each continuous phrase and shows its predicted phoneme boundaries in OpenUtau's phoneme panel instead of displaying placeholder timing.
  • Manual phoneme timing edits override the corresponding model boundary during rendering.
  • Audio rendering uses OpenUtau's final editable pitch, including PITD, as the F0 supplied to the acoustic model.
  • Load Rendered Pitch runs p.bin on demand and writes the predicted natural pitch to PITD, where it can be inspected, edited, flattened, or regenerated.
  • SHFC exposes NEUTRINO's StyleShift behavior. The direct pitch shift is removed from the p.bin result, so only the StyleShift-dependent contour is applied and the intended note pitch is preserved.

This keeps pitch prediction separate from the F0 supplied to acoustic inference: p.bin remains an optional source for natural pitch, while normal rendering follows the pitch that is visible and editable in OpenUtau.

Built-in dictionary

NEUTRINO v3 Japanese voicebanks use a fixed phoneme vocabulary and kana mapping, so the same dictionary does not need to be copied into every voicebank directory.

  • OpenUtau.Core/Neutrino/NeutrinoJapaneseDictionary.cs contains the kana-to-phoneme table and katakana normalization.
  • OpenUtau.Core/Neutrino/NeutrinoPhoneme.cs contains the v3 phoneme IDs, aliases, and romaji mappings.
  • OpenUtau.Core/Neutrino/NeutrinoPhonemizer.cs uses these built-in mappings and loads t.bin timing into the phoneme panel.

No dictionary is read from the singer or model directory.

HTS phoneme integration

NEUTRINO consumes an HTS-style phoneme sequence rather than one opaque lyric token per note. The integration therefore:

  • Keeps each phoneme associated with its parent note and its position inside that note.
  • Distinguishes model-generated boundaries from boundaries explicitly edited in the phoneme panel.
  • Allows phrase-initial consonants to use the real available silence before a note without crossing a preceding note or phrase.
  • Treats +, +~, and +* as extension notes that continue the previous sustain phoneme with the extension note's own pitch and duration.
  • Handles pause and breath tokens as separate inference regions.

The small RenderPhrase additions expose the parent-note index, original/edited phoneme position, and available leading context required for these behaviors. The same timing rules are shared by the phonemizer and renderer.

Code overview

  • NeutrinoSinger.cs registers NEUTRINO singers and manages the four model sessions.
  • NeutrinoRenderer.cs implements native timing, acoustic inference, waveform generation, rendered-pitch loading, caching, and SHFC StyleShift.
  • NeutrinoPhonemizer.cs, NeutrinoPhoneme.cs, and NeutrinoJapaneseDictionary.cs implement the Japanese HTS phoneme frontend and model timing display.
  • NeutrinoInferenceUtil.cs contains the shared score, extension-note, timing, pause/breath, and frame-mapping logic.
  • Singer loading, renderer registration, installation, and singer-type selection are connected through the existing OpenUtau interfaces.

For an official NEUTRINO voicebank, only add the character.txt and character.yaml metadata required by OpenUtau and select neutrino as the singer type.

@rokujyushi

Copy link
Copy Markdown
Contributor

I checked the latest license, and use without going through the NEUTRINO system (exe) is prohibited. This native operation may be in violation of the terms.

@xiawow

xiawow commented Aug 25, 2026

Copy link
Copy Markdown
Author

I checked the latest license, and use without going through the NEUTRINO system (exe) is prohibited. This native operation may be in violation of the terms.

Thanks for pointing this out. I actually checked the NEUTRINO EULA before starting this work.

Development of this renderer began around June 2026, which can also be seen from the commit history in my fork. At that time, the applicable EULA was the [October 24, 2024 version](https://studio-neutrino.com/downloads/manual/EULA_NEUTRINO.pdf). While it prohibited decompilation and reverse engineering in general, it explicitly included the following exception:

逆コンパイル、リバース・エンジニアリング(NEUTRINOでの音楽製作を補助するようなツールの場合は除く)

In other words, tools intended to assist music production with NEUTRINO were explicitly excluded from that restriction. I checked the EULA specifically because of this concern, and proceeded with the implementation based on that wording.

The EULA was later revised on August 21, 2026, after this implementation had already been developed. In the new version, that exception was removed. It also added a new restriction prohibiting the use of singing libraries intended for NEUTRINO without going through the NEUTRINO system, as well as providing those libraries to third parties:

本システムで使用する歌声ライブラリを、本システムを介さずに利用する行為または第三者へ提供する行為

So I agree that the current EULA raises a real concern for this PR.

I am quite frustrated by the timing of this change, since I specifically checked the license before beginning development, and the relevant exception was explicitly present at that time. The new wording substantially narrows what third-party tools appear to be allowed to do compared with the terms under which I began this work.

I don't want to speculate about STUDIO NEUTRINO's reasons for changing the EULA, but from a developer's perspective, it is unfortunate that the relevant terms changed in this way only after the implementation had already been developed.

@xiawow

xiawow commented Aug 25, 2026

Copy link
Copy Markdown
Author

I've sent an email to STUDIO NEUTRINO to clarify whether this implementation is permitted under the current EULA. I'll post an update here when I hear back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants