Add native NEUTRINO v3 ONNX renderer - #2322
Conversation
|
I checked the latest license, and use without going through the NEUTRINO system (exe) is prohibited. This native operation may be in violation of the terms. |
Thanks for pointing this out. I actually checked the NEUTRINO EULA before starting this work. Development of this renderer began around June 2026, which can also be seen from the commit history in my fork. At that time, the applicable EULA was the [October 24, 2024 version](https://studio-neutrino.com/downloads/manual/EULA_NEUTRINO.pdf). While it prohibited decompilation and reverse engineering in general, it explicitly included the following exception:
In other words, tools intended to assist music production with NEUTRINO were explicitly excluded from that restriction. I checked the EULA specifically because of this concern, and proceeded with the implementation based on that wording. The EULA was later revised on August 21, 2026, after this implementation had already been developed. In the new version, that exception was removed. It also added a new restriction prohibiting the use of singing libraries intended for NEUTRINO without going through the NEUTRINO system, as well as providing those libraries to third parties:
So I agree that the current EULA raises a real concern for this PR. I am quite frustrated by the timing of this change, since I specifically checked the license before beginning development, and the relevant exception was explicitly present at that time. The new wording substantially narrows what third-party tools appear to be allowed to do compared with the terms under which I began this work. I don't want to speculate about STUDIO NEUTRINO's reasons for changing the EULA, but from a developer's perspective, it is unfortunate that the relevant terms changed in this way only after the implementation had already been developed. |
|
I've sent an email to STUDIO NEUTRINO to clarify whether this implementation is permitted under the current EULA. I'll post an update here when I hear back. |
Summary
This PR adds native, in-process support for NEUTRINO v3 singers to OpenUtau.
Previous PRs (#2136 #2232 ) have explored NEUTRINO support using the official NEUTRINO executable and a background server. This PR addresses the same use case with a different architecture: the NEUTRINO v3 models run directly in-process through OpenUtau's existing ONNX Runtime integration.
OpenUtau loads the voicebank models directly through its existing ONNX Runtime integration. It does not require the official NEUTRINO executable, a background server, a subprocess wrapper, or a Python environment.
t.binprovides phoneme timing,s.bingenerates the acoustic features, andv.bingenerates the waveform.p.binis used when loading natural pitch and when applying StyleShift.Main features
t.binfor each continuous phrase and shows its predicted phoneme boundaries in OpenUtau's phoneme panel instead of displaying placeholder timing.PITD, as the F0 supplied to the acoustic model.p.binon demand and writes the predicted natural pitch toPITD, where it can be inspected, edited, flattened, or regenerated.SHFCexposes NEUTRINO's StyleShift behavior. The direct pitch shift is removed from thep.binresult, so only the StyleShift-dependent contour is applied and the intended note pitch is preserved.This keeps pitch prediction separate from the F0 supplied to acoustic inference:
p.binremains an optional source for natural pitch, while normal rendering follows the pitch that is visible and editable in OpenUtau.Built-in dictionary
NEUTRINO v3 Japanese voicebanks use a fixed phoneme vocabulary and kana mapping, so the same dictionary does not need to be copied into every voicebank directory.
OpenUtau.Core/Neutrino/NeutrinoJapaneseDictionary.cscontains the kana-to-phoneme table and katakana normalization.OpenUtau.Core/Neutrino/NeutrinoPhoneme.cscontains the v3 phoneme IDs, aliases, and romaji mappings.OpenUtau.Core/Neutrino/NeutrinoPhonemizer.csuses these built-in mappings and loadst.bintiming into the phoneme panel.No dictionary is read from the singer or model directory.
HTS phoneme integration
NEUTRINO consumes an HTS-style phoneme sequence rather than one opaque lyric token per note. The integration therefore:
+,+~, and+*as extension notes that continue the previous sustain phoneme with the extension note's own pitch and duration.The small
RenderPhraseadditions expose the parent-note index, original/edited phoneme position, and available leading context required for these behaviors. The same timing rules are shared by the phonemizer and renderer.Code overview
NeutrinoSinger.csregisters NEUTRINO singers and manages the four model sessions.NeutrinoRenderer.csimplements native timing, acoustic inference, waveform generation, rendered-pitch loading, caching, andSHFCStyleShift.NeutrinoPhonemizer.cs,NeutrinoPhoneme.cs, andNeutrinoJapaneseDictionary.csimplement the Japanese HTS phoneme frontend and model timing display.NeutrinoInferenceUtil.cscontains the shared score, extension-note, timing, pause/breath, and frame-mapping logic.For an official NEUTRINO voicebank, only add the
character.txtandcharacter.yamlmetadata required by OpenUtau and selectneutrinoas the singer type.