Add incremental rendering API to OfflineAudioContext - #630
Conversation
OfflineAudioContext currently offers two rendering entry points, and neither fits hosts that embed a JavaScript engine: - start_rendering_sync() renders the whole buffer in one call and consumes the renderer; - suspend()/suspend_sync() must be registered before rendering starts - suspending after the renderer has been taken panics with "cannot suspend when rendering has already started". The Web Audio suspend/resume contract is inherently incremental: render up to the suspend point, resolve the suspend promise, let script mutate the graph inside the promise callback, resume, render the next segment. An embedder cannot know the suspend points up front (script may schedule new ones from inside a suspend callback), so a pull-style API is needed. This adds three methods, mutually exclusive with suspend/suspend_sync: - render_upto_sync(frame): renders forward to the given frame (rounded up to whole render quanta), returns the total rendered frame count, and can be called repeatedly to continue; - rendered_frames_sync(): lock-free progress accessor (drives currentTime on the embedder side); - take_rendered_sync(): takes the finished AudioBuffer exactly once. Implementation notes: - The rendered frame count is a lock-free AtomicUsize rather than being derived from the incremental state. The early-return paths of render_upto_sync report the count while holding the state mutex, and a std Mutex is not re-entrant: deriving the count through the same mutex self-deadlocks the calling thread on completely legal call sequences (render to the end, take the result, call render_upto_sync again). The atomic also keeps reporting the full length after the result has been taken instead of dropping back to 0. - While parked between segments the context state is set to Suspended (scripts observing `state` from a suspend callback must read "suspended" per spec), Running while a segment renders, and Closed after the final segment. - After finalizing, the event loop is spun once more so that the complete/statechange events queued by the finalization itself get delivered, matching the tail of start_rendering_sync. - RenderThread grows render_offline_quanta() (renders n quanta without consuming self) and finish_offline_render() (destructor run + event drain), splitting the existing one-shot render_audiobuffer_sync into resumable pieces. Tests: chunked rendering produces bit-identical output to a one-shot render of the same graph, state transitions are observable, the taken result is single-shot, and post-completion / post-start_rendering calls are safe no-ops.
|
I'm fine with adding the ability for incremental rendering to this lib, but it will have to follow the spec that is currently developed over at WebAudio/web-audio-api#2675 Can you assess if that spec will help your use case? Otherwise me might need to chime in to that discussion.
I'm curious about the architecture that you envision. Are you not generating the audio on an audio thread? Did you see our sibling project btw https://github.com/ircam-ismm/node-web-audio-api ? |
|
Thanks @orottier — happy to walk through the architecture and assess #2675. "Are you not generating the audio on an audio thread?" For realtime contexts, we do — we drive the crate's cpal audio thread exactly as This PR is only about offline contexts. We embed a JS engine (V8/JSC) and expose the The concrete use case: the standard We're implementing the existing, shipping
So the suspend points are not known up front — a suspend callback can register new To be clear, Does #2675 help? I read it, and yes — its
Given that, my suggestion: land this as a Would that work for you? And does node-web-audio-api already expose an equivalent |
|
Hey @cmzy,
No, the I personally still don't really understand your use case here, but if your suspend end-points are not known up front, couldn't you do something like: let suspendTime = 0;
const duration = 5;
const sampleRate = 48000;
async function someInput() {
await new Promise(resolve => setTimeout(resolve, 1000));
suspendTime += 1;
return Promise.resolve(suspendTime);
}
function suspendAndWaitForInput(context, suspendTime) {
context.suspend(suspendTime).then(async () => {
const suspendTime = await someInput();
console.log(suspendTime);
if (suspendTime < duration) {
suspendAndWaitForInput(context, suspendTime);
}
context.resume();
});
}
const offline = new OfflineAudioContext({
sampleRate,
length: duration * sampleRate,
});
suspendAndWaitForInput(offline, 0);
const buffer = await offline.startRendering();
console.log(buffer);This is a toy example but it works in Chrome, and given Ok, the drawback is that it crashes for some reason with |
|
Thanks — that example pokes at exactly the right thing, so I ran it locally. Here's the Architecture / "are you not generating the audio on an audio thread?" Realtime contexts do go through the cpal audio thread — that path needs none of this PR. Two empirical results
So the real question isn't "add a non-standard method vs. use the standard approach" The standard |
|
Ok, I think I understand your point. Just few remarks
This is not completely true, you can create an online context that does not do through Cpal by doing const audioContext = new AudioContext({ sinkId: { type:'none' } });
In JS maybe you can wrap the thing in such way that it doesn't introduce a new method, but in Rust context this looks like a new user facing API to me: pub fn render_upto_sync(&mut self, upto_frame: usize)So there is still a issue concerning "add a non-standard method vs. use the standard approach". In my opinion this should better be considered as a bug fix (within the existing rust API surface) rather than a new feature then For information, another related spec issue WebAudio/web-audio-api#2662 |
The suspend/resume driving model is inherently dynamic: the standard
pattern schedules the next suspend point from inside the previous point's
promise callback, i.e. *after* `start_rendering()` has begun. Previously
`suspend()` panicked in that case ("cannot suspend when rendering has
already started"), because the suspend registry is moved into the render
future when rendering starts, so the context can no longer reach it.
- `suspend()` now registers the point synchronously on the call and
returns a future that only waits for the point to be reached. This
matches the eager-registration semantics of JS promises and guarantees a
point scheduled right before `resume()` is in place before the render
loop advances past it.
- Points scheduled after rendering has started are sent through an
unbounded injection channel that the async render loop drains each
quantum (and right after each resume), merging them into the pending
suspend list.
The one-shot `start_rendering_sync` path and `suspend_sync` are unchanged;
`suspend_sync` after rendering still panics as before.
Adds a regression test mirroring the dynamic suspend/resume pattern
(suspend points scheduled from inside the previous point's callback).
|
Thanks — I took your suggestion and treated it as a bug fix within the existing suspend/resume API. @orottier’s toy example now runs to completion natively in the lib (no panic, and the dynamically scheduled suspends fire at the right frames); I added it as a regression test. |
|
Hey, just to let you know that I have tested your patch with But then, there seem to be no reason to keep the |
3 participants
OfflineAudioContext currently offers two rendering entry points, and
neither fits hosts that embed a JavaScript engine:
consumes the renderer;
suspending after the renderer has been taken panics with "cannot
suspend when rendering has already started".
The Web Audio suspend/resume contract is inherently incremental: render
up to the suspend point, resolve the suspend promise, let script mutate
the graph inside the promise callback, resume, render the next segment.
An embedder cannot know the suspend points up front (script may schedule
new ones from inside a suspend callback), so a pull-style API is needed.
This adds three methods, mutually exclusive with suspend/suspend_sync:
up to whole render quanta), returns the total rendered frame count,
and can be called repeatedly to continue;
currentTime on the embedder side);
Implementation notes:
derived from the incremental state. The early-return paths of
render_upto_sync report the count while holding the state mutex, and a
std Mutex is not re-entrant: deriving the count through the same mutex
self-deadlocks the calling thread on completely legal call sequences
(render to the end, take the result, call render_upto_sync again). The
atomic also keeps reporting the full length after the result has been
taken instead of dropping back to 0.
(scripts observing
statefrom a suspend callback must read"suspended" per spec), Running while a segment renders, and Closed
after the final segment.
complete/statechange events queued by the finalization itself get
delivered, matching the tail of start_rendering_sync.
consuming self) and finish_offline_render() (destructor run + event
drain), splitting the existing one-shot render_audiobuffer_sync into
resumable pieces.
Tests: chunked rendering produces bit-identical output to a one-shot
render of the same graph, state transitions are observable, the taken
result is single-shot, and post-completion / post-start_rendering calls
are safe no-ops.