AudioBridge audio limiter

[m08pvv]

Worked a bit with AudioBridge and it was painful to hear all crackling noises when multiple streams summed up, so I decided to implement an audio limiter. I took AGC2 from Google's WebRTC (BSD 3-clause license that allows modifications and usage as long as license file stays in repo) and translated it to C (didn't write in C since school, so it might be not clean enough), adapted it for AudioBridge and also added SSE 4.2 and AVX2 versions (with proper initialization from available at runtime).

Also added a parameter to AudioBridge: use_limiter (boolean, default=false), which enables limiter. If limiter is disabled - only clamping (saturation to int16 boundaries) applied.

Here are some results of many participants speaking (5 participants speaking and the record is from incoming ab track):

Top - current Janus release with AudioBridge
Bottom - version with limiter and use_limiter = true
Full-spectrum lines - artifacts (crackling noise and clipping)

As you can see, current Janus AB implementation (just sum up all tracks) results in artifacts (values outside int16 range) whereas limiter reduces volume dynamically to normal level.

How it works:

• On ab initialization we set references to the most efficient (from available) implementation (scalar, sse 4.2, avx2)
• For each segment of audio:
• • If we mix 2 or more tracks (recording 2+ or talking 3+) - calculate scaling factors
• • If we record 2+ tracks mixed - scale and clamp, otherwise just clamp
• • For each participant:
• • • If there are 2 or more tracks besides this participant - scale and clamp, otherwise just clamp

Usage:
By default limiter is disabled and only clamping is applied.
Set use_limiter property in create audiobridge request to true to enable limiter.

Any suggestions are welcome!

[brave44]

@m08pvv @lminiero FYI we tried this in our production environment and got very good results. 1) users with very sensitive mics and loud voices now sounds much better; 2) if they speak at the same time it sounds better too.

Overall I think this can be very good PR for audio bridge, especially for rooms where everybody can speak, not just one speaker.

[m08pvv]

So, @lminiero how it's going?

[lminiero]

It's Easter holidays here, I'll bring it to the rest of the team when we get back. Study of the code apart, we'll need to set it up internally for some specific testing, since as I mentioned it's a very fundamental part of our products, so I won't merge it until I'm 100% sure it won't cause issues. I'm half thinking of putting it on the demo server, and set up one room with and one without: if we set it up so that the default room (1234) uses it, we'll have data in case people using it crash it for any reason.

[m08pvv]

Waiting for test results and hope they'll be ok.

[lminiero]

@m08pvv the current plan is:

1. wait for Make participant encoder/decoder access thread-safe #3606 to be aligned to the latest changes (whch will happen soon, I hope), stress test that, and if everything works merge (hopefully by the end of the week)
2. evaluate the changes in Adding signed_tokens to audiobridge #3635 (low impact) and merge relatively soon (hopefully by the end of the week too)
3. once both are merged, ask you to re-align/re-base to the latest master (not only for the changes in the AudioBridge, but also because we recently merged some security fixes that I want in, before making other tests)
4. use your branch as the version we deploy on the official demo test server, so that it's what people will use for testing (one more reason why it would be important for the code to be aligned to the latest changes)

This will allow me to 1. evaluate the general stability of the code, since random people from random networks doing random things will use it, and 2. by adding different rooms (some with limiting enabled, others not) me and my team can make some more tests related to the audio quality.

Does that make sense for you as a way forward?

[m08pvv]

@lminiero, good plan. Looking forward for upcoming merges!

[lminiero]

@m08pvv FYI I've merged both the PRs I mentioned. When you have some time, could you align to the latest changes there? Hopefully it won't be too complicated, since I see some conflicts there. Thanks!

[m08pvv]

@lminiero, will update my branch today hopefully

[m08pvv]

Merged, will check for any issues after merge (just in case) a bit later

[lminiero]

@m08pvv thanks! I'm out for a conference for a few days, but as soon as I'm back I'll update the demo server as we discussed (which whould also give you time to double check everything's still ok).

[m08pvv]

@lminiero, I'll check today that merge didn't break anything (it was a pretty simple merge, but just in case)

[m08pvv]

So, tested with 5 participants, all works fine

[m08pvv]

@lminiero I'm just curious how it's going? Is it already on your demo server? Hope everything is ok (keeping my fingers crossed🤞)

[lminiero]

My apologies, it's not on the demo server yet... It's been a brutal couple of weeks, as I was to two separate conferences the past two weeks. I'll switch the demo server base to that in the next couple of days 🙏

[lminiero]

I've just updated the demo server to use your branch. There are two rooms configured:

• 1234, the default, has use_limiter set to true, but also denoise (as before we had used it to test the RNnoise integration). This should help verify if the two features together don't break each other.
• 4321 is a room with everything disabled instead.

The second room can be accessed passing the ?room=XYZ query string argument demos support:

https://janus.conf.meetecho.com/demos/audiobridge?room=4321

Both rooms are configured to use 16000 as a sampling rate, as it's probably the most used. We collect core dumps when the server crashes, so that should help identify potential causes of issues.

A couple of notes from while I was upgrading. There's a warning when building:

plugins/janus_audiobridge.c: In function 'janus_audiobridge_mixer_thread':
plugins/janus_audiobridge.c:8654:18: warning: variable 'has_silent' set but not used [-Wunused-but-set-variable]
 8654 |         gboolean has_silent = FALSE;
      |                  ^~~~~~~~~~

Besides, I see the "list" request hasn't been updated to return info on whether the limiter is enabled for specific rooms. That may be helpful, as right now for instance I have no idea if how I configured the rooms actually did the trick or not. I see there are other features we may need to add to "list" (like denoise and others) so that's on me too.

You also didn't update the sample configuration file, but only the description in the Audiobridge doxygen section. The configuration sample does need to list use_limiter as an option, as in the docs, as most people only read what's written there and not the documentation (sad but true).

[m08pvv]

Will check and fix all points mentioned. Not sure when (this/next week I hope)

[lminiero]

@m08pvv sorry for the radio silence! I finally found some time to properly test your patch, with the help of a colleague. We focused on our own conversation and many other fake participants chiming in at different times. I have to say the results were much better than I expected: as I mentioned in previous comments, I was afraid that just unmuting could cause drastic lowerings of the volume, but we didn't experience any of that, and the audio, while chaotic (but that was normal due to the fake participant's audio that was out of context), was always quite good. The end result was even more impressive when we did the same test on the "regular" AudioBridge, where clipping and artifacts were indeed brutal. I want to make some more tests involving my other colleagues (for a "proper" test with "proper" audio), but I think it's safe to say that I definitely see this as a valid improvement, and so there's a practical plan for merging sooner rather than later!

One area where we think the current patch may be improved would be addressing overflow mixing in general, as in louder scenarios we were still getting very close to the maximum levels. I think that simply integrating the overflow mixing done in #3601 may be enough, were the cap is not INT16_MAX but about 0.9 of that. That would provide some nice limiting to stay away from saturation territory.

While we plan for more internal testing, could you address this and the other notes I mentioned in previous comments? I think what's left to be done is mainly:

1. overflow mixing limit
2. naming conventions (e.g., a janus_audiobridge_ prefix for all new functions)
3. info on the feature in requests that return info on rooms and participants (e.g., list, or query_session for the Admin API; not sure if it makes sense for instance to return info on each partipants's state when pinging there)

More may come to mind but I think this is mostly what's missing right now before we get to a mergeable state.

Looking forward to your thoughts, and thanks again for your work (and most importantly patience 😆 )

[m08pvv]

@lminiero, thanks for all the tests and a review. I'm back from vacation and hope will find some time sooner rather than later 😉 to fix all mentioned points.

[m08pvv]

@lminiero, I pushed some of the changes (added janus_audiobridge_ prefix to functions, added info to 'list' command and it's documentation, added use_limiter option to the audiobridge sample configuration)

[m08pvv]

And regarding saturation - can you attach some audio samples that cause issues? Because a simple cut-off for a complex audio signal might sound much worse than an audio that almost reaches maximum 🤷‍♂️

[lminiero]

And regarding saturation - can you attach some audio samples that cause issues? Because a simple cut-off for a complex audio signal might sound much worse than an audio that almost reaches maximum 🤷‍♂️

There's no actual issue: the whole idea was to avoid that the mix ever gets to samples with the loudest value possible, hence limiting the max value we accept to ~0.9 the actual max value (INT16_MAX). This is what some audio solutions do in order to avoid problems with playback. So I'm not proposing a hard cut-off, but a smaller max that your code should take into account when limiting. I guess it would be hard to enforce the same when the limiter is not active, though, i.e., when there's just a couple of people in the room.

[m08pvv]

Basically the limiter itself limits the mix and it almost never gets to such high values (except some cases when volume changes in some non-linear way that it's hard for the algorithm to smoothly catch). If you have an example of such audio I can take a look what we can do to avoid it. Maybe we can just add some 0.9 gain if we encounter such case, but anyway have to see an example.

[m08pvv]

@lminiero, let's sync what's left before the merge?

[lminiero]

The current plan is to merge after the summer break: I'll go to the IETF meeting in a few days, and when I'll be back I'll only have a few days before a bit of a holiday break. When back, unless the testing on the public demo says otherwise, I think we can merge.