feat: (s2n-dc-quic) Adds metrics to offloading feature

[maddeleine]

Release Summary:

• dc-quic now takes in a Registry and uses it to instrument the offloading runtime and offloading tasks.

Resolved issues:

n/a

Description of changes:

dc-quic now takes in a Registry and uses it to instrument the offloading runtime and offloading tasks. Internally we will build dc-quic with a Registry and use that to emit aggregate stats about offloading.

Call-outs:

Testing:

n/a

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[WesleyRosenblum]

How hard would it be to enhance the existing s2n-quic-dc-metrics to give access to the numeric metrics you need (rather than the String)?

[maddeleine]

How hard would it be to enhance the existing s2n-quic-dc-metrics to give access to the numeric metrics you need (rather than the String)?

It's not necessarily hard, I think it would just be kind of tedious. And I wanted to get some consensus before doing that.

For reference, if you use the existing Registry to get TaskMetrics in our s2n-quic-dc-metrics crate, then you end up with this String as output. It's definitely more precise than averages, but the format makes me think these metrics are meant for some other purpose. Honestly I'm not sure why we even have the s2n-quic-dc-metrics crate.

TokioTask:DroppedTasks=1005
TokioTask:FirstPollDelay=115*102+173*101+216*100+267*103+306*102+355*99+408*101+490*100+595*103+673*49+788*42+866*7+890*1 us
TokioTask:IdleDuration=9*204+10*218+11*193+12*194+24*201+681*202+1249*208+1642*196+1978*207+2154*98+2334*79+2498*18+2744*2
TokioTask:InstrumentedTasks=1010
TokioTask:PollDuration=0*304+1*306+3*300+159*332+180*276+211*302+250*307+293*302+308*308+322*156+334*116+357*20+410*3 us
TokioTask:ScheduledDuration=8*208+12*198+29*201+76*204+122*202+150*203+178*202+235*200+326*203+398*102+575*79+718*18+792*2 us 

[Mark-Simulacrum]

I'm more interested in task statistics, which the s2n-quic-dc-metrics sort of has, but then I'd have to go through a Registry, which seems to only output Strings, which I then have to parse to be able to emit them as dc-quic events.

The internal usage directly ties events into a Registry which is how all aggregate metrics are emitted today, so it should be fine to just expect a Registry to be available in the psk code and register the tasks with it. Note that the approach here is (effectively) tracking a histogram of the means (sampled roughly once per second) whereas the dc-metrics implementation of task metrics tracks a histogram of the scheduled durations.

the format makes me think these metrics are meant for some other purpose

Say more? That format is just a dump of the underlying percentiles. I think it's intended for exactly the purpose we're interested in here.

I also thought emitting this data per second made sense, as per connection doesn't really make sense to me, but lmk what you think.

I think having aggregate data is a good first step (we should get a version of that out quickly -- though I'd probably instrument the runtime and s2n-quic tasks too -- just to get everything we can easily get in one go), but I suspect it'll also be very valuable to capture per-handshake metrics. That's more work to wire up so I'd do that as fast follow up though.

Non-aggregate data would allow us to definitively say that a given slow handshake spent X milliseconds scheduled in the Tokio runtime. With just aggregate data, I think we'd at least want a counter for how many times a given handshake's tasks are enqueued and waiting, otherwise moving from the aggregate scheduled/poll durations of the background runtime to the impact on handshake latency is quite hard (involves some extrapolating I think).

[maddeleine]

We have to give an interval for the instrument_runtime function so I chose a second. Seems reasonable 🤷‍♀️

[maddeleine]

The internal usage directly ties events into a Registry which is how all aggregate metrics are emitted today, so it should be fine to just expect a Registry to be available in the psk code and register the tasks with it.

Alright. I changed the server builder to expect a registry and registered the offloading runtime stats with it.

though I'd probably instrument the runtime and s2n-quic tasks too

I'd rather register s2n-quic tasks in a different PR. It's going to require a new API in s2n-quic and don't want to have that block this PR.