feat(s2n-quic-dc): enable cbpf for packet filtering

[boquan-fang]

Release Summary:

Implement a cbpf packet filtering for dcQUIC. This is an attempt to reduce dcQUIC handshake latency.

Resolved issues:

resolves #2954. resolves #2735.

Description of changes:

Implement a ROUTER to filter server's incoming packets into two sockets:

1. INITIAL packet contains client hello to one socket.
2. Route all other packets to the second socket.

The instruction for the router is to load the first and the sixth byte and check if the first byte is 1100 and the six byte is exactly 8 bytes. Refer to https://www.rfc-editor.org/rfc/rfc9000.html#name-initial-packet for more information about packet structure:

Initial Packet {
  Header Form (1) = 1,
  Fixed Bit (1) = 1,
  Long Packet Type (2) = 0,
  Reserved Bits (2),
  Packet Number Length (2),
  Version (32),
  Destination Connection ID Length (8),
...
}

s2n-quic uses 8 bytes CID when the client sends Client Hello to the server, and that's why we are checking the eight bytes CID length.

s2n-quic/quic/s2n-quic-transport/src/endpoint/mod.rs

Lines 1125 to 1136 in 8c64e04

	//= https://www.rfc-editor.org/rfc/rfc9000#section-7.2
	//# When an Initial packet is sent by a client that has not previously
	//# received an Initial or Retry packet from the server, the client
	//# populates the Destination Connection ID field with an unpredictable
	//# value.
	let original_destination_connection_id = {
	let mut data = [0u8; InitialId::MIN_LEN];
	endpoint_context
	.random_generator
	.public_random_fill(&mut data);
	InitialId::try_from_bytes(&data).expect("InitialId creation failed.")
	};

I also restructured the tokio builder for the rx_socket field to takes an array of sockets and allow with_rx_socket to append sockets to rx_sockets array instead of override the existing socket.

After those changes are tested and verified, I make s2n-quic-dc's server endpoint to always use packet filtering features.

Also: #2967 (comment).

Call-outs:

• Should we add some guards to make this feature available only to dcQUIC?
• This feature is only available on Linux and is also not available on aarch64 platform.
  

  s2n-quic/quic/s2n-quic-platform/src/lib.rs

  Lines 11 to 12 in 8c64e04

  	#[cfg(target_os = "linux")]
  	pub mod bpf;

• Check my tests to see if they are enough.
• The integration test has to use tokio runtime, so I just add it in s2n-quic-platform crate. That would requires me to import s2n-quic as a dev-dependency. That shouldn't cause any problem for production code and release.

Testing:

• Unit tests: router_cbpf_packet_filtering_test
  This test primarily test the functionality of the ROUTER. I constructed three packets (1 client hello packet and two other packets). We verified that the first packet should be route to socket 0 and the other two should be routed to socket
  * Integration test: router_multi_socket_server_client_test
  Use tokio runtime to construct s2n-quic server and client. The server should call .with_rx_socket twice to attach two sockets to the router and the handshake should succeed.
• Add PlatformRxSocketStats event to track the number of packets routed to a socket. With this event, I added another integration test client_hello_routed_test.
  • The test track server's socket 1 should only receive one packet, which is the Client Hello received from the client. The server's socket 0 should receive more than one packet.
• dc_server_packet_filtering_load_test: this test simulate a dcQUIC client tries to connect with a dcQUIC server, while a packet generator flood the server with EXAMPLE_CLIENT_INITIAL_PROTECTED_PACKET. With packet filtering, the actual handshake should still succeed. I ran this test before the this feature was implemented, and the test will fail: main...boquan-fang:s2n-quic:before-packet-filtering.
  • I have uploaded a log file for the test.

load_test.log

• Other dcQUIC stream test should verify that with changes in s2n-quic-dc's server, they should still pass.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[Mark-Simulacrum]

This feature is only available on Linux and is also not available on aarch64 platform.

Why can't we enable this on aarch64?

Edit: I see there's a comment saying it's not supported (" This test is Linux x86_64-only because cBPF socket filters are not supported on aarch64.") but that feels unlikely to me. Can we dig into what specific problem we're seeing and hopefully fix it? I'm pretty sure I've used cBPF filters on aarch64 before.

[boquan-fang]

To address #2967 (comment), I implemented a socket polling prioritization: we prioritize socket 1 which handles non-client hello socket over the other socket. To test this feature, I added a dc_rx_priority_scheduling_test: we spawn two packets generator at approximately the same time. One constantly send the EXAMPLE_CLIENT_INITIAL_PROTECTED_PACKET and the other send a PACKET will be routed to socket 1. The test should verify that socket 1 should receive much more packets than socket 0.

The result is this:
The test with the current code:

Socket 1 flood sent: 269825, Socket 0 flood sent: 269825, Socket 1 rx: 1344, Socket 0 rx: 222

The test without socket scheduling prioritization:

Socket 1 flood sent: 280961, Socket 0 flood sent: 280961, Socket 1 rx: 1341, Socket 0 rx: 1512

With socket scheduling, socket 1 will receive about 6 times more packets than socket 0, while without it, both sockets receive approximately the same number of packets.

The CI ubuntu image is running slower than my local machine. Hence, I loosen the requirement for a bit. Requiring socket 1 to receive more than three times packets than socket 0.

[maddeleine]

This PR is now implementing three separate features AFAICT:

1. Adds the ability for s2n-quic to read packets from multiple sockets
2. Adds the ability for s2n-quic to drain one socket before moving on to the next socket.
3. Adds a packet filter to dc-quic's handshake sockets so that Initial packets get processed in a different port than the rest of the packets.

I don't agree with the decision to put all of these in the same PR, because it makes it difficult to tell if you've tested everything sufficiently. I would probably put feature 1 and 2 in the same PR, since those are making changes to s2n-quic, and then do a different PR for 3, since it is only touching dc-quic.

[boquan-fang]

#3026 will finish what this PR is trying to do.