Repository navigation
feat(h3): HTTP/3 transport over quiche, with ECH on both ends - #21
Merged
Merged
Conversation
Adds an HTTP/3 (QUIC) transport alongside the existing TLS+WebSocket one: `http3=true` on the server, `transport=h3` on the client. The payload path is unchanged — a WebSocket Extended CONNECT (RFC 9220) on the same secret path, bridged with copy_bidirectional. - quic: shared plumbing — quiche::Config from the project's BoringSSL cert/key/ECH wiring, per-stream state, and a QuicStream AsyncRead + AsyncWrite adapter so the tunnels reuse copy_bidirectional exactly like the WS path. - h3_server: one task owns the UDP socket and every quiche::Connection (quiche isn't Sync); tunnels run as separate tasks over channels. Non-matching requests get the same nginx-shaped 404 as the TCP listener, so the QUIC port looks equally uninteresting to a scanner. CertSource keeps PEM behind an ArcSwap with a generation counter, so ACME renewals reach the QUIC listener and quiche::Config is only rebuilt when the cert actually rotates. - h3_client: lazy connect, then every accepted sslocal connection becomes a stream on a single QUIC connection — only the first pays for a handshake, and a stalled stream can't head-of-line-block the others. A dead session reconnects on next demand. ECH works on QUIC after all. quiche exposes no ECH API, but under its default boringssl-boring-crate feature quiche::Connection implements AsMut<boring::ssl::SslRef>, and it doesn't start the TLS handshake until the first send() — so there's a window after quiche::connect to apply SSL_set1_ech_config_list, the same call the TCP path makes. quiche and tokio-boring share one boring 4.21.2, so the SslRef types match. The QUIC Initial now carries an ECHClientHello whose outer SNI is only the cover public_name. reject_non_ech is weaker here and is documented rather than papered over: the TCP listener peeks at the cleartext ClientHello before handshaking, so a probe never sees the production cert, but a QUIC Initial is encrypted under keys derived from its own DCID. h3_server can only check SSL_ech_accepted post-handshake — that hides the tunnel, not the cert. Use the TCP transport when cert concealment is the threat model. Also dedupes the client ECHConfigList loader into ech::load_client_config_list (shared with tls_client) and ignores the .docker-cache/ dir left by cross musl builds. Verified with two new e2e tests covering the H3 path with and without ECH. The ECH one was sabotage-checked: stubbing out install_client_ech makes it fail on the server's reject_non_ech drop, so it isn't passing vacuously. fmt, clippy -D warnings, and all 87 tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ection Found deploying to a live server: under concurrency the client dropped roughly a fifth of connections with "h3 send_request: StreamBlocked". quiche returns StreamBlocked when the underlying QUIC stream has no send capacity yet, and TransportError(StreamLimit) when we're at the peer's concurrent-stream cap (MAX_STREAMS_BIDI, 256 here). Both clear as the connection makes progress — quiche's own docs say to retry once the stream is writable again — but open_stream treated every error as fatal and dropped the OpenReq, which the accept task reads as "no stream available" and closes the sslocal connection. open_stream now returns the request back for those two cases and the session re-parks it in `pending`, reusing the slot that already waits out the peer's extended-CONNECT SETTINGS. No busy-loop: while a request is parked the accept branch is disabled and the loop still blocks on the socket/timer, so the retry rides the next packet. Regression test drives 400 concurrent connections through one QUIC connection, well past the 256-stream limit. Sabotage-checked: reverting the fix fails 82/400 streams, with the fix 0 fail. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an HTTP/3 (QUIC) transport alongside the existing TLS+WebSocket one:
http3=trueon the server,transport=h3on the client. Both default to off, so existing deployments are untouched.The payload path is unchanged — a WebSocket Extended CONNECT (RFC 9220) on the same secret path, bridged with
copy_bidirectional.What's here
src/quic.rs— shared plumbing:quiche::Configbuilt from the project's existing BoringSSL cert/key/ECH wiring, per-stream send/receive state, and aQuicStreamAsyncRead + AsyncWriteadapter so the tunnels reusecopy_bidirectionalexactly like the WS path.src/h3_server.rs— one task owns the UDP socket and everyquiche::Connection(quiche isn'tSync); tunnels run as separate tasks over channels. Requests that aren'tCONNECT+:protocol: websocket+ the secret path get the same nginx-shaped 404 the TCP listener serves, so the QUIC port looks equally uninteresting to a scanner.CertSourcekeeps PEM behind anArcSwapwith a generation counter, so ACME renewals reach the QUIC listener and the expensivequiche::Configrebuild only happens when the cert actually rotates.src/h3_client.rs— lazy connect, then every acceptedsslocalconnection becomes a stream on a single QUIC connection: only the first pays for a handshake, and a stalled stream can't head-of-line-block the others. A dead session reconnects on next demand.ECH works on QUIC after all
The initial assumption was that the H3 path would have to leak the real SNI, because quiche exposes no ECH API. It doesn't have to:
boringssl-boring-cratefeature,quiche::ConnectionimplementsAsMut<boring::ssl::SslRef>.send().That leaves a window after
quiche::connectto callSSL_set1_ech_config_list— the same call the TCP path makes. quiche and tokio-boring share oneboring 4.21.2in the lockfile, so theSslReftypes are identical. The QUIC Initial now carries an ECHClientHello whose outer SNI is only the coverpublic_name.Second commit:
StreamBlockedis retryable, not fatalCaught while deploying this to a live server — under concurrency the client dropped roughly a fifth of connections with
h3 send_request: StreamBlocked.quiche returns
StreamBlockedwhen the underlying QUIC stream has no send capacity yet, andTransportError(StreamLimit)when we're at the peer's concurrent-stream cap (MAX_STREAMS_BIDI, 256). Both clear as the connection makes progress — quiche's own docs say to retry once the stream is writable again — butopen_streamtreated every error as fatal and dropped the request, which the accept task reads as "no stream available" and closes thesslocalconnection.open_streamnow hands the request back for those two cases and the session re-parks it, reusing the slot that already waits out the peer's extended-CONNECT SETTINGS. No busy-loop: while a request is parked the accept branch is disabled and the loop still blocks on the socket/timer, so the retry rides the next packet.One caveat worth reviewing
reject_non_echis genuinely weaker on H3, and is documented rather than papered over. The TCP listener peeks at the cleartext ClientHello before handshaking, so a probe never sees the production cert. A QUIC Initial is encrypted under keys derived from its own DCID, soh3_servercan only checkSSL_ech_acceptedpost-handshake — that hides the tunnel, not the cert.Use the TCP transport when cert concealment is the threat model. This is called out in the
quicmodule docs and theconfigoption docs.Also
ech::load_client_config_list, shared withtls_client..docker-cache/, left behind bycrossmusl builds.Testing
Five e2e tests cover the H3 path: plain round-trip, round-trip with ECH, and 400 concurrent streams (past the 256-stream limit). Both the ECH test and the concurrency test were sabotage-checked — stubbing out
install_client_echfails the ECH test on the server'sreject_non_echdrop, and reverting the backpressure fix fails 82/400 streams.cargo fmt --check,cargo clippy --all-targets -D warnings, and all 88 tests pass.Validated in production
Deployed to a live server (
http3=true) and driven from a real client. Confirmed genuinely on QUIC, not falling back: an 8 MB transfer moved with zero TCP connections on :443, and the server loggedHTTP/3 ECH keys installed, which only fires when a real QUIC connection is accepted. ECH verified working over QUIC end to end.Performance note for anyone considering this as a default: on a lossy long-haul link, HTTP/3 measured slower than TCP (1 MB: 3.4–17s over H3 vs 1.9–8.3s over TCP). The feature is correct, but it is not automatically a throughput win — leave it off unless you have a reason.