Programmable Rust hooks for sōzu — design proposal

[FlorentinDUBOIS]

This issue is a self-contained design proposal for compile-time-linked Rust hooks in sozu-lib, in response to the original ask in #1178. It folds the maintainer reply, six design decisions, and findings from a multi-stream code review (correctness, project-guidelines conformance, simplification, security threat-hunt, and a second-model cross-check). Implementation does not start until this proposal is approved.

Implementation phasing, PR sequencing, risk register, items requiring resolution before implementation, and cross-references continue in the first comment below (the proposal is split across two posts because the full body exceeds GitHub's per-comment size limit).

Original ask: #1178.
Deployment target: kemeter/sozune — a Traefik-style higher-level reverse proxy built on top of sozu-lib. Stock sozu binary continues unchanged; Sozune ships the operator-facing middleware (basic auth, rate limit, headers, strip-prefix, redirects, compression, backend-timeout) by linking against sozu-lib and registering custom hooks.
Out of scope: WebAssembly / dynamic plugin loading. Pure Rust, compile-time-linked.

---

1. Context

1.1 The ask

@Shine-neko (issue #1178) wants a way to plug a function/callback at two stages of the pipeline: rewrite or modify requests (path, headers) before routing, and rewrite or modify responses (headers, status) before sending them back. @Wonshtrum (maintainer) replied that the closest existing primitives are kawa_h1::ParserCallbacks::on_request_headers / on_response_headers plus the URL-rewrite / header-injection / redirect / basic-auth feature stack that landed via PRs #1206-#1210, but these are static configuration, not operator-supplied logic. The maintainer explicitly invited a programmable surface.

1.2 The reference points

Three reference designs were studied to anchor sōzu's choices:

• Pingora (Cloudflare): a single ProxyHttp trait with ~30 hooks, all async fn, monomorphised global per Service. Per-session type CTX associated type. Strong fit for tokio-based proxies; doesn't translate to sōzu's mio-only lib/.
• HAProxy: three pluggability surfaces — native filter modules (C, sync function pointers, flt_ops struct with lifecycle + channel + HTTP/TCP hooks), Lua scripting (in-VM coroutines via core.register_action cheap and core.register_filter body-streaming), and SPOE (out-of-process NOTIFY/ACK protocol for verdict-only enrichment).
• Sōzu's existing primitives: kawa_h1::ParserCallbacks::on_request_headers / on_response_headers (parser callbacks, internal), Router::route_from_request (resolves cluster + per-frontend policy), config-time per-frontend programmability (headers: Vec<Header>, redirect, rewrite_*, required_auth, answers."NNN", max_connections_per_ip).

1.3 The design decisions (operator-confirmed)

Six questions were asked, six answers locked:

Q	Question	Decision
Q1	Storage location — where the hook reference is held	(b) Listener-owned registry, frontends/clusters reference hooks by ID
Q2	Sync vs async — what shape the hook signatures take	(i) Pure sync + (ii) yield-on-pending mio integration. Both ship in v1, NOT deferred to v1.5
Q3	API surface — single trait or two-tier	(b) Two-tier — HeaderHook cheap + always available, BodyHook opt-in
Q4	Operator deployment shape — distribution model	(a) Pingora-style binary build — operator (Sozune) recompiles their own binary linking sozu-lib
Q5	ABI stability — facade layer or direct internal exposure	Expose internal types directly — &mut HttpContext to hooks; Sozune accepts coupling
Q6	Header edit type — new HeaderEdit enum or reuse existing Header	(a) Reuse Header — operators emit Vec<Header> from hooks, same downstream pipeline

Plus the per-stream state slot:

• TypeId-keyed HashMap on HttpContext, lazy-allocated (Option<HashMap<TypeId, Box<dyn Any + Send + Sync>>>). Each hook's state type is its own slot — no collisions, no string IDs, no operator burden.

1.4 Hard constraints inherited from sōzu

1. No async fn in lib/. mio + edge-triggered epoll, single-threaded per worker. Hooks cannot be async fn. The yield-on-pending path satisfies operator I/O needs without breaking this rule.
2. No Arc<Mutex> inside the event loop. Slab-allocated session state. Hooks see &mut HttpContext per-session; cross-session shared state belongs in &self on the hook impl (e.g. Arc<RwLock<BanList>>).
3. No panic on network-facing input. Hooks must convert errors into HookOutcome::Reject(status) or HookError. unwrap / expect / panic! forbidden in the protocol layer.
4. Edge-triggered epoll discipline. When a hook queues bytes for write, it MUST call signal_pending_write on the readiness tracker — there is no wake-up for free.
5. Buffer-pool-owned bodies. Body-stage hooks operate on per-session buffer-pool checkouts; chunk size cannot grow beyond the buffer slot.
6. H2 stream ordering. Per-stream pending state must respect HTTP/2 framing — pending on stream 5 must NOT block stream 7 from making progress, but stream 5's response cannot go out before stream 5 finishes its hook chain.

---

2. Public API specification

2.1 Module layout

New module: lib/src/hooks/

lib/src/hooks/
├── mod.rs           # public trait + types + builder API
├── outcome.rs       # HookOutcome / WakeupSource / WakeupReason / HookError
├── registry.rs      # HookRegistry — per-listener storage
├── dispatch.rs      # Router-side dispatch (Phase B + C glue)
├── wakeup.rs        # mio Poll integration (Phase C)
└── builtins/        # Phase E — sōzu's own middlewares as hook impls
    ├── mod.rs
    ├── basic_auth.rs
    ├── header_edit.rs
    ├── rate_limit.rs
    ├── redirect.rs
    └── rewrite.rs

2.2 Trait surface

// lib/src/hooks/mod.rs
use std::any::Any;
use std::time::Duration;
use crate::protocol::http::editor::HttpContext;
use sozu_command::proto::command::Header;

/// Stage-1 hook — sees parsed request / response headers, can mutate
/// `HttpContext.headers_request` / `headers_response` (which feed the
/// existing `apply_request_rewrites_and_headers` /
/// `apply_response_header_edits` pipeline). Cheap path; always
/// available. Hook output is byte-validated in the dispatcher before
/// reaching the wire (see §2.7 step 6).
pub trait HeaderHook: Send + Sync + 'static {
    /// Operator-supplied identifier (no default). Used as the metric
    /// label suffix (`hooks.duration.<name>`) and log envelope tag —
    /// must be a stable identifier (`type_name::<Self>()` includes
    /// module paths and is documented as best-effort, so it is not
    /// the default).
    fn name(&self) -> &'static str;

    /// Called after `ParserCallbacks::on_request_headers` and after
    /// `Router::route_from_request` resolves the cluster.
    fn on_request_headers(
        &self,
        _ctx: &mut HttpContext,
        _wake: WakeupReason,
    ) -> HookResult {
        Ok(HookOutcome::Continue)
    }

    /// Called just before `kawa.prepare(...)` on the response side, in
    /// reverse scope order vs. request (cluster → frontend → listener).
    fn on_response_headers(
        &self,
        _ctx: &mut HttpContext,
        _wake: WakeupReason,
    ) -> HookResult {
        Ok(HookOutcome::Continue)
    }
}

/// Stage-2 hook — opt-in body-byte interception. Implementing this
/// trait also requires `HeaderHook` (the body stage runs after the
/// header stage of the same hook). Body hooks are gated behind a
/// listener-builder method `with_body_hook(name, Arc<dyn BodyHook>)`
/// distinct from `with_hook(...)` so the buffer-pool path is only
/// enabled for hooks that opt in.
pub trait BodyHook: HeaderHook {
    /// In-place body chunk mutation. Cannot grow the chunk beyond the
    /// buffer-pool slot. Hooks that shrink the chunk MUST keep
    /// downstream framing consistent — sōzu auto-recomputes
    /// `Content-Length` from the post-hook total on H1 (or marks the
    /// response as `Transfer-Encoding: chunked` if the original CL is
    /// authoritative on a different hop) before forwarding. H2:
    /// post-mutation DATA-frame size drives outbound flow-control
    /// debit. END_STREAM is never swapped under the operator's feet.
    /// Returns `HookOutcome::Continue` to forward, `Reject(status)`
    /// to abort BEFORE response bytes are emitted, `AbortStream { error }`
    /// to abort mid-response, `Pending(...)` to yield.
    fn on_request_body_chunk(
        &self,
        _ctx: &mut HttpContext,
        _chunk: &mut BodyChunk<'_>,
        _wake: WakeupReason,
    ) -> HookResult {
        Ok(HookOutcome::Continue)
    }

    fn on_response_body_chunk(
        &self,
        _ctx: &mut HttpContext,
        _chunk: &mut BodyChunk<'_>,
        _wake: WakeupReason,
    ) -> HookResult {
        Ok(HookOutcome::Continue)
    }
}

pub struct BodyChunk<'a> {
    pub bytes: &'a mut [u8],
    pub end_stream: bool,
}

pub type HookResult = Result<HookOutcome, HookError>;

2.3 Outcome enum

// lib/src/hooks/outcome.rs
use std::time::Duration;
use std::os::fd::RawFd;
use sozu_command::proto::command::Header;

/// Hook return value. Four terminal cases. Header edits are emitted
/// by mutating `HttpContext.headers_request` / `headers_response`
/// directly (Q6 reuses the existing `Header` plumbing); the
/// dispatcher byte-validates every emitted `Header` before it reaches
/// the wire (rejects CRLF / NUL / non-tchar in keys, C0 controls in
/// values per `parse_header_edit` in `command/src/config.rs:1397-1408`,
/// promoted to a shared crate location).
#[non_exhaustive]
pub enum HookOutcome {
    /// Proceed to the next hook in the chain (or to the next pipeline
    /// stage if this is the last hook).
    Continue,

    /// Short-circuit BEFORE response bytes have been emitted. Sōzu
    /// renders the matching default answer template via
    /// `set_default_answer` (`mux/answers.rs:193`); for 429 only,
    /// `set_default_answer_with_retry_after` (`answers.rs:207`)
    /// threading the cluster's effective retry_after via
    /// `SessionManager::effective_retry_after` (`server.rs:243`).
    /// `RejectStatus` is a newtype enforcing 4xx/5xx (and 301/302/308)
    /// at construction; out-of-range values surface as `HookError`
    /// at the dispatcher.
    Reject {
        status: RejectStatus,
        /// Optional cause carried into the per-hook error log; not
        /// rendered on the wire.
        message: Option<String>,
    },

    /// Abort an in-flight response stream. Used when a body hook
    /// detects a problem after response headers (or partial body)
    /// have already been emitted. H2: emit RST_STREAM with the
    /// operator's chosen H2 error code. H1: write-only shutdown
    /// (NEVER `Shutdown::Both` on TLS frontends — write-only
    /// shutdown avoids the TCP RST that truncates already-queued
    /// response bytes); the truncated body is the best-effort
    /// signal to the client.
    AbortStream {
        error: AbortStreamError,
    },

    /// Hook needs to wait for an external event. Sōzu's mio loop
    /// re-enters the hook (with the same `&mut HttpContext` and the
    /// hook's own state slot) when the wake source fires. `timeout`
    /// is capped per worker by `hooks.max_pending_timeout_seconds`
    /// (default 5 s, hard ceiling 60 s); concurrent `HookPending` per
    /// worker is capped at `hooks.max_concurrent_pending_per_worker`
    /// (default ≤ 32 768 to keep headroom inside the 65 536-token
    /// range — see §4.1 / §4.5).
    Pending {
        wake: WakeupSource,
        timeout: Duration,
        /// Same `RejectStatus` constraint as `Reject`.
        wake_timeout_status: RejectStatus,
    },
}

/// Newtype over `u16` enforcing 4xx/5xx (and 301/302/308) at
/// construction. `TryFrom<u16>` returns `InvalidRejectStatusError` for
/// out-of-range values; the dispatcher converts those into
/// `HookError { status: 500, message: "invalid Reject status: <N>" }`
/// and bumps `hooks.invalid_reject_status.<hook_name>`.
pub struct RejectStatus(u16);

#[non_exhaustive]
pub enum AbortStreamError {
    /// H2: emit RST_STREAM with this error code. H1: write-only shutdown.
    Internal,
    Cancel,
    EnhanceYourCalm,
    Custom(u32),
}

/// What the hook is waiting for. The mio loop registers the
/// appropriate event source on the worker's Poll.
#[non_exhaustive]
pub enum WakeupSource {
    /// File descriptor became readable (e.g. a TCP socket the hook
    /// opened to a sidecar Redis). Ownership transfers to sōzu on
    /// `Pending` return — `EPOLL_CTL_DEL` runs in
    /// `WakeupSourceHandle::Drop` before fd close, so a double-close
    /// cannot race a kernel fd-reuse. Operator MUST set `O_CLOEXEC`
    /// and MUST NOT register the same fd on any other Poll.
    FdReadable(OwnedFd),

    /// Channel message arrived. The caller must pre-register the
    /// channel via `HookRegistry::register_channel(token)` at
    /// listener-build time; this variant just stores the token. The
    /// registry monitors `Disconnected` on the channel handle — if
    /// the operator's background thread drops its `Sender`, the
    /// pending hook is re-entered with `WakeupReason::Timeout` (and
    /// `hooks.channel.disconnected.<hook_name>` increments) instead
    /// of leaking the wake-up entry until `timeout` elapses.
    ChannelReady(usize),

    /// Wall-clock delay (e.g. exponential backoff between retries).
    /// Integrates with the worker's existing Timer machinery.
    Delay(Duration),
}

/// Passed when a hook is invoked — `Initial` on first entry,
/// `Ready` / `Timeout` on re-entry from a previous `Pending`. Hooks
/// that issue multiple Pendings across phases use `source` to
/// identify which pending operation completed.
#[non_exhaustive]
pub enum WakeupReason {
    Initial,
    Ready { source: WakeupSourceId },
    Timeout { source: WakeupSourceId },
}

/// Opaque identifier for a `WakeupSource` returned by a `Pending`.
/// Generated at Pending-return time, plumbed through the registry,
/// surfaced on re-entry so hooks can disambiguate concurrent waits.
pub struct WakeupSourceId(NonZeroU64);

pub struct HookError {
    pub status: RejectStatus,
    pub message: String,
}

2.4 HttpContext extension

// lib/src/protocol/kawa_h1/editor.rs (existing struct extended)
use std::any::{Any, TypeId};
use std::collections::HashMap;

pub struct HttpContext {
    // ... all existing fields unchanged ...

    /// Per-session, per-hook state slot. Each hook stores its state
    /// keyed by its own `TypeId`, so two hooks with different state
    /// types never collide. `None` until the first hook stashes
    /// anything — stock sōzu deployments without hooks pay zero.
    pub(crate) hooks_state: Option<HashMap<TypeId, Box<dyn Any + Send + Sync>>>,
}

impl HttpContext {
    pub fn set_hook_state<T: Any + Send + Sync>(&mut self, state: T) {
        self.hooks_state
            .get_or_insert_with(HashMap::new)
            .insert(TypeId::of::<T>(), Box::new(state));
    }

    pub fn hook_state<T: Any + Send + Sync>(&self) -> Option<&T> {
        self.hooks_state
            .as_ref()?
            .get(&TypeId::of::<T>())?
            .downcast_ref::<T>()
    }

    pub fn hook_state_mut<T: Any + Send + Sync>(&mut self) -> Option<&mut T> {
        self.hooks_state
            .as_mut()?
            .get_mut(&TypeId::of::<T>())?
            .downcast_mut::<T>()
    }

    pub fn take_hook_state<T: Any + Send + Sync>(&mut self) -> Option<T> {
        self.hooks_state
            .as_mut()?
            .remove(&TypeId::of::<T>())?
            .downcast::<T>()
            .ok()
            .map(|b| *b)
    }
}

Construction site impact. HttpContext is constructed at five known sites today: mux/h2.rs:7422, mux/answers.rs:358, mux/mod.rs:494 (HttpContext::new), kawa_h1/editor.rs:833 + :862 (test fixtures), kawa_h1/mod.rs:314. The new hooks_state: Option<...> field defaults to None; the explicit struct literals at the first three sites still need their fields updated in Phase A.

Debug impl. HttpContext has #[derive(Debug)] at editor.rs:115 (struct definition at editor.rs:116). Box<dyn Any> is not Debug; the derive must be replaced with a hand-rolled impl Debug for HttpContext that lists every existing field by name and renders the new field as hooks_state: Option<...> (count only). A unit test in Phase A pins format!("{:?}", ctx) is panic-free with stashed Box<dyn Any> state — error-path log dumps ({:?} at e.g. mux/router.rs:~374) must keep working. The derivative crate is NOT a workspace dep; do not add it. No Clone / PartialEq impact.

State-discipline contract. Two hooks both stashing common types (Vec<u8>, String) collide under TypeId::of::<Vec<u8>>() — last writer wins, reader silently misclassifies. Concrete failure mode: auth hook stores user ID, logging hook overwrites with fingerprint, rate-limit hook reads ctx.hook_state::<Vec<u8>>() and gets the fingerprint — per-IP rate-limit silently degrades to per-fingerprint, allowing distributed credential stuffing under a single IP. Hook state MUST be a hook-private newtype:

struct MyAuthState(Vec<u8>);  // OK — distinct TypeId

lib/src/hooks/macros.rs ships a define_hook_state! macro that generates the newtype + Send/Sync bound + custom Debug (skips bytes). Documented as the only supported way to declare hook state. The registry logs a rate-limited warn! when two registered hooks call set_hook_state::<T> with the same T::TypeId.

Field-mutability matrix. Hooks see &mut HookMut (a sealed wrapper over HttpContext), NOT &mut HttpContext directly. The HookMut API gates writes on routing-frozen fields:

Field	Hook access
cluster_id, backend_id	read-only
method, authority, path (post-route)	read-only
tls_*, keep_alive, protocol	read-only
headers_request, headers_response	read-write
hooks_state (via set_/hook_state* helpers)	read-write
frontend_id, request_id, session_id	read-only

Mutating authority after routing was the explicit operator-footgun risk; the read-only gate forces hooks to emit Header { key: "Host", val: ... } edits instead, which flow through downstream re-validation (see §2.7 step 7).

Backing store. §3 estimates 1–3 stashed states per request. For that size, Vec<(TypeId, Box<dyn Any + Send + Sync>)> with linear scan is cache-friendlier than HashMap (no hash, no drop overhead, smaller). The four convenience methods (set_/hook_state*) keep their signatures; backing store is an implementation detail.

2.5 Builder API

// lib/src/protocol/listener.rs (or HttpListener / HttpsListener equivalent)
use std::sync::Arc;
use crate::hooks::{HeaderHook, BodyHook, HookRegistry};

impl HttpListener {
    /// Register a header-stage hook by name. The name is the ID
    /// frontends and clusters reference in their `hook_ids: Vec<String>`.
    /// Must equal `hook.name()` — mismatch returns
    /// `HookRegisterError::NameMismatch`. Registering twice with the
    /// same name returns `HookRegisterError::Duplicate`. The registry
    /// also runs the state-discipline lint (§2.4) and emits a `warn!`
    /// when two hooks declare the same `TypeId` for their state slot.
    pub fn with_hook(
        &mut self,
        name: impl Into<String>,
        hook: Arc<dyn HeaderHook>,
    ) -> Result<&mut Self, HookRegisterError>;

    /// Register a body-stage hook. Implies header-stage registration
    /// at the same name (since `BodyHook: HeaderHook`).
    pub fn with_body_hook(
        &mut self,
        name: impl Into<String>,
        hook: Arc<dyn BodyHook>,
    ) -> Result<&mut Self, HookRegisterError>;
}

2.6 Proto schema additions

// command/src/command.proto
message RequestHttpFrontend {
    // ... existing fields ...
    repeated string hook_ids = 16;  // pick the next available tag at edit time
}

message Cluster {
    // ... existing fields ...
    repeated string hook_ids = 16;  // pick the next available tag at edit time
}

Validation: at config-load time AND at runtime AddHttpFrontend / AddCluster, every hook_id must resolve in the listener's registry. Unknown ID → reject the request with a typed HookConfigError carrying the offending ID, do not silently accept.

Cluster scope on a listener-owned registry — open question. Cluster is a global resource; AddCluster carries no listener context. Resolution of where the cluster's hook_ids get validated (worker-global registry vs. validation deferred to AddHttpFrontend time when a frontend pulls a cluster) is deferred to a dedicated planning session before Phase A starts. v1 keeps cluster hook scope per Q1 — only the validation site is to be decided. Tracked at §8.1 item 2.

Cargo-clean dependency. The workspace's prost map-field derive handling can leave stale generated code after a command.proto edit; run cargo clean -p sozu-command-lib && cargo build -p sozu-command-lib once after editing hook_ids tags before the next workspace build.

Cross-version safety. prost skips unknown fields silently → an older sōzu worker deserialising a config with hook_ids set silently treats the frontend as no-hook (degrade-open). For v1 this is the documented contract; future work may add a WorkerCapabilities advertisement so the master can refuse-application on mixed-version partial upgrades.

2.7 Dispatch order

The dispatch helpers in lib/src/hooks/dispatch.rs are called from Router::connect (mux/router.rs:90 → :141), NOT from inside Router::route_from_request (which would require widening that function's return type). Both H1 and H2 reach Router::connect through the unified mux dispatch path (HTTP/1.1 listeners flow through HttpStateMachine::Mux(MuxClear)), so a single hook insertion point covers both protocols.

Naming note. The "listener-post-route" chain fires AFTER route_from_request returns Ok — chosen for v1 because rewriting path/authority and then re-routing is a deeper change to Router than the wave can absorb. v2 may add a true pre-route stage that runs before route_from_request. The original issue #1178 ask for "rewrite path/headers before routing" is therefore deferred — operators can mutate via headers_request edits but those do not change the routing decision in v1.

Request side (fires AFTER existing config-time programmability):

1. ParserCallbacks::on_request_headers (sōzu invariants — Host validation, CL/TE reconciliation)
2. Router::route_from_request (config-time: redirect, auth, rewrite_*)
3. listener-post-route hook chain (listener.hook_ids on the listener config)
4. frontend hook chain (frontend.hook_ids)
5. cluster hook chain (cluster.hook_ids)
6. Hook output validation: every Header emitted by hooks 3-5 is byte-validated
   (rejects CRLF / NUL / non-tchar in keys, C0 in values) by the helper
   promoted from command/src/config.rs:1397-1408 (parse_header_edit) into a
   shared crate location. Invalid bytes → HookError { status: 500 }, bump
   hooks.invalid_header.<hook_name>.
7. Invariant re-validation: re-fire CL/TE consistency, Host single-value, no
   forbidden-byte check on the post-hook header set. Mandatory whenever the
   hook chain emitted any header edit. Failure → HookError { status: 500 }.
8. Backend connect

Trailer-frame coverage. pkawa::handle_trailer (lib/src/protocol/mux/pkawa.rs:1038) is plumbed through the same hook chain (steps 3-7) so trailer headers do not bypass listener-scoped policy. Concrete plumbing shape (a second invocation with a trailer-flag in WakeupReason, vs. a separate hook entry-point on HeaderHook) is settled at Phase B implementation time.

Response side (reverse):

1. Backend response received
2. cluster hook chain (response variant, reverse iteration)
3. frontend hook chain (response variant)
4. listener-post-route hook chain (response variant)
5. Hook output validation (same byte-validation gate as request side)
6. Invariant re-validation (response variant)
7. ParserCallbacks::on_response_headers + existing config-time response edits
8. kawa.prepare → wire bytes

Outcome routing:

• First Continue accumulates header edits and proceeds. Header-edit ordering across scopes is deterministic last-writer-wins in dispatch order (listener → frontend → cluster on request, reversed on response).
• First Reject(status) short-circuits via set_default_answer (mux/answers.rs:193) — or set_default_answer_with_retry_after (answers.rs:207) for 429 specifically, threading SessionManager::effective_retry_after (server.rs:243). Reject is valid only BEFORE response bytes have been emitted. A Reject on the response side after :status has been queued is converted to AbortStream { error: Internal } and a warn! is logged so the hook author sees the misuse.
• First AbortStream { error } aborts an in-flight response. H2: emit RST_STREAM with the operator's chosen H2ErrorCode. H1: write-only shutdown (NEVER Shutdown::Both on TLS frontends — write-only shutdown avoids the TCP RST that truncates already-queued response bytes).
• First Pending(...) parks the session via the wake-up registry (Phase C). Subject to per-worker caps (§4.5).

2.8 Panic-safety contract

&mut HookMut exposed to a hook is not naturally UnwindSafe (it wraps interior pointers, kawa storage). The dispatcher wraps every hook call in catch_unwind. On Err, the dispatcher MUST:

1. Clear ctx.headers_request and ctx.headers_response post-route snapshots (the panicking hook may have left them half-populated).
2. Drop ctx.hooks_state entirely (the panicking hook may have left dangling invariants in its own slot).
3. Emit hooks.panicked.<hook_name> (rate-limited counter — operators see drift; the worker stays up).
4. Route to set_default_answer with status 500.
5. Mark the session for close after the default answer is written.

Phase A includes a regression test that panics from a HeaderHook mid-mutation and asserts no partial bytes reach the wire. Workspace Cargo.toml commits to panic = "unwind" so a future change is a deliberate decision the hook subsystem can react to. "Worker should not crash" is the load-bearing operator posture.

---

3. Per-stream state slot — semantics + cost

3.1 Lookup pattern

Each hook stashes state at TypeId::of::<MyHookPrivateState>() and reads it back at the same key. Two hooks with different nominal state types occupy different slots; common-type collisions (Vec<u8> × Vec<u8>) are an operator footgun (see §2.4 "State-discipline contract" — newtype mandatory; the registry warns on TypeId duplication at registration time).

References into per-stream kawa storage MUST NOT be stashed; only owned bytes (a Vec<u8> newtype, a String newtype, etc.). Buffer-pool slots are per-stream; aliasing across streams is undefined behaviour. The define_hook_state! macro in lib/src/hooks/macros.rs enforces the newtype + the Send/Sync bound at compile time.

3.2 Cost analysis

Steady-state per request: ~5-10 hook calls (3 scopes × 1-3 middlewares). Each hook access does 1 TypeId comparison + 1 HashMap lookup + 1 downcast = single-digit nanoseconds. Compared to TLS handshake (~ms) or backend connect (~ms), invisible.

Memory: each stashed state boxed once. Sozune-typical mix (rate-limit verdict, auth result, compression negotiation) = 3-5 boxed allocations per request, ~100 bytes. Negligible.

When hooks are unused (stock sōzu, no Sozune-style binary): hooks_state: None — one extra usize on HttpContext, zero allocations.

3.3 Why TypeId beats alternatives

Alternative	Rejected because
Make HttpContext generic over Extra	Refactor every call site. Sozune wants several middleware-specific state types, not one big struct.
Single untyped Box<dyn Any> slot	Only one hook can use it. Two middlewares collide silently.
Pingora-style type CTX associated type	Sōzu's hook trait is dyn-dispatched. Associated types don't survive type erasure.
Slab-keyed slot ID (operator manages a numeric ID per hook)	Operator burden. TypeId map gets the same lookup cost without manual ID tracking; if profiling shows hot, swap the backing store.

---

4. Yield-on-pending wake-up subsystem (Phase C)

The most complex sub-system. Documented here in full because it touches the mio loop directly.

4.1 Token namespace

The worker's mio Poll currently allocates tokens for: front sockets (slab-keyed, low range), back sockets (slab + offset), command channel (fixed token), timer (fixed token), health-check probes (carved-out range [1<<24, 1<<24 + 1<<16), with the assertion at lib/src/health_check.rs:55-56 that session tokens stay capped well below 1<<24).

Hooks claim a fresh range:

const HOOKS_TOKEN_BASE: usize = 1 << 25;
const HOOKS_TOKEN_CAPACITY: usize = 1 << 16;  // 65 536 tokens

Allocation via a modulo allocator that skips in-flight tokens, refusing allocation rather than colliding. The wake-up registry caps active entries at min(HOOKS_TOKEN_CAPACITY / 2, hooks.max_concurrent_pending_per_worker) so a peer cannot exhaust the token range — refused allocations bump hooks.pending.refused_cap and the hook is rejected with HookError { status: 500 }. Verify before Phase A lands that max_connections * slab_entries_per_connection < (1 << 24) still holds with hooks adding pressure.

4.2 Registry

// lib/src/hooks/wakeup.rs
struct HookWakeupEntry {
    session_id: SessionId,
    stream_id: Option<StreamId>,    // Some on H2, None on H1 / TCP
    chain_position: ChainPosition,  // listener-post-route[i], frontend[j], cluster[k]
    direction: HookDirection,        // request | response
    hook: Arc<dyn HeaderHook>,
    /// The fd or channel handle registered on the Poll. Owned here
    /// so its lifetime is bounded by the registry; `Drop` runs
    /// `EPOLL_CTL_DEL` (for fds) or `disconnect` (for channels)
    /// before the source is closed, eliminating fd-reuse races.
    source: WakeupSourceHandle,
    /// Identifier surfaced on re-entry as
    /// `WakeupReason::Ready { source }` so hooks issuing multiple
    /// Pendings across phases can disambiguate.
    source_id: WakeupSourceId,
    deadline: Instant,
    wake_timeout_status: RejectStatus,
}

pub struct HookWakeupRegistry {
    entries: HashMap<Token, HookWakeupEntry>,
    /// Reverse index: per-(session_id, stream_id) so RST_STREAM and
    /// session-close sweeps run in O(active-pending-for-stream)
    /// without walking all entries.
    by_stream: HashMap<(SessionId, Option<StreamId>), SmallVec<[Token; 4]>>,
    next_token: Token,  // modulo allocator
    active_count: AtomicUsize,
    max_active: usize,
}

4.3 Re-entry path

When the mio loop returns an event with a token in the hooks range:

1. Look up the HookWakeupEntry. If absent (already cancelled by RST_STREAM or session close), drop the event without bumping any counter.
2. Resolve (session_id, stream_id) against the slab. If gone (session closed mid-pending), drop the entry + bump hooks.cancelled_pending.
3. If alive: borrow &mut HttpContext, re-invoke the hook with WakeupReason::Ready { source: source_id } (or Timeout { source } from the timer path). The hook may return:
   • Continue — proceed to the next chain step (or pipeline stage).
   • Reject(status) — convert via set_default_answer (answers.rs:193) for non-429 codes, set_default_answer_with_retry_after for 429. If response bytes have already been emitted (response-side wake), convert to AbortStream { error: Internal } and warn!.
   • AbortStream { error } — emit RST_STREAM (H2) or write-only shutdown (H1).
   • Pending(...) — re-register on a new token; deadline accumulates against the original timeout-cap. A second-Pending after Timeout re-entry is rejected with wake_timeout_status to bound starvation.

Edge-triggered epoll discipline. On any path that queues bytes for write — header-edit application, default-answer write, AbortStream RST emission — the dispatcher MUST call signal_pending_write on the readiness tracker before returning to the loop, per §1.4 invariant 4. Edge-triggered epoll does not deliver a wake-up event for free when readiness is computed from the read path; forgetting this produces "stuck session" bugs that only manifest at exact byte boundaries.

4.4 Cancellation

Three cancellation paths:

1. Session close. SessionManager::on_session_close calls HookWakeupRegistry::sweep_session(session_id), which uses the by_stream reverse index to find every entry for that session in O(active-pending-count-for-session) and drops them. Source handles drop → fds get unregistered from the Poll, channels disconnect their readiness signal. Counter hooks.cancelled_pending increments per swept entry.
2. H2 RST_STREAM on a hook-pending stream. mux::h2::handle_rst_stream (or the equivalent pkawa path) calls HookWakeupRegistry::sweep_stream(session_id, stream_id) synchronously, BEFORE the H2 flood detector glitch counter sees the close. Late-firing wakes for that stream return "already cancelled" at step 1 of §4.3 and drop without bumping glitch_count — preserving the existing H2FloodDetector posture (the detector currently counts WINDOW_UPDATE on closed streams as a glitch, so race conditions between hook resume and stream close must be cleaned synchronously).
3. Channel sender drop. When the operator's background thread panics and drops its mpsc::Sender, the registry detects Disconnected on the channel handle and re-enters the hook with WakeupReason::Timeout immediately (no need to wait for the configured timeout). The hook chooses to short-circuit or retry. hooks.channel.disconnected.<hook_name> increments.

4.5 Timeout enforcement

Every Pending carries timeout: Duration. The dispatcher caps this at hooks.max_pending_timeout_seconds (worker-config, default 5 s, hard ceiling 60 s) — operator-supplied timeouts above the cap are rejected with HookError { status: 500, message: "Pending timeout exceeds cap" } and hooks.pending.timeout_capped.<hook_name> increments. The worker's existing Timer machinery (the same one that drives front_timeout / back_timeout) registers a Timeout event at now + min(timeout, cap).

If the Timeout fires before a Ready wake, the hook is re-invoked with WakeupReason::Timeout { source }. If it returns Pending again, sōzu rejects with wake_timeout_status to bound starvation — the configured fallback. Per-worker concurrent-Pending count is also capped at hooks.max_concurrent_pending_per_worker (default ≤ 32 768) to keep headroom inside the 65 536-token range; exceeding the cap returns HookError { status: 500 } at registration without entering Pending state.

4.6 H2 stream ordering invariant

Per-stream pending state lives on the H2 connection's Stream slot in mux::stream. Hook-pending is a flag on Stream, NOT a new variant of StreamState. The actual StreamState enum at lib/src/protocol/mux/stream.rs:35 is a linkage state machine (Idle / Link / Linked(Token) / Unlinked / Recycle); RFC 9113 §5 protocol state lives elsewhere (front_received_end_of_stream, back_received_end_of_stream, per-stream readiness on the H2 Connection). Adding HookPending to StreamState would lose the backend Token carried by Linked(Token).

The shape:

// lib/src/protocol/mux/stream.rs (extended)
pub struct Stream {
    pub state: StreamState,
    // ... existing fields ...
    pub hook_pending: Option<HookPendingTag>,
}

pub struct HookPendingTag {
    pub token: Token,                      // index into HookWakeupRegistry
    pub direction: HookDirection,
    pub started_at: Instant,
}

Transitions: hook_pending = Some(_) on HookOutcome::Pending; cleared on hook re-entry returning anything other than Pending; cleared on RST_STREAM / session close. StreamState itself is untouched — Linked(backend_token) still carries its backend identity through a hook-pending window.

The connection-level scheduler skips streams with hook_pending.is_some() and rotates to ready streams. Response order is preserved by the existing FIFO bookkeeping that sequenced :status emission per RFC 9113 §5.

LIFECYCLE.md gets a new section: "Hook-pending streams and the response-order invariant" — written against the actual StreamState enum (linkage states), NOT the RFC-9113 abstract names.

4.6.1 Read-side flow control while hook-pending

While a request-side hook is parked (hook_pending.is_some()), the connection MUST NOT WINDOW_UPDATE the client for that stream and MUST NOT forward DATA payloads to a backend (which may not yet be connected — request hook chain fires after route_from_request but before backend connect, per §2.7). Without this gate, a hostile (or merely fast) client streaming the request body during a hook park is a memory-pressure DoS surface multiplied by concurrent connections.

Concretely:

• mux::h2::handle_data_frame (lib/src/protocol/mux/h2.rs:4264) treats hook-pending streams as suspended for forwarding; received DATA accumulates against max_request_body_size and trips 413 on overflow. No WINDOW_UPDATE is emitted.
• After resume (Continue clears hook_pending), the dispatcher emits the deferred WINDOW_UPDATE and resumes the read path.
• After resume (Reject / AbortStream), the read path closes via the standard short-circuit; deferred bytes are discarded.
• For response-side hook_pending, the read path (request body) is unaffected — only the write path is suspended.

4.7 Channel-fed background-thread pattern

The canonical I/O workaround for "I need to consult Redis":

1. Operator's Sozune binary spawns a background thread holding the Redis connection.
2. Background thread maintains an in-memory snapshot (Arc<RwLock<BanList>>).
3. When a hook needs the snapshot fresh, it sends a request via a mpsc::Sender, then returns Pending { wake: WakeupSource::ChannelReady(channel_token), timeout: Duration::from_secs(1), wake_timeout_status: RejectStatus::SERVICE_UNAVAILABLE }.
4. Background thread queries Redis, sends the verdict back via the channel.
5. Hook re-enters with WakeupReason::Ready { source }, reads the verdict from ctx.hook_state::<RedisVerdict>(), emits Continue or Reject.

Sender-drop semantics. If the background thread panics and the Sender half drops, the registry detects Disconnected on the channel handle and re-enters the hook with WakeupReason::Timeout immediately — no waiting for timeout to elapse. The hook then chooses to short-circuit or retry. hooks.channel.disconnected.<hook_name> increments. Phase C ships an e2e cell "background-thread-panics-during-pending" to pin this contract.

Sozune's middleware crates would ship this pattern as a generic utility (SozuneAsyncResolver<T>) so individual middlewares don't reimplement it. Documented as an example in doc/programmable-hooks.md, NOT shipped inside sozu-lib.

4.8 What's NOT in scope of Phase C

• Per-listener thread-pool for hook I/O (would break lib/ no-async by introducing a tokio runtime). Operators doing thread-pool I/O run their pool in their Sozune binary, send results via the channel pattern above.
• Cross-worker coordination of pending state (each worker's wake-up registry is independent). A pending hook on worker 1 doesn't block worker 2; same as sōzu's existing per-worker isolation.

[FlorentinDUBOIS]

Continuation of the design proposal above — this comment carries the implementation phasing, PR sequencing, risk register, items requiring resolution before implementation starts, and cross-references. The two posts together form the complete self-contained proposal.

---

5. Phased implementation plan

Six phases, sequenced. Phases A → B → C are the critical path; D/E/F can parallelise after C.

Phase A — Trait module + registry skeleton (no dispatch)

Goal: public surface compiles, registered hooks are inert, stock sōzu binary unchanged.

Lands:

• lib/src/hooks/{mod.rs, outcome.rs, registry.rs, macros.rs} — trait + types + per-listener registry + define_hook_state! macro.
• lib/src/hooks/mod.rs::log_module_context! — HOOK tag, mirroring the MUX-ROUTER overload pattern at mux/router.rs:46-68 (() and ($http_context) arms calling HttpContext::log_context()). Direct log::*! calls inside lib/src/hooks/ are forbidden by lib/tests/log_layout.rs; new entry added to that test's allowlist OR the new module joins the canonical envelope.
• HttpContext.hooks_state field (Vec<(TypeId, Box<dyn Any + Send + Sync>)>, lazy None) + 4 convenience methods + hand-rolled impl Debug for HttpContext (replaces the existing #[derive(Debug)] at editor.rs:115; lists every existing field by name and renders hooks_state count-only). Update the 5 known construction sites (mux/h2.rs:7422, mux/answers.rs:358, mux/mod.rs:494, kawa_h1/editor.rs:833 + :862, kawa_h1/mod.rs:314).
• HookView / HookMut sealed wrappers over HttpContext (per §2.4 field-mutability matrix).
• HttpListener::with_hook(name, Arc<dyn HeaderHook>) + with_body_hook builder methods. Operator-supplied name() is required (no default).
• Proto: RequestHttpFrontend.hook_ids = 16, Cluster.hook_ids = 16 (verify next available tag at edit time). Run cargo clean -p sozu-command-lib once after the proto edit.
• Config-load validation + AddHttpFrontend runtime validation: every hook_id resolves in the listener's registry. Cluster-on-listener-registry validation site is the open question deferred to a dedicated session before this phase starts (§8.1 item 2).
• Workspace Cargo.toml: commit to panic = "unwind" on the relevant profiles (default is unwind on Linux x86_64; this commits to it explicitly so the hook subsystem's catch_unwind works in release and a future change to panic = "abort" is a deliberate decision the hook subsystem can react to). The hook dispatcher wraps every hook call in catch_unwind (Phase B); panic-recovery contract is in §2.8 (rolled-back state, 500 default-answer, session close).
• lib/src/hooks/macros.rs::define_hook_state! — generates a hook-private newtype + impl Debug (skips bytes) + Send/Sync bound. Documented as the only supported way to declare hook state.

Excluded: dispatch, wake-up, body hooks, builtins.

Tests:

• 4 unit tests on the registry (insert, duplicate, lookup-by-id, validation rejects unknown ID + emits typed error with the offending ID).
• 1 unit test pinning format!("{:?}", ctx) is panic-free with stashed Box<dyn Any> state.
• 1 unit test on define_hook_state! macro expansion + Debug skip behaviour.
• 1 unit test on the HOOK log envelope passing lib/tests/log_layout.rs.

Estimate: ~500 LoC (incl. hand-rolled Debug, macros, HookView/HookMut, log macro). ~1.5 days. One commit feat(hooks): trait module + per-listener registry skeleton. Updates CLAUDE.md (HOOK tag in the canonical log-context list, hooks feature flag), CHANGELOG.md (first ## [Unreleased] block, plain ### Added), and doc/configure.md (no metrics yet, but hooks.max_pending_timeout_seconds / hooks.max_concurrent_pending_per_worker config knobs are reserved here for Phase C). Panic-safety contract details live in §2.8.

Phase B — Sync dispatch (Continue / Reject / AbortStream)

Goal: hooks fire on every request, can mutate HookMut-exposed HttpContext fields, can short-circuit with a status code, can abort an in-flight response. HookOutcome::Pending is rejected at hook registration time with HookRegisterError::PendingRequiresPhaseC if the hook impl advertises Pending capability — the Pending variant compiles in the trait surface but cannot actually fire in PR1 (operator footgun avoidance — shipping a Pending that silently means 503 violates "no panic on network-facing input" in spirit).

Lands:

• lib/src/hooks/dispatch.rs — request + response dispatch helpers, called from Router::connect (NOT from inside route_from_request). Single dispatch site shared by H1 and H2 through the unified mux entry point; per-cluster metrics fire from the dispatcher only, satisfying H1/H2 metric parity by construction.
• Router::connect extension: listener-post-route → frontend → cluster chain on request, reverse on response.
• Hook-output validation gate: every Header emitted by a hook is byte-validated by the helper promoted from command/src/config.rs:1397-1408 (parse_header_edit's name + value validators) into a shared crate location. Invalid bytes → HookError { status: 500 }, bump hooks.invalid_header.<hook_name>. Phase B test: CRLF in hook-emitted header → 500 on the wire, no \r\n bytes leak.
• Invariant re-validation step: post-hook CL/TE consistency, Host single-value, no forbidden bytes. Mandatory whenever the chain emitted any header edit.
• HPACK budget accounting: per-hook header-byte additions tracked; soft cap hooks.max_emitted_header_bytes_per_chain (default 2 KB total). Exceeding → HookError { status: 500 }, bump hooks.headers.bytes_added.<hook_name> and hooks.headers.budget_exceeded.<hook_name>.
• Trailer-frame coverage: hook chain plumbed through pkawa::handle_trailer (lib/src/protocol/mux/pkawa.rs:1038) so trailer headers do not bypass listener-scoped policy.
• catch_unwind wrap per hook call + the §2.8 rollback path (clear post-route header snapshots, drop hooks_state, route to set_default_answer(500), close session).
• First-Reject(status) routing through set_default_answer (mux/answers.rs:193); set_default_answer_with_retry_after (answers.rs:207) for 429 only, threading SessionManager::effective_retry_after (server.rs:243).
• First-AbortStream { error } routing: H2 emits RST_STREAM with the operator's chosen H2ErrorCode; H1 write-only shutdown (NEVER Shutdown::Both on TLS frontends).
• Per-hook timing histograms (hooks.duration.<hook_name>) + chain-level cumulative hooks.chain.cumulative_duration (alarms on chain-level p99/p999, default 5 ms budget — exceeding bumps hooks.chain.budget_exceeded and short-circuits with HookError { status: 503 }).

Tests: 4-cell protocol_pair_matrix! × 5 scenarios (no-op hook, header-injection hook, reject-with-403 hook, abort-stream-mid-response hook, CRLF-in-emitted-header → 500 hook) = 20 e2e cells. All TCP-response assertions use loop_read_until_eof per project CLAUDE.md. Plus 5 unit tests on the dispatch helper, the RejectStatus TryFrom rejection path, the byte-validator promotion, the panic-recovery rollback, and the chain-budget short-circuit.

Estimate: ~1000 LoC (the rollback path, byte validator, HPACK accounting, and trailer plumbing add ~200 LoC over the original estimate). ~2 days. One commit feat(hooks): sync dispatch with Continue/Reject/AbortStream; PendingRequiresPhaseC. Updates CHANGELOG.md (first ## [Unreleased] block, plain ### Added) and doc/configure.md (hooks.duration.<name>, hooks.invalid_header.<name>, hooks.headers.bytes_added.<name>, hooks.headers.budget_exceeded.<name>, hooks.chain.cumulative_duration, hooks.chain.budget_exceeded, hooks.panicked.<name>).

Merge target is the integration branch agreed at PR-creation time. Use plain git merge --ff-only <branch> from that target.

Phase C — Yield-on-pending wake-up subsystem

Goal: hooks can return HookOutcome::Pending(...) and sōzu re-enters them when the wake source fires, respecting H2 stream ordering, session-close + RST_STREAM cancellation, the read-side flow-control gate, and per-worker resource caps.

Lands:

• lib/src/hooks/wakeup.rs — registry + Poll integration + cancellation (3 paths: session close, RST_STREAM, channel sender-drop) + timeout enforcement (capped per worker) + per-worker concurrent-Pending cap.
• mio token namespace [HOOKS_TOKEN_BASE, HOOKS_TOKEN_BASE + HOOKS_TOKEN_CAPACITY) (= [1<<25, 1<<25 + 1<<16)); active_count capped at min(HOOKS_TOKEN_CAPACITY / 2, hooks.max_concurrent_pending_per_worker).
• WakeupSource::FdReadable(OwnedFd) (transfers ownership; EPOLL_CTL_DEL on WakeupSourceHandle::Drop before fd close).
• WakeupReason::{Ready { source }, Timeout { source }} + WakeupSourceId plumbed through the registry so hooks issuing multiple Pendings can disambiguate.
• Stream.hook_pending: Option<HookPendingTag> flag (NOT a new StreamState variant — StreamState stays Idle/Link/Linked(Token)/Unlinked/Recycle). Scheduler skips streams with hook_pending.is_some(). Backend Token carried by Linked(Token) is preserved across the pending window.
• Read-side flow-control gate (§4.6.1): mux::h2::handle_data_frame (h2.rs:4264) treats hook-pending streams as suspended for forwarding; received DATA accumulates against max_request_body_size and trips 413 on overflow. No WINDOW_UPDATE while parked. Deferred WINDOW_UPDATE emits on resume.
• RST_STREAM cancel path: mux::h2::handle_rst_stream calls HookWakeupRegistry::sweep_stream(session_id, stream_id) synchronously BEFORE flood-detector glitch counting, so late wakes return "already cancelled" and do NOT bump glitch_count (preserving H2FloodDetector's existing closed-stream-WU-as-glitch posture).
• Channel sender-drop detection: registry monitors Disconnected on registered channel handles; re-enters hook with WakeupReason::Timeout immediately on disconnect.
• LIFECYCLE.md update — "Hook-pending streams and the response-order invariant" section, written against the actual StreamState enum (linkage states), NOT the RFC-9113 abstract names.
• Sozune-shaped utility: SozuneAsyncResolver<T> documented as an example in doc/programmable-hooks.md (NOT shipped in sozu-lib).

Tests: 8 e2e cells (h2-h1, h2-h2, plus 6 hot-paths) — hook-with-channel-wake, hook-timeout, session-close-during-pending, H2 stream-5-pending-doesn't-block-stream-7, peer-RST_STREAM-on-HookPending-no-flood-trip, background-thread-panics-during-pending, request-body-flowing-during-HookPending-respects-413, FdReadable-double-close-race-impossible. All TCP-response assertions use loop_read_until_eof. Plus 8 unit tests on the registry, scheduler, by-stream sweep, channel-disconnect detection, timeout cap, OwnedFd drop ordering, WakeupSourceId disambiguation, and the per-worker concurrent-Pending cap rejection path.

Risk: highest in this plan. Hot-path mio change. Three review passes (implementer + independent reviewer + final auditor); the same discipline applies to Phase B and Phase D given e2e cell counts and H2 framing risk. Re-run reviews after each fix commit until a clean pass surfaces no new finding — complex protocol fixes routinely expose adjacent issues.

Estimate: ~1700 LoC (read-side flow-control gate + by-stream reverse index + sender-drop detection + per-worker caps add ~200 LoC over the original). ~3.5 days. Two commits — feat(hooks): wake-up subsystem + test(hooks): yield-on-pending e2e + LIFECYCLE.md. Updates CHANGELOG.md (first ## [Unreleased] block) and doc/configure.md (hooks.cancelled_pending, hooks.pending.refused_cap, hooks.pending.timeout_capped.<name>, hooks.channel.disconnected.<name>). Merge target is the integration branch agreed at PR-creation time.

Phase D — Body-stage opt-in (BodyHook)

Goal: hooks that opt in via with_body_hook(...) see request / response body chunks before they hit the wire.

Lands:

• BodyHook trait extending HeaderHook.
• BodyChunk { bytes: &mut [u8], end_stream: bool } shape.
• H1 framing: kawa chunk-encoded payload mutation before kawa.prepare. Buffer slot lower bound is 16393 bytes when H2 is enabled (CLAUDE.md gotcha); a Phase D unit test pins this.
• H2 framing: DATA frame interception in the post-prepare collector path on the H2 write side (mux::h2::flush_stream_out, called from write_streams at h2.rs:2255). Highest framing risk. Must respect frame size limits, flow control credits, END_STREAM ordering. Post-mutation byte total drives outbound credit accounting (NOT the original payload length).
• Length reconciliation. Hooks may shrink a chunk in-place (cannot grow beyond the buffer-pool slot). On shrink, sōzu auto-recomputes Content-Length from the post-hook total on H1 (or marks the response Transfer-Encoding: chunked and recomputes if the original CL was authoritative on a different hop) before forwarding. H2: post-mutation DATA-frame size drives the outbound flow-control debit. END_STREAM is NEVER swapped under the operator's feet — final-flag preservation guards in the converter remain authoritative. Defends against the CL.TE-style smuggling primitive that a naive shrink without recomputation creates.
• Buffer-pool integration: in-place mutation, size cannot grow beyond slot.
• Body-side AbortStream { error } routing: H2 emits RST_STREAM with the operator's chosen error code; H1 write-only shutdown.

Tests: 4-cell × 4 scenarios (body header-strip on request, body bytes-mutation on response, body-shrink-with-Content-Length-auto-recompute, body-stage-Reject-after-header-Continue-cleans-up-state) = 16 e2e cells. All TCP-response assertions use loop_read_until_eof. Plus 6 unit tests on the H2 DATA-frame path (frame size, credit accounting, END_STREAM preservation, post-shrink CL coherence, body-stage RST_STREAM emission, fuzz target on the hook-mutated chunk path).

Estimate: ~700 LoC (length reconciliation adds ~100 LoC). ~2.5 days. One commit feat(hooks): body-stage opt-in BodyHook. Updates CHANGELOG.md (first ## [Unreleased] block).

Phase E — Built-in middleware ports

Goal: sōzu's existing config-time programmability (basic auth, redirect, rewrite, header-edit, rate-limit) is reimplemented as HeaderHook impls in lib/src/hooks/builtins/. Sozune can compose them, override them, or replace them with custom impls.

Lands: five sub-modules under hooks::builtins::, each gated behind a feature flag (hooks-builtin-*):

Sub-module	Wraps
basic_auth.rs	cluster.authorized_hashes + www_authenticate
header_edit.rs	Vec<Header> (request + response variants)
rate_limit.rs	per-(cluster, IP) cap
redirect.rs	RedirectPolicy::{Permanent, Found, PermanentRedirect, Unauthorized}
rewrite.rs	rewrite_host / rewrite_path / rewrite_port

The existing config-driven path stays the default. A migration path lets operators flip a frontend / cluster from "config-time policy" to "hook-driven policy" by setting hook_ids = ["basic-auth"] instead of required_auth = true. Both shapes coexist.

Tests: existing e2e tests for each feature (e.g. redirect_rewrite_auth_tests.rs) gain a parallel test pointing at the hook-driven path. Same fixtures, same assertions.

Estimate: ~800 LoC across five sub-modules. ~2 days. Five commits, one per built-in.

Phase F — Operator documentation

Goal: operator-facing docs. Per "Docs stay current," each preceding phase already updated CHANGELOG.md, doc/configure.md, LIFECYCLE.md, and CLAUDE.md in the SAME PR — Phase F holds the operator guide only.

Lands:

• doc/programmable-hooks.md — operator guide. Sections: trait surface, builder API, HookOutcome semantics (Continue / Reject / AbortStream / Pending), RejectStatus constraints, hook output byte-validation contract, the field-mutability matrix (HookView/HookMut), the state-discipline newtype rule with the define_hook_state! macro, the channel-fed background-thread pattern (SozuneAsyncResolver<T> example), per-stream state slot semantics, panic-safety contract (§2.8), opt-out for stock builds, migration from config-time policy.
• doc/upgrade/1.x-to-2.0.md addendum (or a fresh 1.x-to-3.0.md if hooks ship in v3.0) — note that hooks are additive; cross-version contract for hook_ids (degrade-open on older workers).
• CHANGELOG.md — final harmonisation pass (target the FIRST ## [Unreleased] block per the dual-Unreleased convention; per-phase entries already added in Phases A-E).
• e2e/COVERAGE.md matrix — new "hook scope" axis covering (listener-post-route × frontend × cluster) × (sync × yield-on-pending) × (header × body).
• RELEASE.md — note hooks ship behind --features hooks so binary-built operators (Sozune) opt in explicitly.

Estimate: ~2 days. One commit docs(hooks): operator guide + matrix axis + RELEASE note. Merge target is the integration branch agreed at PR-creation time.

---

6. PR sequencing

Three pull requests, each independently reviewable + mergeable. Each PR opens only after maintainer approval of the corresponding phase; the merge target is the integration branch agreed at PR-creation time. Commit messages and PR descriptions inline the technical rationale and do not reference "Phase A/B/C" or "see plan" — those labels are private to this document and would not survive review without context.

PR	Phases	Scope	Reviewer focus
1	A + B	Trait, registry, sync dispatch (Continue / Reject / AbortStream); Pending rejected at registration.	Public API surface + dispatch correctness + byte-validation gate + panic-safety contract
2	C	Wake-up subsystem with read-side flow-control gate, RST_STREAM sweep, channel sender-drop detection, OwnedFd, per-worker caps. Solo PR because of the H2 scheduler integration + new Stream.hook_pending field.	mio integration + H2 stream ordering + flood-detector interaction
3	D + E + F	Body hooks (with length reconciliation), builtin ports (gated behind hooks-builtin-* feature flags), operator docs. Body hooks ship after the wake-up subsystem so body-stage Pending works from day one.	H2 framing + length reconciliation + operator guide

Total: ~7-8 commits across 3 PRs, ~5000 LoC (incl. byte-validation gate, panic-recovery, read-side flow-control, RST sweep, length reconciliation), ~50 new tests.

Calendar: 3-4 weeks of focused work, contingent on review pace.

---

7. Risk register

Risk	Severity	Mitigation
Hook panics inside the worker event loop	High	Workspace Cargo.toml commits to panic = "unwind". Dispatcher wraps every hook call in catch_unwind. On panic: clear post-route header snapshots, drop hooks_state, bump hooks.panicked.<hook_name> (rate-limited), route to set_default_answer(500), mark session for close. Phase A regression test asserts no partial bytes reach the wire after a mid-mutation panic. Documented operator contract: hooks SHOULD NOT panic; sōzu defends so the worker stays up. See §2.8.
H2 stream ordering broken by hook-pending streams	High	Stream.hook_pending flag (NOT a new StreamState variant); read-side flow-control gate per §4.6.1; LIFECYCLE.md invariant; e2e cells cover stream-5-pending-doesn't-block-stream-7 + RST-on-HookPending-no-flood-trip + body-flowing-while-parked-respects-413; auditor pass on Phase C.
Hook-emitted headers contain CRLF/NUL/non-tchar bytes (response splitting, cache poisoning)	High	Promoted byte-validator from parse_header_edit runs at the dispatcher (§2.7 step 6). Invalid bytes → HookError { status: 500 }. Phase B test pins no \r\n reaches the wire.
HookOutcome::Reject { status: u16 } accepts arbitrary codes (200 / 1xx / 6xx)	High	RejectStatus newtype with TryFrom<u16> enforcing 4xx/5xx (and 301/302/308). Out-of-range surfaces as HookError { status: 500 } and bumps hooks.invalid_reject_status.<hook_name>.
Pseudo-header tampering after routing	High	HookView / HookMut field-mutability matrix. Routing-decided fields (cluster_id, backend_id, authority, path, method, tls_*) are read-only to hooks. Edits flow through headers_request / headers_response and re-validate.
Hook-emitted headers bypass HPACK encode budget	High	Per-hook header-byte tracking + 2 KB chain-level cap (hooks.max_emitted_header_bytes_per_chain). Exceeding fails closed before HPACK abort, bumping hooks.headers.budget_exceeded.<hook_name>.
Trailer-frame spoofing bypasses listener-scoped hook policy	High	Hook chain plumbed through pkawa::handle_trailer:1038; trailer headers see the same chain as initial HEADERS. Phase B test: trailer-injected Authorization blocked.
Body-hook framing bug corrupts H2 DATA frames or smuggles via Content-Length mismatch	High	Phase D length-reconciliation auto-recomputes Content-Length on shrink (or switches to chunked); end_stream guards remain authoritative; framing test matrix; auditor pass; fuzz target on the hook-mutated chunk path.
HookOutcome::Pending with unbounded timeout pins per-stream resources (DoS amplification)	High	hooks.max_pending_timeout_seconds worker-config (default 5 s, hard ceiling 60 s); hooks.max_concurrent_pending_per_worker cap (default ≤ 32 768 of 65 536 token range); refused allocations bump hooks.pending.refused_cap.
Wake-up token leak on session-close-during-pending or RST_STREAM-during-pending	Medium	Three-way cancellation (session close, per-stream RST, channel sender-drop) with O(1) reverse index. hooks.cancelled_pending counter; e2e regression tests for each path.
Slow hook starves the worker (single-threaded mio)	Medium	Per-hook hooks.duration.<hook_name> histogram + chain-level hooks.chain.cumulative_duration. 5 ms chain budget short-circuits with HookError { status: 503 } and bumps hooks.chain.budget_exceeded. p99 > 10 ms warn! on individual hooks.
Operator hook holds a Mutex and deadlocks the worker	Medium	Documented contract: hooks must not hold cross-worker locks. Latency histogram + chain budget surface drift before deadlock manifests.
Arc<RwLock<_>> read-guard held across route_from_request borrows starves unrelated streams	Medium	Documented contract: hooks must not hold long-lived guards while invoking sōzu APIs. State-discipline newtype encourages copy-on-read access.
TypeId collision via common state types (Vec<u8>, String)	Medium	State-discipline contract: newtypes mandatory; define_hook_state! macro generates them; registry warns when two hooks declare the same TypeId. Operator guide highlights the footgun.
Channel Sender drops mid-pending (background thread panics) → wake never fires	Medium	Registry monitors Disconnected on channel handles; re-enters hook with WakeupReason::Timeout immediately. Phase C e2e cell covers it.
ABI drift between sōzu releases breaks Sozune	Medium	Acceptable per Q5. Sōzu ships a stable 0.x until v1; Sozune pins sozu-lib versions. No facade layer.
Older sōzu binary deserialising hook_ids silently treats frontend as no-hook	Medium	Documented degrade-open contract for v1; future WorkerCapabilities advertisement could refuse-application instead.
Cluster-on-listener-registry validation site (Cluster is global, AddCluster has no listener)	Medium	Resolution deferred to a dedicated planning session before Phase A starts (§8.1 item 2); v1 keeps cluster scope per Q1.
Hook config validation rejects valid configs at hot-reload	Low	Validation runs at AddCluster / AddHttpFrontend against the registry; typed HookConfigError carries the offending ID.
Memory leak from unbounded hooks_state storage	Low	Vec<(TypeId, _)> is per-session; freed on session close. No cap needed.
RawFd ownership confusion / double-close race	Low	WakeupSource::FdReadable(OwnedFd) transfers ownership; EPOLL_CTL_DEL runs in WakeupSourceHandle::Drop before fd close. Operator must set O_CLOEXEC.

---

8. Open questions deferred to v2

These were considered and intentionally NOT in this plan's scope:

1. Cross-worker coordination of pending state. Each worker's wake-up registry is independent — same trade-off as max_connections_per_ip. Cross-worker sync (Redis broadcast or master-fanout) is a separate design pass, only worth doing when an operator hits the limitation in production.
2. Per-listener thread-pool for hook I/O. Would require a tokio runtime in lib/, breaking the no-async invariant. Operators using a thread pool host it in their Sozune binary, communicate via channels — same pattern as the Redis example.
3. Built-in compression / strip-prefix / backend-timeout middlewares. Sozune's middleware list (documentation/middleware/) goes beyond what sōzu has historically supported; the matching builtins (Phase E) cover the existing primitives only. Compression / strip-prefix / backend-timeout / etc. ship in Sozune's own crates linking the HeaderHook / BodyHook traits.
4. Hook hot-reload via dynamic-library loading. Explicitly out of scope per Q4 (Pingora-style binary build is the deployment model).
5. First-class WebAssembly hook runtime. Tracked in WebAssembly plugins #685, wirefilter integration #687. The Rust trait API is the immediate path; wasm could ship later as WasmHook impl of HeaderHook (the wasm runtime is the operator-supplied layer, sōzu just sees a HeaderHook impl).
6. True pre-route hook stage. Original issue [Question] Hook to rewrite requests and responses #1178 asked for "rewrite path/headers BEFORE routing." v1 ships hooks as listener-post-route (after route_from_request) because rewriting and re-routing is a deeper change to Router than the wave can absorb. v2 may add a pre-route stage; v1 operators who need this can pre-process at the listener level via custom config or accept the post-route limitation.

8.1 Open items requiring resolution before Phase A

These need a decision before code lands. They are NOT silent gaps — each is tracked in the §7 risk register row that names the relevant mitigation.

1. Sozune's confirmed v1 middleware list. Not yet confirmed at plan-edit time. If the list is "auth, headers, redirects, rewrites" (already covered by config-time policy), the case for Phase E builtins weakens. If it's "rate-limit with external state, mTLS-to-OAuth bridging, custom WAF rules," the Phase C wake-up subsystem is essential. Confirm before opening PR1.
2. Cluster-scope hook validation site. Cluster is global; AddCluster carries no listener context. Resolution of where cluster hook_ids get validated (worker-global registry vs. validation deferred to AddHttpFrontend time when a frontend pulls a cluster) is the subject of a dedicated planning session before Phase A. v1 keeps cluster scope per Q1; only the validation site is to be decided.
3. Trailer-frame plumbing concrete shape. §2.7 commits to plumbing the hook chain through pkawa::handle_trailer. The exact wiring (a second invocation of the dispatcher with a trailer-flag in WakeupReason, vs. a separate hook entry-point on HeaderHook) is settled at Phase B implementation time. Decision: plumb (per user q5).
4. Mid-body Reject semantics. Resolved by user q3: AbortStream { error } is the response-side abort path; Reject after response bytes are emitted is converted to AbortStream { error: Internal } with a warn! (operator misuse signal).
5. hook_ids cross-version contract. Resolved as v1 documented degrade-open. Future WorkerCapabilities advertisement is v2 work.
6. Per-stream HttpContext ownership. Project memory says each H2 stream owns a distinct HttpContext; one of the security findings was unable to fully verify the current state. 5-minute confirmation against lib/src/protocol/mux/stream.rs and mux/h2.rs:7422 (the H2 HttpContext construction site) before Phase A lands.
7. Panic strategy. Resolved by user q2: workspace Cargo.toml commits to panic = "unwind"; the dispatcher wraps every hook call in catch_unwind; the §2.8 rollback contract is mandatory. "Worker should not crash" is the load-bearing posture.

---

9. Cross-references

• Issue #1178 — original ask + maintainer reply + @Shine-neko's trait sketch.
• kemeter/sozune — deployment target (Sozune middleware list at documentation/middleware/).
• lib/src/protocol/kawa_h1/editor.rs — HttpContext definition. pub struct HttpContext at editor.rs:116; #[derive(Debug)] at editor.rs:115 (replaced by hand-rolled impl Debug in Phase A). Re-exported as crate::protocol::http::editor::HttpContext via protocol/mod.rs:21. Construction sites: mux/h2.rs:7422, mux/answers.rs:358, mux/mod.rs:494, kawa_h1/editor.rs:833 + :862, kawa_h1/mod.rs:314.
• lib/src/protocol/mux/router.rs::connect — dispatch entry point for Phase B (line ~90 → ~141), wraps route_from_request (line ~540). Both H1 and H2 reach connect through the unified mux dispatch path.
• lib/src/protocol/mux/answers.rs::set_default_answer (line 193) for non-429 codes; ::set_default_answer_with_retry_after (line 207) for 429 only.
• lib/src/protocol/mux/stream.rs:35 — actual StreamState enum (Idle/Link/Linked(Token)/Unlinked/Recycle); hook-pending is a separate Option<HookPendingTag> field on Stream, NOT a StreamState variant.
• lib/src/protocol/mux/pkawa.rs:1038 — handle_trailer; hook chain is plumbed through here for trailer-frame coverage.
• lib/src/protocol/mux/h2.rs:4264 — handle_data_frame; read-side flow-control gate lands here for hook-pending streams.
• lib/src/protocol/mux/h2.rs:5152 — closed-stream glitch counting; per-stream RST sweep runs synchronously before this path.
• lib/src/protocol/mux/LIFECYCLE.md — H2 stream state machine documentation; gains a "Hook-pending streams and the response-order invariant" section in Phase C (against the actual StreamState enum).
• lib/src/health_check.rs:55-56 — assertion that session tokens stay below 1<<24; verify still holds with hooks adding pressure.
• command/src/config.rs:1397-1408 — parse_header_edit byte-validator; promoted to a shared crate location and reused at the hook dispatcher.
• lib/src/metrics/mod.rs — metric macros (incr!, count!, gauge!, time!); update doc/configure.md when adding/renaming a public metric.
• Pingora ProxyHttp trait (pingora-proxy/src/proxy_trait.rs:32 in cloudflare/pingora) — reference design for the hook surface.
• HAProxy flt_ops (include/haproxy/filters-t.h:160-203 in haproxy/haproxy) — reference design for the per-stream state pattern.
• Pre-existing config-time programmability stays the user-facing default; hooks are additive.

---

Plan originally written 2026-05-02 in response to operator decisions on the six design questions. Revised 2026-05-02 to fold in 15 must-fix findings from a five-stream review (correctness, guidelines, simplification, security, second-model cross-check) plus operator decisions on panic strategy (panic = "unwind" workspace-wide), AbortStream for response-side reject, trailer-frame plumbing, and cluster scope retention. Outstanding items are tracked in §8.1; resolution required before Phase A starts. Implementation does NOT start until this revised plan is approved.