Serving: remaining work after batching and fused kernels (#93, #97) #107

#93 and #97 made a Kev container batch concurrent requests through fused Qwen3.5 kernels. Kev-4B on an H100 now answers 6 questions in 18 ms of model time and serves 121 decision-v7 records/s at 64 in-process clients (runs/fused-4b-h100/report.json). This issue collects what is left, roughly in priority order, with the evidence for each item.

1. Documents over 1,024 tokens are not batched in their state pass

A new state longer than GRAPH_STATE (1,024 tokens) gets its own eager state pass; only its question rows join the batch (DecisionModel.probs_batch). Five questions on a 2,200-token state stay at ~12 req/s on an H100 whatever the load (11.4-13.1 at 1-64 clients), and ~7 on an L40S.

• The state pass is compute-bound at that length. Batching several long states into one eager pass (left-padded, the mask states() already builds) would save the per-pass weight reads, but not much compute.
• Graphing it was slower than eager in Serve the Qwen3.5 family through CUDA graphs: warm requests in tens of milliseconds, not 60 ms per pass #90 (explicit mask, no causal flash attention: 223 vs 208 ms on an L40S). A causal-aware path (flash attention with per-row lengths) might change that.
• Measure with scripts/serving_bench.py (the "5 questions, 2,200-token state" throughput rows).

2. The HTTP front door caps a container well below what it can serve

Kev-4B, H100, closed-loop client in the same region (numbers in #97):

• Web endpoint. Per container it tops out at ~40-50 req/s. server-timing shows requests waiting outside the process, in Modal's input plane.
• Flash. Past ~32 concurrent requests per container it queues at the proxy (p99 ~4 s at 64 clients) and did not scale out in our runs. From zero it answered 503 no upstreams available for minutes, so the template keeps one container up. Modal marks it experimental. Ask Modal about Flash per-container concurrency and scale-out, and whether they sign off on it for production traffic, before relying on it for large customers.
• Also worth trying: HTTP/2 between the proxy and the container (h2_enabled=True), which uvicorn cannot serve (hypercorn can).
• Load test: /tmp-style closed-loop client from a Modal function in the endpoint's region (the one used in kev-deploy: batched serving on the #93 kernels, 64 concurrent inputs per container, opt-in Flash front door #97; worth committing as scripts/http_load.py).

3. Autoscaling and cold starts under bursts

• Burst behaviour. New containers take ~20-55 s with cached weights, and 1-2 min on a fresh volume (weight download, Triton compiles). The web endpoint's 128-client run fell to 52 req/s with p99 ~8 s while containers scaled.
• What to decide:
  • KEV_MIN_CONTAINERS and target_inputs guidance per traffic shape;
  • Modal GPU memory snapshots to cut the cold start;
  • whether warm-up should replay a sample of the customer's real requests, so their batch shapes are captured before traffic.

4. GPU time per batch

A 16-request, 6-question batch on an H100 takes ~127 ms of GPU time. About 40% is GEMMs, and the rest is mostly fla's delta-rule kernels, plus the per-row copy of each state's fp32 DeltaNet state from the bank.

• FP8 (e4m3 W8A8, rowwise): measured and rejected. Against fp32 it moved probabilities by up to 0.074 (mean 0.012, 6x bf16) and changed 3 of 120 answers. Only revisit with a calibration-aware evaluation.
• Token-by-token delta rule for short rows (fused_recurrent): no gain over chunked (149 vs 146 ms).
• Open:
  • fla kernels that read each row's initial state from the bank by index, instead of a gathered copy (vLLM's fla fork has state indices);
  • bf16 state storage (a parity risk).

5. Robustness and maintenance

• fla pin. kev.fused_qwen35 patches fla 0.5.2's NB-keyed kernel launches (_FixedNB) and refuses any other version. Upgrading fla means re-checking those kernels (conv, gated norm, l2norm, small-D layer norm) and re-running serving_bench parity.
• Graph variety. A busy server met ~160-210 distinct graph buckets on mixed traffic (GRAPHS_KEPT=256). Each new bucket runs eagerly first (~50-90 ms) and is captured when idle or once it recurs.
• kev-finetune's serving class (skills/kev-finetune/scripts/kev_modal.py) still pins an older KEV_REF: no graphs, no batching, no fused kernels. Moving it needs its own end-to-end fine-tune test.
• Deploy template. Bump skills/kev-deploy's KEV_REF after any serving change it should pick up.

Not planned

• vLLM (Serve Kev on Modal through vLLM: concurrent requests batch in one engine, and a request's rows share its state through vLLM's prefix cache #88): measured head to head in Serve Kev on Modal through vLLM: concurrent requests batch in one engine, and a request's rows share its state through vLLM's prefix cache #88's thread. vLLM matched on short-request throughput but was 3-12x slower per request and 4x slower on long documents, and it needs a second torch stack. Left open for reference.