Modal: bypass input queueing with a 64-concurrency direct server

AGENTS.md

@@ -329,7 +329,8 @@ Title Case sections, API tables, Authors + License); model cards are formal.
     Measure with `uv run modal run modal_app.py::serving --run <hub id> --gpu <GPU> --name <name>` (`scripts/serving_bench.py`: latency,
     parity vs fp32, throughput at 1/8/32/64 in-process clients, `--flags=--isolation` for question isolation on the served path; reports in `runs/serving-*` (graphs only), `runs/fused-*` and, for the current checkpoints and code, `runs/serve-*` / `runs/grouping-4b-h100` / `runs/fused-27b-*`, which the README serving table reads).
     `tests/test_model.py::test_cuda_graphs_match_eager` needs CUDA (run it on Modal). Over HTTP, Modal's `asgi_app` path caps a container at
-    ~40-50 req/s; `modal.experimental.http_server` served ~99 req/s at 64 clients (Kev-4B, H100).
+    ~40-50 req/s in earlier measurements; the current direct Modal Server and rejected higher-limit web endpoint are paired
+    by `scripts/modal_http_bench.py` (`runs/modal-http-4b-h100/report.json`: HTTP/process/model times, errors, kernel set).
     Loading merges the fp32 adapter straight into bf16 weights (same bits as the old fp32 merge + cast), so Kev-9B needs ~17 GB, not 36 GB.
 - Extra endpoints for the demo: `POST /v1/systemone/permute` (one Choice under n option orders), `POST /v1/systemone/separate`
   (each question alone; packed-vs-separate comparison). `/v1/systemone` also returns `latency_ms`.
@@ -361,7 +362,7 @@ Repo skills (`.agents/skills`, tracked in git):
 Installed from other repos by `npx skills add` and pinned in `skills-lock.json` (`deslop`, `unslop` from cursor/plugins,
 `grill-me` from mattpocock/skills); `.agents/skills/modal/` is gitignored and reinstalled with `uv run modal skills install`.
 
-Published from this repo: `skills/kev-deploy` (`npx skills add jaredpalmer/kev@kev-deploy`), one self-contained Modal file (`scripts/kev_serve.py`, pinned `KEV_REF`, GPU list per released model from `runs/serve-*` / `runs/fused-*`: 0.8B L4, 4B L40S, 9B H100, 27B B200 (H200, H100); CUDA graphs captured at start (`server.wait_idle()`); 64 concurrent inputs per container, autoscaling at 32; optional `KEV_REGION`; `KEV_FLASH=1` = Modal's experimental direct HTTP server, needs `KEV_REGION`, keeps one container up; fla pinned to 0.5.2; bearer key via a deploy-time Modal secret) that serves a released Kev checkpoint as a System One endpoint in one `modal deploy`; bump its `KEV_REF` after a serving change it should pick up. And `skills/kev-finetune` (`npx skills add jaredpalmer/kev@kev-finetune`): the user-facing fine-tuning skill (SKILL.md for agents,
+Published from this repo: `skills/kev-deploy` (`npx skills add jaredpalmer/kev@kev-deploy`), one self-contained Modal file (`scripts/kev_serve.py`, pinned `KEV_REF`, GPU list per released model from `runs/serve-*` / `runs/fused-*`: 0.8B L4, 4B L40S, 9B H100, 27B B200 (H200, H100); CUDA graphs captured at start (`server.wait_idle()`); 64 concurrent web inputs per container, autoscaling at 32; optional `KEV_REGION`; `KEV_FLASH=1` = direct HTTP via `app.server`, target 64 per container, needs `KEV_REGION`, keeps one paid container up, tested with Modal 1.5.5; fla pinned to 0.5.2; bearer key via a deploy-time Modal secret) that serves a released Kev checkpoint as a System One endpoint in one `modal deploy`; bump its `KEV_REF` after a serving change it should pick up. And `skills/kev-finetune` (`npx skills add jaredpalmer/kev@kev-finetune`): the user-facing fine-tuning skill (SKILL.md for agents,
 README.md is the human cookbook, `references/` the long-form docs). agentskills.io format (validate with `uvx --from skills-ref agentskills validate skills/kev-finetune`).
 Stdlib scripts: `extract_workload` (find Jev/TypeSafe call sites + labelled files, draft the spec), `convert_data` (CSV/JSONL -> records),
 `generate_data` (OpenAI-compatible endpoint), `plan_size` (paired McNemar sizing; `--from-result` post hoc), `split_data` (`--holdout` keeps real rows out of train).

runs/modal-http-4b-h100/deploy-smoke/report.json

@@ -0,0 +1,107 @@
+{
+  "script": "skills/kev-deploy/scripts/kev_serve.py",
+  "model": "jaredpalmer/kev-4b",
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "region": "us-east",
+  "mode": "direct",
+  "client": "session VM; not in-region; not comparable to the paired benchmark RTT",
+  "settings": {
+    "KEV_FLASH": "1",
+    "KEV_MIN_CONTAINERS": "0",
+    "effective_min_containers": 1,
+    "target_concurrency": 64
+  },
+  "protocol": {
+    "models": 200,
+    "invalid_bearer": 401,
+    "invalid_question": 422,
+    "sdk": true
+  },
+  "answers_equal_to_paired_benchmark": true,
+  "image_versions": {
+    "torch": "2.8.0",
+    "transformers": "5.19.0",
+    "peft": "0.21.2",
+    "flash-linear-attention": "0.5.2",
+    "triton": "3.8.0"
+  },
+  "warm_serial_requests": 20,
+  "p50": {
+    "ms": 99.66,
+    "process_ms": 23.15,
+    "model_ms": 18.4
+  },
+  "model_card": {
+    "models": [
+      {
+        "name": "kev-latest",
+        "description": "Kev pointer head on Qwen/Qwen3.5-4B-Base, serving jaredpalmer/kev-4b at temperature 2.41",
+        "release_date": "2026-10-01",
+        "run": "jaredpalmer/kev-4b",
+        "base": "Qwen/Qwen3.5-4B-Base",
+        "lora": 16,
+        "device": "cuda",
+        "backend": "torch",
+        "dtype": "bfloat16",
+        "temperature": 2.406050072164233,
+        "max_state_tokens": 65536,
+        "truncate_states": false,
+        "cuda_graphs": {
+          "captured": 6,
+          "kept": 6,
+          "pending": 0,
+          "failed": 0
+        },
+        "prefix_cache": {
+          "size": 4,
+          "min_state_tokens": 0,
+          "max_tokens": 65536,
+          "hits": 0,
+          "misses": 3,
+          "cached_states": 3,
+          "oom_retries": 0
+        },
+        "batches": {
+          "count": 3,
+          "requests": 3,
+          "queued": 0
+        }
+      },
+      {
+        "name": "jev-latest",
+        "description": "Kev pointer head on Qwen/Qwen3.5-4B-Base, serving jaredpalmer/kev-4b at temperature 2.41",
+        "release_date": "2026-10-01",
+        "run": "jaredpalmer/kev-4b",
+        "base": "Qwen/Qwen3.5-4B-Base",
+        "lora": 16,
+        "device": "cuda",
+        "backend": "torch",
+        "dtype": "bfloat16",
+        "temperature": 2.406050072164233,
+        "max_state_tokens": 65536,
+        "truncate_states": false,
+        "cuda_graphs": {
+          "captured": 6,
+          "kept": 6,
+          "pending": 0,
+          "failed": 0
+        },
+        "prefix_cache": {
+          "size": 4,
+          "min_state_tokens": 0,
+          "max_tokens": 65536,
+          "hits": 0,
+          "misses": 3,
+          "cached_states": 3,
+          "oom_retries": 0
+        },
+        "batches": {
+          "count": 3,
+          "requests": 3,
+          "queued": 0
+        }
+      }
+    ]
+  },
+  "image_versions_source": "Cached base image layers, whose package versions are recorded in their earlier build log; no causal-conv1d installation in the final deployment image."
+}