[devin-ai-integration]

Summary

Modal's ordinary web endpoint routes each request through the Function input plane, adding substantial queueing time to Kev's GPU execution. The existing opt-in Flash path bypasses that plane, but uses a deprecated API and targets only 32 concurrent requests per container.

Register KEV_FLASH=1 through the supported app.server() API instead, with target_concurrency=64. The server routes HTTP directly to the existing Uvicorn/FastAPI app; Kev still owns bearer authentication, batching, CUDA graphs and response construction. Model code, the checkpoint pin and the deployment image are unchanged. KEV_REGION selects the routing/compute regions, and direct mode retains at least one warm container so it does not return 503 while waking from zero.

Measured benefit and rejected alternatives

The paired report uses jaredpalmer/kev-4b, one H100 per variant, four CPUs, us-east, an in-region kept-alive client, bearer auth and six-question requests. Each level measures max(128, clients × 8) requests after warm-up.

Clients	Web p50	Direct p50	Web requests/s	Direct requests/s
1	81.44 ms	34.06 ms	12.15	28.71
64	1,236.21 ms	652.04 ms	43.82	65.54

Warm single-request median falls 58%; direct's scaling target doubles from 32 to 64. This is not a claim of 2× throughput or halved tail latency: p99 at 64 clients was 4.50 s versus 3.98 s, and 128 clients still queue. CUDA graph capture can occur during load, and repeated pilots vary.

Raising the ordinary web limit to 128 regressed throughput at 64 clients: 43.82 → 37.07 requests/s with target 32; the target-64 pilot regressed 56.07 → 35.97. Keep the regular endpoint at 64/32 and preserve scale-to-zero rather than shipping either regression.

Direct mode stays opt-in (KEV_FLASH=1 KEV_REGION=us-east) because it keeps a paid GPU warm. No production deployment, model-quality change or inference refactor is included. Modal currently reports $5.89 for the temporary pilots; all temporary containers are stopped.

Test plan

• Full fast-suite command from kev-verify: 482 passed, 26 skipped, one existing Starlette deprecation warning.
• uv run --extra serve python -m pytest tests/test_skill_scripts.py tests/test_conventions.py -q: 44 passed, including scale-to-zero, direct registration, region requirements, bearer-secret preservation and minimum-container behavior.
• Paired H100 HTTP pilots: zero recorded load errors, identical six-question answer payloads across variants; latest run also verifies 401, 422, /v1/models aliases and TypeSafe SDK behavior.
• Actual standalone deployment-script smoke: independent pinned-Git image, valid/invalid bearer requests, validation, SDK, identical six-question answers against the paired report, 20 warm requests with 18.4 ms model p50. Its client is not in-region, so its RTT is not a paired comparison.
• git diff --check, Python compilation and thermonuclear-code-review self-review: no structural blocker; runtime files remain below 1,000 lines.

Reproduce the paired run with uv run --extra serve modal run scripts/modal_http_bench.py --out runs/modal-http-4b-h100 --levels 1,32,64,128; set KEV_HTTP_TARGET=64 to retry the rejected target-64 variant.

Link to Devin session: https://app.devin.ai/sessions/d60ffea0b58946a9a30f8c7a5ebe1291
Open in Devin Desktop: https://app.devin.ai/desktop/session/d60ffea0b58946a9a30f8c7a5ebe1291?variant=devin
Requested by: @jaredpalmer

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring