Skip to content

AGENTS.md

Latest commit

 

History

History
439 lines (314 loc) · 37.6 KB

AGENTS.md

File metadata and controls

439 lines (314 loc) · 37.6 KB
Raw

AGENTS.md

Purpose

Deqio is a small multi-engine runtime for typed AI decisions.

It exposes one stable HTTP API and a lightweight browser UI over independent decision engines while keeping installation reproducible with uv.

Keep the project small, deterministic, observable, fast, and easy to reinstall.

Core principles

  1. Preserve the public HTTP API unless a task explicitly permits a breaking change.
  2. Keep model/engine-specific behavior behind runtime adapters.
  3. Do not vendor upstream engine source code or model weights.
  4. Do not commit .venv, .model-runtimes, model weights, local Deqio state, or request logs.
  5. Use uv exclusively for Python and dependency management.
  6. Keep one selected decision model resident per server process.
  7. Do not convert classification into autoregressive answer generation.
  8. Do not claim backend support that the upstream runtime does not actually provide.
  9. Keep the browser UI dependency-free: plain HTML, CSS, and JavaScript only.
  10. Treat probability calibration as engine-specific.

Naming

The project, Python package, CLI, logs, and local state use the Deqio name:

project:      deqio
package:      deqio
CLI:          deqio
console:      [deqio]
local state:  .deqio/

SemIf remains the name of one supported upstream engine. Do not rename upstream engine names, model IDs, package names, or protocol concepts to Deqio.

Runtime architecture

The main environment contains the Deqio core only. Every decision engine, including SemIf, runs in an isolated environment under:

.model-runtimes/<runtime-key>/

This isolation is intentional. Independent engines can require conflicting PyTorch, Transformers, MLX, or accelerator versions.

Keep runtime environments separate from local model checkpoints. Downloaded or prepared weights that Deqio manages explicitly live under:

models/

Prepared local checkpoints such as Nimble use this directory. Do not move managed weights into .model-runtimes/: runtime environments should be rebuildable without forcing large model downloads or checkpoint merges again. SemIf MLX now consumes the canonical Hugging Face checkpoint directly and applies the upstream 8-bit MLX quantization path at load time.

The model catalog is models.json.

The active selection is stored in config.json as:

engine
model_id
backend
model

Supported wrapper backend labels are:

mlx
gguf
cuda
remote

The PyTorch MPS backend was removed in 0.6.4: no catalog profile, launcher or probe uses it, config.json naming it fails to load, and a registry row left by an older Deqio (model_id::mps) is listed by deqio models installed as stale and removed by deqio models delete MODEL --backend mps (the row and the PyTorch runtime directory its CUDA sibling declares). Apple Silicon runs MLX or GGUF; Decider 0.8B and Von are CUDA-only since 0.6.4.

A catalog profile must exist for every allowed (model_id, backend) pair. remote is the backend of the catalogued remote decision APIs (openai-decisions, typesafe-jev): no runtime directory, no weights, served by deqio.remote_runtime over HTTPS; see "Remote decision profiles".

Package management

Do not use:

pip install
python -m venv
requirements.txt
poetry
pipenv

Use:

uv sync
uv add
uv remove
uv lock
uv run
uv venv
uv pip install --python ...

The installed public CLI is:

deqio serve
deqio decide
deqio openapi
deqio models list
deqio models installed
deqio models setup
deqio models use
deqio models delete
deqio models status
deqio status
deqio --version
deqio version

deqio decide ROUTE (0.6.2) asks a running server for one decision through the SDK deqio-client and prints the answer; it never starts a server, loads a model or retries. deqio openapi prints the OpenAPI 3.1 document.

From a source checkout, prefix these commands with uv run. User-facing runtime errors and help text for the installed tool should use deqio ..., not uv run deqio .... Missing model/runtime guidance should point users to deqio models setup.

Every interactive terminal selection must offer q as a non-error exit path. quit and exit may be accepted as aliases.

Configuration

Default configuration lives in config.json.

Environment overrides use the DEQIO_ prefix, including:

DEQIO_CONFIG
DEQIO_ENGINE
DEQIO_MODEL_ID
DEQIO_BACKEND
DEQIO_MODEL
DEQIO_MODEL_REVISION
DEQIO_MAX_TOKENS
DEQIO_MLX_CACHE_MIB
DEQIO_LOG
DEQIO_TORCH_DTYPE
DEQIO_RUNTIME_DIR
DEQIO_MODEL_CATALOG
DEQIO_SIDECAR_STARTUP_SECONDS
DEQIO_SIDECAR_PROCESS_READY_SECONDS
DEQIO_HF_OFFLINE_RUNTIME
DEQIO_WATCH_TOKEN

Remote decision APIs are selected through the catalog (backend: remote), never through environment variables: there is no provider-selection variable family.

Keep project configuration under the DEQIO_ namespace. SemIf remains the name of one upstream engine.

Public HTTP API

GET /openapi.json (deqio openapi, committed copy docs/openapi.json, kept equal by a test) is the OpenAPI 3.1 document of the API. deqio.openapi adds the response contracts of the decision routes — the portable result with provenance (provider, remote_request_id, cost_estimate), the shared batch, the native System One reply with its deqio block, the error shapes — as documentation and validation models only: never attach them as FastAPI response_models, the handlers keep returning the dicts they always did. A new response field is added to the document in the same change (the models reject undocumented top-level fields; engine-specific nested blocks stay open). The Python SDK deqio-client (clients/python, package deqio_client, no dependencies, never imports deqio) and deqio decide send exactly the request shapes the server models accept; both are tested against the live server.

Keep these endpoints backward compatible:

GET  /
GET  /health
GET  /ui
GET  /v1/models
GET  /v1/models/installed
GET  /v1/stats
GET  /v1/recent
GET  /v1/watch
GET  /v1/watch/settings
GET  /v1/watch/{event_id}

POST /v1/watch/settings
POST /v1/watch/clear
POST /v1/noul
POST /v1/choice
POST /v1/decision
POST /v1/shared
POST /v1/score
POST /v1/multi
POST /v1/act
POST /v1/soam
POST /v1/systemone
POST /v1/cache/clear
POST /v1/models/activate

/v1/decision is a compatibility alias for /v1/choice.

Decision handlers keep blocking work off the event loop with exactly one worker thread per request: the async handler only negotiates the input contract, then one asyncio.to_thread call does validation, inference, the request log and Watch. Every rejection that reaches a decision handler (validation, input contract, runtime/protocol error) goes through server._reject, which adds exactly one to stats.errors, writes one console line and one Watch row. Portable inputs are validated before inference: no blank questions or explicitly supplied IDs, unique IDs, at most MAX_OPTIONS options per decision and MAX_DECISIONS decisions per Shared request. Native SystemOne requests (/v1/systemone, /v1/soam) accept at most MAX_QUESTIONS questions and are otherwise rejected with 422 through server._reject.

External engines should be normalized to the same response shape. If an engine does not expose raw logits, return an empty option_logits object. Never fabricate logits from probabilities and label them as raw logits.

Decision responses expose Decision Provenance / Attestation v1 under provenance. The model/runtime identity must be captured under the same inference lock as the score so a live model switch cannot pair a result with the wrong model identity. Provenance must include the loaded runtime instance ID, engine, model ID, backend, model source, recorded artifact revisions, and structured score provenance.

Score provenance must distinguish engine-reported probabilities from Deqio transforms or synthetic fallbacks. In particular, one-hot probabilities synthesized from a choice-only engine response must be marked synthetic and must never be presented as native model confidence. Renormalization/clamping must be explicit transforms. Calibration remains engine-specific and must not be inferred by Deqio.

Attestation v1 is a local unsigned response binding. Its digest binds the decision result, prompt hash, runtime identity, and score provenance, but it is not a cryptographic signature or remote trust proof. complete may only be true when runtime identity, resolved model artifact revisions, and score provenance are all available. /v1/shared must preserve per-result provenance and expose a batch-level binding to the same runtime instance.

An optional versioned input-completeness contract is negotiated with Deqio-Contract: input-completeness-v1. Clients without the header retain the existing schema. Unknown versions must fail before inference; never silently downgrade. For negotiated requests, hash the exact HTTP body bytes before FastAPI parsing, reject duplicate JSON keys, and require explicit stable decision/option IDs. input_policy.require_complete=true is fail-closed: if the active runtime cannot prove exact model-boundary completeness, reject before inference with input_completeness_unavailable. Never infer completeness from truncation=false, configured max_tokens, token counts alone, or absence of an error. Missing usage is null/unknown, never a fabricated zero.

A negotiated response may use attestation schema v2 to bind its input_receipt to the same request/result/runtime/score identity while preserving attestation v1 semantics for non-negotiated clients. status=unknown must remain distinct from complete. Shared receipts require unique decision IDs and must describe each decision rather than copying one aggregate token count as if it were per-decision measurement. Do not claim model-boundary completeness until the engine adapter actually instruments rendered input, logical token IDs, masks/cache behavior, and the effective scoring limit.

Model catalog rules

models.json is the source of truth for selectable models.

Every entry needs:

  • stable local id
  • engine
  • human-readable label and description
  • upstream source URL
  • explicit compatible backends
  • model identifier/path for each backend
  • isolated runtime install packages when applicable

When a runnable profile uses quantized weights, record the quantization contract in the profile (quantization metadata and any engine-specific loader option) so provenance and tests can distinguish it from full precision. Prefer verified 8-bit MLX variants, then verified 6-bit variants, but never replace an engine-specific adapter/head with a backbone-only quant. GGUF availability alone does not justify a Deqio profile. A GGUF backend may be catalogued only when the exact current model identity has an official/authoritative Q8_0 (8-bit or better) artifact and a native decision/SystemOne readout. Community conversions and similarly named but different model versions must not be substituted silently.

Do not silently fall back from an unsupported backend to CPU or another backend.

Every runtime package is pinned: catalog packages, every *_package field and every package an installer adds itself is an exact PyPI version (name==version) or a GitHub commit (@<40-hex> or /archive/<40-hex>.tar.gz), never a branch, tag, range or bare name. FastAPI/Uvicorn inside engine runtimes only carry Deqio's sidecars and use the core uv.lock versions. Source checkouts (llama.cpp, Nimble) are fetched at the catalog commit, never pulled. Moving a pin to a newer upstream release is its own change with a per-profile benchmark. Pins for Apple Silicon runtimes are resolved for macOS 14 or newer (MACOSX_DEPLOYMENT_TARGET=14.0 with uv pip compile --python-platform aarch64-apple-darwin); the macOS 13 default of a Linux resolver silently falls back to old mlx releases.

Every profile with a runtime declares exclude_newer, an RFC 3339 UTC instant shared by all profiles of its runtime (remote profiles install nothing and declare none). Installers pass it as uv pip install --exclude-newer to every resolution from PyPI, so transitive dependencies resolve the same way on every install, and refuse a profile without it. A pinned version must be older than its profile's instant. Only a command that installs one exact pin from an index without upload times (the PyTorch CUDA wheels) runs without it. Moving the instant is a benchmarked change like moving a pin. python -m deqio.pin_audit compares an installed workspace with these pins and the Hub revisions (read-only).

Hugging Face artifacts carry a 40-hex commit revision. The remaining exceptions are listed in tests/test_catalog_pinning.py::HF_REVISION_PENDING, which may only shrink. Pin a revision only where the launcher makes the engine load it (an argument, LAYA_REVISION, a snapshot path); otherwise the offline cache, the loaded weights and provenance disagree. An engine whose CLI takes a repo id but no revision (Decider, JevK5) is given the path of the pinned commit's cached snapshot and refuses to start when the cache lacks it. Where the catalog itself names the commit of a pin-on-install profile, an installation verified at another commit is refused with the repair command.

Every profile declares source: official (runtime and weights from the model's upstream, or its designated GGUF org) or source: community (a third-party port, which must also name source_url).

Installed-profile state is local machine state under .deqio/ and must not be committed. The registry key model_id::backend is authoritative: shared runtime directories or Hugging Face cache entries must never make a different backend appear installed.

Successful install/update should also record model artifact provenance in the local registry. For Hugging Face artifacts, store the requested revision (if any) and best-effort resolved immutable commit SHA. Existing pre-provenance registry entries remain valid, but their response attestation is incomplete until an install/update refreshes the artifact metadata. Updating a shared runtime (runtime_key) is a registry state transition: every registered profile of that runtime loses verified_at (kept in an update_pending block) until the update passes the readiness probe, so a failed update never leaves profiles claiming a verification the rebuilt runtime did not pass. deqio models delete unregisters a profile only after its artifacts are removed.

Each installed profile may also record max_input_tokens. models setup must present a bounded interactive token-budget list after model selection. Prefer the standard presets 4096, 8192, 12288, 16384, and 32768, while honoring runtime-specific hard limits and conservative host-memory guardrails; values that fail those guards should be shown as unavailable rather than selectable. Direct install/update may accept --max-input-tokens, and selecting/activating an installed profile must restore its recorded token budget. The memory calculation is only a provisioning guardrail and may not be presented as proof of a backend's effective context capacity or completeness.

models setup / models install are the network-enabled provisioning phase. They must install the runtime, resolve/download all weights (including transitive base-model dependencies), start the engine, and pass a real typed-decision model-readiness probe before marking a profile verified. Normal serving/benchmarking should run Hugging Face in offline mode by default and fail clearly if the prepared cache is incomplete. Installed-state detection for Hub-backed artifacts must validate the exact requested revision/tag/commit when the catalog pins one; repository presence alone is insufficient because offline snapshot_download(..., revision=...) may still fail.

Model installation must run the host compatibility preflight. Catalogued memory requirements are conservative guardrails; direct install may expose an explicit force override, but interactive setup should not offer profiles below their minimum requirement.

Installation and activation are separate concerns: the live activation endpoint must never install packages or silently download a new runtime. It may only switch to a profile already detected as installed.

Sidecar process readiness and model readiness are distinct states. Opening the localhost port only proves the process is listening; a runtime is ready only after a typed-decision probe succeeds. Use the short process-ready timeout for the port and the longer configurable startup timeout for first model initialization.

Live model switching must keep one resident model at a time. Unload the old runtime before loading the new one, block inference during the transition, persist config.json only after the new runtime passes warmup, and attempt to restore the previous runtime on failure.

MLX is the only Apple Silicon accelerator stack (MPS was removed in 0.6.4); never alias a removed backend to an existing one.

The catalog profile is the only source of the model an engine loads: launchers take model, revision and checkpoint paths from the profile, and config.json / DEQIO_MODEL / DEQIO_MODEL_REVISION are a mirror written by deqio models use/select. A mirror value that differs from the catalog profile must fail loading with an explicit error before any sidecar is spawned (pin-on-install profiles may run the immutable revision recorded by the installer). Never override the catalog silently.

Basal 2.0 fully replaces the removed Basal 1.5 catalog profiles (0.6.1). The only Basal family is basal-2.0-mini, basal-2.0-main (upstream Remek/basal-2.0-4.5B), and basal-2.0-max. Use the official rkinas/basal v2.0.0 basal-serve runtime pinned by commit and the author's official -MLX-8bit and -GGUF (Q8_0) artifacts; the accepted backends are MLX 8-bit and GGUF Q8_0 (MPS and CUDA profiles are not catalogued for 2.0). Advanced semantics remain native SystemOne semantics internally: Choice, Noul, Score, Multi, Act, facts: "auto", SOAM and option keys; Basal 2.0 ships no evidence head, so evidence: true is refused explicitly on every 2.0 profile. The thin convenience HTTP wrappers for native score, multi, act, and soam must inject/forward to the same SystemOneRuntime.system_one path and must not implement alternative scoring. Multi emulation (0.6.3, deqio.multi_emulation) lives inside that path, in both runtimes: a profile with systemone, noul and multi_question but no native multi reports multi_emulated: true (its multi stays false; never claim the native head), each multi question becomes one noul branch per label appended to the same engine request (one HTTP call per System One request, the state sent once), the branch P(yes) are the label probabilities unchanged, the set is chosen by Basal's rules (threshold/min/max, objectives with exact expectations, exclusive, fallback), the answer carries emulation and the reply deqio.multi_emulation, and any field the emulation cannot honour is refused — never ignored. Basal's native multi is not routed through the emulation. Do not create standalone Facts/Evidence/OptionKey endpoints: those remain request/question modifiers. Do not softmax Multi probabilities, do not re-decide Act results, and never silently ignore unsupported Evidence. The 2.0 tool question type is not exposed yet. A Basal profile's max_input_tokens is the checkpoint's max_position_embeddings (mini 8192, main/max 32768), and every Basal 2.0 profile declares basal_context_overflow: refuse so basal-serve rejects a prompt longer than the model context (HTTP 422) instead of truncating the state's middle (its default, marked only in usage.truncated).

Decision 3.0 (decision3, 0.6.1) replaces the removed Decision 2.0 engine. The official vllm-sr/d3* packages ship their own System One server (d3_server.py, stdlib HTTP, POST /v1/systemone); the CUDA launcher runs it from the pinned snapshot path (--model <snapshot> --verify fast), so the pinned commit is the code that runs, and _decision3_runtime_metadata refuses a snapshot whose MODEL_MANIFEST.json identity differs from the catalog's d3_model_sha256. GGUF profiles use the ggml-org/vllm-<model>-GGUF Q8_0 files through the shared llama.cpp runtime; Decision 3.0 runs non-causal attention there, so llama-server reads a whole prompt in one batch and the launcher passes -b and -ub equal to the token budget (-c) for every engine llama.cpp evaluates in one batch (systemone_runtime.LLAMA_CPP_ONE_BATCH_ENGINES: decision3, clef, laya; 0.6.4 fix 1 — with the defaults -b 2048 -ub 512 the server batched the prompts of its four slots together and llama_decode aborted on n_ubatch >= n_tokens; Kev's prompts split and keep the defaults). The compute buffers scale with -ub, so the token budget chosen at setup is also a memory choice for these profiles. Decision 3.0 has no MLX profile: the community BF16 conversions (ahmedandaloes) and Deqio's own MLX forward for them (0.6.1 fix 2, 0.6.2 fix 3) were removed in 0.6.2 after the owner's benchmark showed the MLX path at 41.9 % / 56.8 % decision accuracy against 90.3 % / 90.8 % for the same model on GGUF; the launcher refuses decision3 on mlx by name (no fallback), deqio benchmark tables excludes such rows by name, and a Decision 3.0 MLX profile may return only with a conversion that reproduces the official readout on the decision set (its own patch, benchmarked before the tag). Decision 3.0 probabilities are native and preserved unchanged (no Deqio renormalization), as Decision 2.0's were.

Clef Flash and Clef use pinned mlx-community 8-bit snapshots on MLX, retaining the BF16 joint schema head and the snapshot-bundled clef_mlx.py System One server with truncation disabled. They also expose audited official ggml-org Q8_0 GGUF profiles through llama.cpp's native SystemOne endpoint. Do not replace the decision head with a normal text-generation path; MPS was removed in 0.6.4 and is never aliased to MLX.

Profiles with a native SystemOne API must declare capabilities.systemone plus their actual per-feature support. /ui, POST /v1/systemone, POST /v1/soam, and the native typed convenience wrappers use the same capability metadata; do not expose a feature in the GUI that the active runtime cannot execute, and do not silently hide a supported native feature from the GUI. Preserve native probabilities and response fields rather than synthesizing replacements.

Change workflow

Before work:

git status --short

After changes:

uv run ruff check src tests clients
uv run python -m compileall -q src tests clients
uv run pytest
git diff --check
git diff

Record user-visible changes in CHANGELOG.md (Keep a Changelog).

Documentation has three audiences (0.6.4 release): README.md is for users — what Deqio is, install with uv, the models table, the benchmark results (the current snapshot only), install from source, the API in short, the Python client, a summary; it stays under 600 lines (tests/test_server.py guards the sections and the limit) and carries no reference material. The reference is docs/API.md (every route with an example, modifiers, capabilities, errors and provenance, the SDK and deqio decide, OpenAPI, the browser UI and Watch), docs/MODELS.md (backends, engine families, remote APIs, token budget, deqio models commands, design principles) and docs/BENCHMARKS.md (running, suites, scoring, aborted profiles, tables, comparison); benchmarks/summary_*.md keep the full benchmark history. This file is for engineers and agents. A behaviour change updates the page that documents it, and a test that pins documentation wording names the page it reads. .github/workflows/ci.yml runs the same lint, byte-compile and test steps on every push and pull request with Python 3.10 and 3.12; keep tests runnable on 3.10 (no stdlib-only-in-3.11 imports without an explicit version check). A release follows docs/RELEASE_CHECKLIST.md.

Produce a patch with:

git diff --binary > change.patch

Before applying a supplied patch:

git apply --check change.patch
git apply change.patch

Do not use git apply --reject unless explicitly requested.

Do not use destructive Git commands unless explicitly requested.

Validation

Minimum validation after Python changes:

uv run ruff check src tests clients
uv run python -m compileall -q src tests clients
uv run pytest

Changes to model management must also verify:

uv run deqio models list
uv run deqio models installed
uv run deqio status
uv run deqio --version

Changes to serving must verify:

/health
/v1/noul
/v1/choice
/v1/shared
/v1/score
/v1/multi
/v1/act
/v1/soam
/v1/systemone

Changes to UI must verify:

  1. /ui loads.
  2. Noul requests can be submitted.
  3. Choice options can be added and removed.
  4. Shared decisions can be submitted.
  5. Native Score/Multi/Act/SOAM examples follow the active profile capabilities and reload when the active profile changes.
  6. generated JSON matches the API request.
  7. responses render without page reload.
  8. active engine/model/backend are visible.
  9. installed model profiles are listed.
  10. switching an installed profile keeps the same public API URL.
  11. /ui/watch loads and links back to /ui.
  12. watch rows refresh for API decision requests and direct benchmark cases, and row details expose the full temporary request/response payload.
  13. model switches keep history in the same server session while every row retains its own engine/model/backend/runtime identity.
  14. Watch auto-clear can be configured from both /ui and /ui/watch.
  15. rotated Watch files never exceed 10,000 JSONL event lines.

Watch UI and session inspection

/ui/watch is a separate operational view for decision traffic. Track parsed request/response pairs for /v1/noul, /v1/choice, /v1/decision, /v1/shared, /v1/score, /v1/multi, /v1/act, /v1/soam, and /v1/systemone, including failed calls that reach those endpoint handlers. Direct deqio benchmark case execution must write equivalent events to the same Watch store for the same workspace. Do not record watch/stats/health polling endpoints or internal readiness/warmup probes as user decision traffic.

Full Watch payloads are temporary disk-backed session data under .deqio/watch/, not unbounded RAM state and not part of the normal persistent JSONL request log. Store one JSON event per line and rotate to a new events-*.jsonl file after 10,000 event lines. Listing must be paginated/lightweight and must not read an entire potentially large event file into memory just to render one page; detailed request/response bodies should be fetched by event ID.

Delete Watch event files when the server starts, on explicit Watch clear, and when the configured automatic cleanup interval expires. Retention preferences may survive those resets. Model switches do not clear a running server's Watch history: each event must carry source, engine, model ID, backend, and runtime instance identity so UI filtering can separate models safely. If a reset/auto-clear replaces the session while a request is still in flight, reject that stale append instead of inserting it into the new session. When deqio serve binds a non-loopback address, Watch (/v1/watch*, /ui/watch) requires the per-server Watch token (DEQIO_WATCH_TOKEN or generated at startup; a configured token is never echoed). Loopback binds keep Watch open, and the token never gates decision routes.

The server and benchmark CLI can be separate processes writing the same workspace store. Keep appends/rotation/session metadata cross-process safe and keep file permissions private where the host supports it.

The main /ui must link to /ui/watch, and the watch page must link back to /ui. Keep both pages dependency-free. Runtime-instance IDs in Watch details must not expand the grid into neighboring latency fields: show a compact/ellipsized value, expose the full value on hover, and keep a direct copy action. The automatic refresh must keep every page loaded with "Load older" and must not pile up requests behind a slow one; every Watch handler reports failures in the health badge instead of leaving an unhandled rejection. The Watch script is executed under Node in tests/test_watch_ui_behavior.py; keep it free of browser-only globals beyond document, fetch, setInterval, alert, confirm, prompt and navigator.clipboard.

Benchmark suites

Benchmark suites are editable schema-version-1 JSON files under the workspace benchmarks/ directory. deqio benchmark without --suite must discover all valid *.json files there and present a suite selector before model selection. Packaged defaults currently include basic.json (ENG Bench, deqio-basic-220: 55 cases of each type) and pl.json (PL Bench, deqio-pl-120: 30 of each), but the runtime must not hard-code those two filenames for discovery. --suite PATH remains the explicit/non-interactive override. Invalid JSON/suites should be skipped with a clear warning rather than making every other valid suite unusable.

Case types are noul, choice, shared and, since 0.6.4, multi; the run loop, the summary and the UI follow the types a suite holds (benchmark.suite_kinds), never a hard-coded three. A multi case holds one native multi question (instructions, criteria, Basal's optional set fields) and expected, a list of label keys; load_suite validates them. It runs through the runtime's system_one (native on Basal, the noul-branch emulation elsewhere) and is scored by set equality (the case) and per label (each label is one decision, right when its presence matches — label accuracy, printed as "Multi label accuracy" in the tables). A suite-level multi_branch is sent as the question's branch only to profiles that report multi_emulated. A case where no label applies expects the empty set, never a self-referential "none" label with a fallback (0.6.4 fix 1: Basal selects a fallback only when its own probability passes the threshold, so such a label measured the wording, not the model). The v2 cases carry a domain tag (agent, security, retail, politics, world, office, business); keep the packaged copies under src/deqio/data/benchmarks/ identical to the workspace copies.

A profile whose sidecar process exits mid-run is stopped at that case (0.6.4 fix 1): SystemOneRuntime._request raises SystemOneSidecarExitedError (a SystemOneUnavailableError carrying the supervisor's exit status) when the connection fails and the process is gone, the run loop prints one sidecar exited (status N) during CASE; M of T cases not run line, writes no rows for the cases that did not run, and records aborted (case_id, reason, cases_run, cases_not_run) on the profile's summary entry; the CLI summary, /ui, deqio benchmark tables ("did not finish", no row) and deqio benchmark compare (unfinished_left / unfinished_right, never common) report it like a load failure, never as a scored profile. The sidecar is not restarted and the cases are not retried.

Benchmark runtime ownership

A benchmark must have exclusive inference ownership for its workspace without killing the Deqio HTTP server. If deqio serve is running, the benchmark asks the server through the workspace-local authenticated control channel to close its active runtime; /ui, /ui/watch, health, history and administrative APIs remain alive. Inference requests must return a clear temporary 503 and model activation must return 409 while the benchmark owns inference. Load one benchmark profile at a time and always close it before loading the next. At benchmark completion restore the server's previous profile. If the benchmark owner process disappears, the server watchdog should restore automatically. Never implement this by broad pkill/process-name matching. A workspace benchmark lock prevents two benchmark CLIs from running model workloads concurrently. Workspace ownership is decided by kernel locks, never by probing the HTTP port (uvicorn binds it only after the model has loaded): a registered server holds .deqio/.server-owner.lock for its lifetime and registers only while holding model-management.lock, so deqio serve, deqio models mutations and benchmarks exclude each other atomically.

Benchmark comparison

deqio benchmark compare compares concrete completed run directories, not suite names. New benchmark summaries must persist canonical profile identity including at least model ID, backend, precision/quantization where relevant, significant runtime identity/version, and pinned artifact revisions. Only exact canonical-profile intersections are common. Never merge the same model across different backends or quantizations. Show only-A/only-B explicitly. Comparison output must include accuracy, decision accuracy, median/P95 latency, throughput, rank/rank shift, and per-decision-type breakdown only where both runs have the metric. Different suites may be compared factually but must not be interpreted automatically as a pure language/quality delta. JSON output is part of the agent/automation contract. Legacy summaries without canonical identity must not be silently treated as common profiles. A profile whose runtime failed to load (load_error in the summary) is reported as such — CLI table, UI and load_failed_left/load_failed_right in the comparison — never as 0 cases or as a legacy profile.

deqio benchmark tables (0.6.2, deqio.benchmark_tables) renders the README / benchmarks/summary_*.md tables of a completed run from its summary.json (schema 2) and results.jsonl: every number is the run's own, rows are sorted by decision accuracy, remote profiles carry the estimated cost per 1 000 scored decisions computed from the per-row cost_estimate amounts (n/a when any decision lacks one), profiles with load_error are listed under "did not load", and a row of a profile removed from the catalog (benchmark_tables.REMOVED_PROFILES: the Decision 3.0 MLX conversions) is excluded by name — its numbers are not comparable with the supported runtimes. README benchmark tables are pasted from this command, never typed by hand; --json is an additive automation contract. The README tables are rendered from the committed runs/<group>-<lang>/summary.json (the owner's 0.6.4 runs on the v2 suites; the suite value is reduced to the suite's file name before committing, results.jsonl stays git-ignored) and tests/test_benchmark_snapshot.py fails when README or benchmarks/summary_*.md differ from the generator's output over those files — a new run replaces its group's summary and its tables in the same change; the previous snapshot's tables move to the historical sections of benchmarks/summary_*.md (never to README).

Logging

Console logging is part of the developer-facing interface.

Rules:

  • identify the public server as [deqio];
  • prefix external runtime output as [sidecar:<engine>];
  • suppress internal sidecar HTTP access-log noise;
  • keep the main Uvicorn access log disabled;
  • log wrapper-owned /v1/noul, /v1/choice, /v1/decision, /v1/shared, /v1/score, /v1/multi, /v1/act, /v1/soam, and /v1/systemone requests clearly;
  • print the UI, API docs, and health URLs at startup;
  • describe random sidecar ports as internal inference-only;
  • keep useful model download, loading, accelerator, and runtime diagnostics visible.

Do not persist the full request state by default.

Performance

Avoid changes that:

  • reload model weights per request;
  • create duplicate resident models;
  • add text generation to typed decisions;
  • disable native batching/shared-state behavior without reason;
  • merge incompatible engine dependencies into the main environment;
  • put UI rendering or model-selection work in the normal decision request path.

External engines should remain resident in a localhost-only child process for the lifetime of the main server.

Remote decision profiles (D11, backend form since 0.6.1)

A remote decision API is a catalog profile on the remote backend (openai-decisions, typesafe-jev) and behaves 1:1 like a local profile: deqio models setup installs it (model identifier typed by the user, API key, the same input-token budget list as a local model, then one typed-decision probe against the real endpoint — nothing is registered before the probe passes), deqio models use / POST /v1/models/activate select it, deqio benchmark runs it as an installed profile, deqio models delete removes it. There is no separate provider configuration, flag or environment selection, and a remote profile is never a fallback for a sidecar or for another provider. The API key lives only in the operating-system keychain (deqio.credentials, keyring, service deqio, account = provider); it is read when the profile loads and never written to config.json, the registry, provenance, logs or recordings; a host without a keychain backend is an explicit error, not a file or environment fallback. The registry record holds remote_model, endpoint and api_key_store; config.json model mirrors remote_model (a mismatch refuses to load, as for local profiles). Every answer names its provider in provenance (provider, remote_request_id, cost_estimate); cost is an estimate from the API's reported input tokens and the published input price, never a bill, and amount stays null when usage is not reported. The wire mapping of each API (OpenAI Decisions: input + questions array, predicate for noul, answers array, refusal is an error; TypeSafe: the native System One wire at /v1/systemone) lives in deqio.providers only, is corrected from recorded cassettes, never guessed twice, and the contract tests replay the cassettes under tests/fixtures/providers/ — real exchanges recorded by the owner with python -m deqio.provider_record (0.6.2; both confirmed the documented wires), whose expectations the tests read from the recording itself, never from numbers typed into the test. Unsupported request options and question types are refused with an explicit error.