Skip to content

harnesses.md

Latest commit

 

History

History
395 lines (331 loc) · 27.6 KB

harnesses.md

File metadata and controls

395 lines (331 loc) · 27.6 KB
Raw
title Harnesses — advanced behavior
description Deep reference for Aeon's harness axis (the nine agent CLIs behind one run-harness contract) - token accounting, capability-mode mapping, MCP, and per-surface harness selection.

Harnesses — advanced behavior

The harness is the coding-agent CLI that runs your skills. The README's harnesses section covers the basics - nine agent CLIs behind one run-harness contract, with claude as the default. Every harness, claude and grok included, runs through the same adapter; glm is not a harness but a gateway provider (see LLM Gateways). This page collects the deeper behavior for anyone running a non-default harness (especially grok) in anger.

Additional harnesses via run-harness (codex, pi, vibe, kimi, fx, cursor, hermes)

Seven more harnesses are selectable in the dashboard's harness dropdown and the harness: config: codex (OpenAI Codex CLI), pi (Pi Coding Agent), vibe (Mistral Vibe), kimi (Moonshot Kimi), fx (Vercel's fx - fx.sh, a minimal native Zig coding agent), cursor (Cursor CLI), and hermes (Nous Research Hermes Agent). Like claude and grok, they run through harness-adapter's run-harness, which wraps each CLI in the same Claude-Code-shaped {result, usage, session_id} contract, so everything downstream (scoring, token accounting, memory, notifications) is unchanged.

fx differs on auth — see the table below: it has no OpenRouter fallback, so it runs on a Vercel AI Gateway key (or VERCEL_OIDC_TOKEN inside Vercel's own CI) rather than the shared OPENROUTER_API_KEY.

Each one runs on its own provider login (see Native auth below); a single shared OPENROUTER_API_KEY is the zero-setup alternative for codex, pi, vibe, kimi and hermes at once (and, as a gateway, for claude). Their model picker offers OpenRouter ids rather than the claude-*/grok-* ids, and the model you pick is what actually runs. Each of these harnesses carries its own curated list (CODEX_MODELS / VIBE_MODELS / PI_MODELS / KIMI_MODELS): codex defaults to openai/gpt-6-luna (newer and cheaper than the prior default) and also offers gpt-6.1-sol for quality (older pins like gpt-6-sol, gpt-5.1-codex-mini, gpt-5-mini, gpt-5.3-codex, and the gpt-5.6 family still dispatch); it fails on gpt-5-nano; vibe's generic ProviderConfig drives any OpenRouter model, so it defaults to mistralai/mistral-medium-3-5 and offers deepseek/deepseek-v4.1-flash; pi (litellm openrouter/<slug> routing) runs the two models DeepSeek's own API serves: deepseek-v4.1-flash (default) and deepseek-v4-pro (older deepseek-v4-flash pins still dispatch); hermes runs its configured default, or anthropic/claude-sonnet-5.5 / openai/gpt-6.1-sol as overrides; kimi is Moonshot, so it runs Moonshot's own Kimi family through OpenRouter: moonshotai/kimi-k2.7-code (default, code-tuned) and kimi-k3 (strongest, ~2× slower; older kimi-k2.6 and kimi-k2.5 pins still dispatch). The scorer routes through the same harness the skill ran on, so a repo with no Claude credentials still gets every run scored.

OpenRouter app attribution

Every OpenRouter request aeon makes carries app attribution headers, so usage is credited to Aeon's OpenRouter app: HTTP-Referer: https://www.aeon.fun, X-OpenRouter-Title: Aeon (plus the legacy X-Title) and X-OpenRouter-Categories: personal-agent,cloud-agent. One helper (harness-adapter/lib/openrouter-attribution.sh) sets them for claude (via gateway: openrouter), codex, kimi, vibe, pi and hermes; cursor, fx and grok have no OpenRouter path. A fork can set its own with OPENROUTER_SITE_URL, OPENROUTER_APP_TITLE and OPENROUTER_APP_CATEGORIES in the run environment. (#1235, #1239)

Native auth — run on your own provider account

OpenRouter is the shared fallback; each harness can also run on its own provider, the same way grok runs on your X-account session. Two have real login flows captured for CI (the exact GROK_CREDENTIALS pattern — drive the login locally, store the session as a repo secret, restore it on the runner):

harness native auth how to set it
codex ChatGPT OAuth aeon auth --harness codex (or dashboard Connect ChatGPT) → CODEX_AUTH. Or an OpenAI key: --key sk-… → OPENAI_API_KEY
kimi Moonshot device login aeon auth --harness kimi (or Connect Kimi) → KIMI_AUTH. Or --key → MOONSHOT_API_KEY
pi provider API key aeon auth --harness pi --key <sk-ant-…|sk-…> → the matching ANTHROPIC_API_KEY / OPENAI_API_KEY (auto-detected)
vibe Mistral key aeon auth --harness vibe --key <key> → MISTRAL_API_KEY (vibe's default provider)
fx Vercel AI Gateway key (or VERCEL_OIDC_TOKEN) aeon auth --harness fx --key <key> -> AI_GATEWAY_API_KEY. No login flow and no OpenRouter fallback - see below.
cursor Cursor API key aeon auth --harness cursor --key <key> -> CURSOR_API_KEY; headless entry point is agent -p --trust because every CI run starts with a fresh home. --force remains write-mode only.
hermes Nous Portal OAuth aeon auth --harness hermes → HERMES_AUTH; the adapter restores ~/.hermes/auth.json.

Which one runs is decided at dispatch by which secret is set, native first, OpenRouter last (authSecretsForHarness / the HARNESS_AUTH registry in apps/dashboard/lib/harness-auth.ts). On native auth pi, vibe and kimi use their own default model, and the OpenRouter model picker applies only when the run falls back to OPENROUTER_API_KEY. cursor and hermes keep the picked model (both document model overrides). codex follows the pick too: on CODEX_AUTH or OPENAI_API_KEY an explicit OpenAI pick (config model:, per-skill, dispatch input or vars.HARNESS_MODEL) is forwarded as its bare id (openai/gpt-6-luna becomes --model gpt-6-luna); with no pick it keeps the account default. A ChatGPT plan does not serve every id, so when codex refuses the model the adapter retries once on the account default and warns codex refused model <id>; retried on the account default. The run records the model that actually ran, read from codex's session rollout. The workflow's Install harness CLI step restores/configures the selected provider; the CLI + dashboard flows share lib/harness-auth-server.ts.

Credentials at a glance

Generated from the credential manifest (harness-adapter/harnesses.json, credentials + default_model), which CI holds to scripts/resolve-harness.sh. Each harness uses the first secret in its list that is set; set any one. Connect a model in Aeon Connect or the local dashboard, and ./aeon init, walk you through the same list; bin/onboard checks it.

harness precedence (first set wins) default model
claude 1. CLAUDE_CODE_OAUTH_TOKEN - Claude subscription (Pro/Max) (login token; get, ./aeon auth --harness claude-code)
2. ANTHROPIC_API_KEY - Anthropic API key (pay as you go) (API key; get, ./aeon auth --key <sk-ant-api...>)
then any gateway key (openrouter, bankr, usepod, venice, surplus, grok, glm, hivemindos)		 claude-sonnet-5-5
codex 1. CODEX_AUTH - ChatGPT login (Plus/Pro) (login capture; get, ./aeon auth --harness codex)
2. OPENAI_API_KEY - OpenAI API key (API key; get, ./aeon auth --harness codex --key <sk-...>)
3. OPENROUTER_API_KEY - OpenRouter key (one key covers most harnesses) (API key; get, ./aeon secrets set OPENROUTER_API_KEY --stdin)		 openai/gpt-6-luna
cursor 1. CURSOR_API_KEY - Cursor API key (API key; get, ./aeon auth --harness cursor --key <key>) auto
fx 1. AI_GATEWAY_API_KEY - Vercel AI Gateway key (API key; get, ./aeon auth --harness fx --key <key>)
2. VERCEL_OIDC_TOKEN - Vercel OIDC token (OIDC token; get, vercel env pull)		 the harness's own default
grok 1. GROK_CREDENTIALS - X account login (grok login) (login capture; get, ./aeon auth --harness grok)
2. XAI_API_KEY - xAI API key (API key; get, ./aeon auth --harness grok --key <xai-...>)		 grok-4.7
hermes 1. HERMES_AUTH - Nous Portal login (login capture; get, ./aeon auth --harness hermes)
2. OPENROUTER_API_KEY - OpenRouter key (one key covers most harnesses) (API key; get, ./aeon secrets set OPENROUTER_API_KEY --stdin)		 the harness's own default
kimi 1. KIMI_AUTH - Kimi (Moonshot) login (login capture; get, ./aeon auth --harness kimi)
2. MOONSHOT_API_KEY - Moonshot API key (API key; get, ./aeon auth --harness kimi --key <sk-...>)
3. OPENROUTER_API_KEY - OpenRouter key (one key covers most harnesses) (API key; get, ./aeon secrets set OPENROUTER_API_KEY --stdin)		 moonshotai/kimi-k2.7-code
pi 1. ANTHROPIC_API_KEY - Anthropic API key (API key; get, ./aeon auth --harness pi --key <sk-ant-api...>)
2. ANTHROPIC_OAUTH_TOKEN - Claude subscription token (login token; get, ./aeon auth --harness pi --key <sk-ant-oat...>)
3. OPENAI_API_KEY - OpenAI API key (API key; get, ./aeon auth --harness pi --key <sk-...>)
4. OPENROUTER_API_KEY - OpenRouter key (one key covers most harnesses) (API key; get, ./aeon secrets set OPENROUTER_API_KEY --stdin)		 deepseek/deepseek-v4.1-flash
vibe 1. MISTRAL_API_KEY - Mistral API key (API key; get, ./aeon auth --harness vibe --key <key>)
2. OPENROUTER_API_KEY - OpenRouter key (one key covers most harnesses) (API key; get, ./aeon secrets set OPENROUTER_API_KEY --stdin)		 mistralai/mistral-medium-3-5

Notes: GROK_CREDENTIALS needs GH_GLOBAL (or the optional GH_SECRETS_PAT) to save its rotating refresh token after each run (see Token accounting below). cursor and fx have no OpenRouter fallback; with no key cursor reports AUTH_MODE=none and stops at install.

fx breaks the "OpenRouter last" rule above — it's the one harness with no OpenRouter fallback at all (confirmed: fx has no OpenRouter integration anywhere in its own docs). install-harness.sh fails closed at the Install harness CLI step with a named error if neither AI_GATEWAY_API_KEY nor VERCEL_OIDC_TOKEN is set, rather than staging a CLI that's guaranteed to fail later inside the actual run.

The full deployment runbook — the added/modified workflow steps, per-harness install and config, runner gotchas (AppArmor, the codex pin), and the measured reasons opencode/copilot/agy are excluded — is harness-adapter/docs/aeon-integration.md.

Verification status

The original six harnesses (claude, grok, codex, pi, vibe, kimi) were verified end-to-end through run-harness on 2026-07-22 — each dispatched live on GitHub Actions, exercising auth, read-only enforcement, token accounting, and the post-run health scorer:

harness auth as-tested read-only enforcement token usage live result
claude CLAUDE_CODE_OAUTH_TOKEN allowlist-strip + post-run revert (--no-sandbox) real green, both modes
grok GROK_CREDENTIALS (X-OAuth) allowlist-strip + post-run revert (--no-sandbox) real scored 4/5
codex OpenRouter (OPENROUTER_API_KEY) wrapper OS sandbox (codex's own sandbox off in read-only to avoid nesting) real 5/5 on gpt-5-mini, real fetch verified
pi OpenRouter wrapper OS sandbox real 5/5 on gpt-5-mini
vibe OpenRouter wrapper sandbox + --disabled-tools write_file,edit char/4 estimate 4/5 on gpt-5-mini
kimi OpenRouter wrapper OS sandbox (its only read-only guard) char/4 estimate Kimi K3 5/5, K2.5 & K2.7-code 4/5 — slates verified real

Notes from the sweep:

  • Model matters for the score. pi/vibe scored 2/5 on gpt-5-nano (the model punted / truncated) but 5/5 and 4/5 on gpt-5-mini — the harness scoring path is sound; nano was just too weak for the task.
  • Auth precedence is a trap. codex resolves CODEX_AUTH → OPENAI_API_KEY → OpenRouter and grok resolves GROK_CREDENTIALS → XAI_API_KEY, native-first — so a stale or quota-dead native secret keeps failing even when a working fallback is present. Delete the native secret to fall through. Deleting CODEX_AUTH is also how you pin a cheap model: the OpenRouter path forwards -f model=openai/gpt-5-*, while native auth uses the harness's own (pricier) default. (Since #1144 codex also honours an explicit pick on native auth, so this no longer applies to codex.)
  • The scorer grades stdout, not ./notify. A run that routes its deliverable into a channel and leaves a thin final message is under-graded even though the work was real (observed on codex/gpt-5-mini, which narrated pessimistically in stdout while its full slate went to notify). Keep the substance in the run's final message.

Fixes shipped during the sweep: harness unification through run-harness (#2), codex read-only inline notify (#4), keep-substance-in-stdout guidance (#5), kimi/vibe token estimate (#6), grok scorer fallback to grok-4.5 (#7), and codex read-only under the wrapper sandbox so fetches work (#14 — its native --sandbox read-only was also blocking the network).

fx was added later and was not part of the 2026-07-22 sweep above. It's verified locally against a real, locally-built fx binary (0.0.5): the missing-credential path, the mcpServers → fx's mcp.json shape translation, FX_MODEL/FX_MAX_AGENT_STEPS env resolution, and the full success/failure envelope paths (with a stubbed fx recording real argv) all pass. What it has not had is a live dispatch on GitHub Actions with a real AI_GATEWAY_API_KEY and a real model call — I don't have that credential to test with. Treat it as "should work, mechanically verified end-to-end short of a real model call" rather than "verified live" like the other six.

Token accounting

Every harness runs through run-harness, which normalizes usage into the Claude-Code {input, output, cache_read, cache_creation} shape. What each CLI exposes differs:

  • claude, grok, codex, pi report real usage. grok's adapter uses --output-format streaming-json, whose terminal {"type":"end"} event carries input/output/cache tokens + cost (harness-adapter/adapters/grok.sh) — the older scripts/run-grok.sh plain-json path reported 0, but the unified adapter does not.
  • fx also reports real usage, but not inline: fx ask --json doesn't carry it, so the adapter makes a second call, fx session <session_id> --json, to pull input_tokens/output_tokens/cache_read_tokens/total_cost once the first call succeeds. If that second call fails for any reason, the run still succeeds with its real output — usage just reports as 0/0/0/0 for that run rather than failing the whole thing over a usage lookup.
  • kimi and vibe expose no usage field (kimi's stream-json and vibe's --output json carry none — verified live), so their adapters fall back to a transparent char/4 estimate of the assembled input + result rather than a misleading 0/0/0; real counts always win if a future build emits them.

The captured OAuth sessions (grok's GROK_CREDENTIALS, codex's CODEX_AUTH) rotate their refresh tokens. grok's session is kept durable automatically (mirroring the MCP OAuth contract): the access token in GROK_CREDENTIALS lasts only 6h, and xAI rotates + revokes the refresh token on every refresh (confirmed live - auth.x.ai returns invalid_grant "Refresh token has been revoked"), so a static capture self-destructs ~6h after Connect. To fix that, scripts/run-grok.sh (§2b) refreshes the access token from the refresh token before each run and persists the rotated auth.json back to the GROK_CREDENTIALS secret. Persisting a secret needs a secrets-write credential - the default GITHUB_TOKEN cannot - so set GH_GLOBAL (the instance's classic PAT with repo + workflow, see Cross-repo access; ./aeon init sets it from your gh login). GH_SECRETS_PAT is optional and tried first, for keeping secrets-write on its own token. Without either, grok warns loudly and auth breaks one run after the first post-expiry refresh. After adding the PAT, re-connect the X account once to seed a valid refresh token (a token already consumed by a prior run can't be revived by the PAT alone). Concurrent grok runs that each hit an expired token still race - each refresh revokes the other's token - so for high parallelism refresh centrally on a schedule so exactly one run mints and persists per interval. codex's CODEX_AUTH just expires (no auto-persist yet) - re-capture via Connect ChatGPT or use the OpenAI key.

Capability mode carries over — but not the way you'd guess

A mode: read-only skill is read-only on grok, same as on Claude Code. The mechanism is different, and the difference matters if you are reasoning about blast radius.

Grok is the one harness that cannot be gated by an allowlist. Headless grok aborts the entire turn on a denied tool (stopReason=Cancelled, empty or partial output) instead of degrading the way Claude does, and skills are authored for Claude Code — they reach for tools no allowlist predicted. So harness-adapter/adapters/grok.sh runs --permission-mode bypassPermissions with no --allow and no --deny rules, deliberately. Grok's own --sandbox read-only is no help either: on grok 0.2.101 it is silently ignored (writes still land) and it nest-conflicts with the wrapper sandbox.

What actually enforces read-only is the dispatcher's OS sandbox (harness-adapter/lib/sandbox.sh) — bwrap --ro-bind write-locks the workspace for the whole run, network open — plus the workflow's post-run revert as defense-in-depth. That is the same enforcement every other harness gets, grok included. See CAPABILITIES.md for the three layers and which one is load-bearing.

Grok Build has no free tier — it needs a SuperGrok / X Premium+ subscription (OAuth) or xAI API credits (XAI_API_KEY).

Standing instructions

AGENTS.md is generated by scripts/gen-agents-md.js as a full, self-contained mirror of CLAUDE.md with every @import (e.g. @STRATEGY.md) expanded inline, because the non-claude harnesses load instruction files verbatim and don't expand Claude Code imports. Each run loads exactly one of the two: claude reads CLAUDE.md, every other harness (grok included) reads AGENTS.md. Grok natively discovers both files, so the workflow's "Single-source standing instructions" step hides CLAUDE.md for the run and the manual is never double-loaded. Regenerate with node scripts/gen-agents-md.js after editing CLAUDE.md or STRATEGY.md (--check verifies it in CI).

MCP on grok — and the --trust gate

Grok discovers the project .mcp.json natively (walking cwd→git-root) and expands ${VAR} from the environment itself — the same secrets the workflow's MCP preflight resolves — so no --mcp-config flag and no schema translation is needed. (Verified directly: a Bearer ${TOKEN} header arrives at the server fully expanded.) harness-adapter/adapters/grok.sh adds two things:

  • one --allow 'MCPTool(<server>__*)' per server, so the model can actually call the tools — MCP tools aren't auto-approved under a headless run;
  • --trust, without which none of the above ever happens.

Folder trust is the whole ballgame on CI. Grok gates every repo-local (project-scoped) MCP server behind its folder-trust store (~/.grok/trusted_folders.toml, the same gate that governs project hooks and LSP). A runner checks the repo out into a path that has never been trusted — on every single run — so without --trust the server is silently never started and no mcp__<server>__* tool exists.

The failure gives you almost nothing to go on: the run does not fail. The agent reports the server "not connected", falls back to plain HTTP, and finishes green with a plausible answer. grok mcp doctor is the one place that names it:

glim (http: https://glim.sh/mcp)
  ✗ folder untrusted (repo-local (project-scoped) server not started for an untrusted folder)
  → re-run with --trust to allow repo-local servers

--trust is passed only when an MCP config is in play, so a non-MCP run keeps the default untrusted posture. The flag is hidden on grok --help but accepted. (GROK_FOLDER_TRUST=0 disables the gate wholesale, but it ungates project hooks along with MCP — prefer the scoped flag.)

This is applied by harness-adapter/adapters/grok.sh, which every surface now goes through — see one run path below.

Trusting is sound here: the folder is the operator's own checked-out repo, which the harness is already executing as the agent's workspace.

(On a dev machine grok additionally sees your user-global MCP servers from ~/.claude.json/~/.cursor/mcp.json; CI runners are clean, so only the repo's .mcp.json applies.)

Other harnesses. MCP is not grok-only: claude, codex, vibe and kimi all call live MCP tools too (codex and kimi needed their own dispatcher fixes — see the harness-adapter README). pi joined them with its built-in MCP support (0.99+): the adapter translates .mcp.json into pi's mcp.json inside a temp PI_CODING_AGENT_DIR (the user's own ~/.pi/agent/mcp.json is never read or written) and declares each server's tools directly as mcp__<server>__<tool>, with - in the server name turned into _. pi waits for those servers before the first model request, but only up to a hard-coded 10s; a server slower than that misses the first turn. pi rejects the legacy sse transport, so sse entries are skipped with a warning (as are other types and server names outside letters, digits, _ and -); the run itself goes on. See footnote 9 in the harness-adapter README.

Newer grok knobs (opt-in per skill)

A skill's SKILL.md frontmatter can shape the grok run — ignored by the Claude harness:

max_turns: 120     # agentic-turn cap (default 60; a runaway/cost guard) → --max-turns
effort: high       # low|medium|high|xhigh|max → --effort  (reasoning models only)

effort/reasoning_effort map to the API's reasoningEffort, honoured by grok-4.7 (the default) and grok-4.6 (an older grok-4.5 pin still works): the reasoning models the X-account login exposes to the CLI (see Verification status; older xAI model ids are api.x.ai strings the CLI rejects as "unknown model id").

best_of_n and verify used to map to --best-of-n / --check. grok 1.x removed both flags (1.0.46 rejects them as "unexpected argument"), so the adapter now ignores those two keys with a notice and always runs with --no-subagents.

These knobs are read by harness-adapter/adapters/grok.sh, ported from run-grok.sh §3c.

There is exactly one run path. Every surface that runs a skill or a reply — scheduled and manual runs, the scorer, inbound messages, the local MCP server — goes through run-harness. scripts/run-grok.sh is now setup-only: it installs the pinned grok CLI and stages/refreshes GROK_CREDENTIALS (§2b), and callers invoke it as run-grok.sh setup.

That consolidation is deliberate. messages.yml and apps/mcp-server used to call run-grok.sh directly and so bypassed the adapter, which is how grok's MCP folder-trust gate stayed broken on those two surfaces after it was fixed for skill runs — a fix landed in one path and silently didn't reach the other. With one path, an adapter-level fix reaches every surface by construction.

Structured output is run-harness --json-schema, on every harness (native on claude/grok/codex, prompt-shim on pi/vibe/kimi). Callers that don't pass it get plain text. (A grok-only GROK_JSON_SCHEMA env var existed on the old script path until it was removed: nothing in the repo ever set it, and the scorer it was reserved for goes schema-less on purpose so a single parse path covers all nine harnesses.)

Both hosted surfaces stage all nine. aeon.yml (skill runs) and messages.yml (inbound messages) share the same two scripts, so a repo answers messages on the harness it runs skills on:

  • scripts/resolve-harness.sh — decides HARNESS / AUTH_MODE / HARNESS_MODEL / MODEL_ARG and prints them as KEY=VALUE lines. Pass a skill name to pick up per-skill harness:/model: overrides; omit it (as messages.yml does) and the repo-global keys decide.
  • scripts/install-harness.sh — stages that harness's CLI and provider auth.

Each workflow still declares its own env: block, because secrets.* only resolves inside a workflow — but the logic is shared, which is the half that drifted before. messages.yml used to carry a second, weaker copy that knew only claude and grok, so a repo on codex/pi/vibe/kimi had its messages answered on claude with a ::warning::. The local MCP server (apps/mcp-server) resolves all nine but expects the CLI to already be installed on your machine.

Every entry point runs on either harness

The harness split isn't just the scheduled skill run — it's wired through every surface that launches the agent, so a grok-only fork (no Claude credentials) behaves identically everywhere. All of them dispatch through run-harness:

Surface How grok is selected Notes
Scheduled / manual skill run (aeon.yml) dispatch Harness input → per-skill harness: → global harness: → claude full flags + MCP + scorer
Skill chains (chain-runner.yml) inherits — each step dispatches aeon.yml, which resolves per-skill/global
Inbound messages (messages.yml, Telegram/Discord/Slack) global harness: in aeon.yml conversational reply in write mode; the resolved harness's CLI is staged here (all nine), same as skill runs
Local MCP server (apps/mcp-server) AEON_HARNESS env → global harness: resolveHarness() in skill-executor.ts; resolves all nine, expects the CLI installed locally
Webhook (apps/webhook) relay only → dispatches messages.yml harness-agnostic
Post-run quality scorer (aeon.yml) scores through the same harness the skill ran on

Two surfaces stay Claude-only by design: the AI gateway (scripts/llm-gateway.sh) only reshapes the model behind Claude Code — grok has its own auth and bypasses it — and the json-render feed (notify-jsonrender) renders via claude -p and is skipped on grok (the feed is a display nicety; skill output, memory, and notifications are unaffected).