Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -302,6 +302,45 @@ Semver applies from 1.0.0. A breaking change to a documented API needs a major
the throws that caused it reached the job retry classifier as bare `Error`s with no code; and
restored `localStorage` was written on `about:blank` where it can never reach the site's origin.

- **An apostrophe in JSX text silently disabled the `errors` gate for a whole file, and it was not
hypothetical.** `maskLiterals` treated `'` as a literal opener and blanked everything up to the
next one, so `scanFixes` returned nothing for the rest of the file — while `scanCodes` kept
passing, masking the hole. `packages/http/src/errors.ts` contains
`…already route "${input.otherRoute}"'s`, so **eight real `fix:` lines in that file had never been
checked**. All eight pass now. One test asserted "nothing in the installed framework raises
`X_DRAINING`" — disproved by `draining()` in the very file the gate had stopped reading.

- **A page with zero executable JavaScript could fail its JS budget**, with a `fix:` line naming an
import that does not exist. Every inline `<script>` body was counted, including
`application/ld+json` (which the SEO helpers emit) and island props. `@ultimat3/render`'s `head.ts`
already owned the rule: *"the body is data, not code."*

- **`x verify --workers 5000` was accepted** although both flag summaries say "max 8", and
`planShards` clamps only to the file count — 842 concurrent Bun processes, each with the framework
module graph and a cloned database. Both summaries also named `CPUs - 1` as the default, a value
the code measured and rejected as *"slower than not sharding at all"*.

- **`agent()` sent the Anthropic API a transcript it rejects, in two places.** A turn emitting a tool
call **and** `respond` replayed the `respond` `tool_use` with no matching `tool_result` → 400. The
repair path had the same hole and is more reachable: **any** output-schema mismatch in an `agent()`
run was a 400. The loop now answers the superseded `respond` with an `is_error` result telling the
model to read the tool results and answer again — rather than discarding a block the model emitted,
or using an answer composed before the tools it called had run.

- **MCP `additionalProperties: false` accepted and dropped every argument named after an
`Object.prototype` member** (`constructor`, `__proto__`, `toString`), because the check walked the
prototype chain. Third instance of this class in one release, after `@ultimat3/i18n`'s catalog and
`@ultimat3/schema`'s coercion. `Object.hasOwn` alone was **not** sufficient: it turns the
`__proto__` *drop* into a `__proto__` *re-prototype* of the record the handler reads, so every
write now goes through `Object.defineProperty`.

- Also: a request arriving with an inbound `traceparent` never appeared in `/_x/timeline`; a second
`x dev` died with a bare `Error` on a port collision and `METRICS_PORT` was honoured in the
container but ignored in dev; a missing binary produced `fix: x doctor --json`, which checks
nothing about missing binaries; a root `tsconfig.json` written as JSONC silently disabled
`X_PACKAGE_UNREFERENCED`; and `x doctor --port 65535` suggested `--port 65536`, which `x dev`
refuses.

- **The outbox relay's claim locked nothing, so two relays could publish one batch and a job could
run twice.** `SQL_OUTBOX_CLAIM` ends in `for update skip locked`, but the relay issues it on a
**pooled** connection with no transaction — a bare statement runs in an implicit transaction that
Expand Down
34 changes: 34 additions & 0 deletions packages/ai/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ until 2026-08, naming a tool no catalog contained (`llm.test.ts`, `agent.test.ts
| `llm.ts` | `llm()` — the model call, declared as an `action`; and what a streamed answer must satisfy |
| `llm-stream.ts` | `.stream()`'s plumbing: the sink, the ambient mark, the one-turn drive |
| `agent.ts` | `agent()` — the tool loop, declared as an `action` |
| `agent-transcript.ts` | what one turn leaves in the transcript: the assistant replay, the tool results, the correction |
| `agent-facts.ts` | `describeAgents()` — the agent registry and the row a manifest publishes |
| `agent-job.ts` | `agentJob()` — an agent as a real `JobHandle`, composed from `job()` |
| `hive.ts` | `hive()` — one action fanned out over many inputs, declared as an `action` |
Expand Down Expand Up @@ -102,6 +103,12 @@ until 2026-08, naming a tool no catalog contained (`llm.test.ts`, `agent.test.ts
zero leaks the ceiling upward on every release.
- Cost is `Money` (integer minor units), rounded **up**. Never a float, never a division
that loses a fraction.
- **The gateway's two reads of a provider's throw are total.** A `Provider` is the APP's object, so
the value it rejects with is one the framework did not build: `isRetryable` indexes it (a getter,
or a `Proxy` trap) and fails closed if the read raises, and the failure line goes through core's
`renderThrowable` rather than `error.message` / `String(error)` — a renderer that throws replaces
`X_AI_PROVIDER_UNAVAILABLE` with a bare `TypeError` nothing catches by code, and it bounds a
provider's 1MB body out of the `cause`.
- Every non-2xx and every in-band `error` frame becomes `AiTransportError`, which carries a real
`status` field — that field IS the gateway's retry rule. A body parsed as a message would read
as an empty, successful answer, which is the one outcome nothing downstream can detect.
Expand Down Expand Up @@ -317,6 +324,26 @@ until 2026-08, naming a tool no catalog contained (`llm.test.ts`, `agent.test.ts
contract from turns in a loop, and half-shipping it would be the second path axiom 1 refuses.
- **No semantic cache on `agent()`.** Similar prompts do not have similar answers once the answer
depends on what `lookupOrder` returned this second.
- **Every `tool_use` block the transcript replays is answered by a `tool_result` in the very next
message** — the Messages API's own rule, and `agent-transcript.ts` owns both halves of it for
that reason. Two paths broke it, both through `respond`, which is filtered out of the calls
that RUN and replayed like any other block: a turn emitting a tool call AND `respond` together
(ordinary parallel tool use), and a `respond` whose input failed the output schema, followed by
a plain user message. Both were a 400 (`tool_use ids were found without tool_result blocks`),
i.e. `X_AI_PROVIDER_UNAVAILABLE` in place of a completed run, and `agent.test.ts` never mixed
the two so neither shipped visible. An unaccepted `respond` now comes back as its own
`tool_result`, `is_error`, saying why — the answer is SUPERSEDED, not wrong: it was written
before the results of the tools the same turn asked for existed, so the loop continues and the
model answers again with them in hand. Discarding the block instead would have been the other
legal fix and loses the record; USING the speculative answer would skip the tool results the
model itself asked for, after those tools already ran.
- **A tool result is rendered totally.** `runLlmToolCall` returns `content: string` and the loop
TRUNCATES it, so `JSON.stringify`'s other two answers both have to be handled: `undefined` for
an action that returns nothing (`'null'`), and a throw on a bigint, a cycle or a `toJSON` of
the value's own — reported as "the tool ran and its result is not JSON", never as a failure,
because a model told the tool failed calls it again and buys its side effects twice. The throw
it catches is read with `stringField`, never `typeof error.code === 'string'`: the value is an
app's, so the probe is a getter call or a `Proxy` trap inside the catch block.
- `AiMessage.content` widened to `string | readonly AiContentBlock[]` for this: a `tool_result`
has to name the `tool_use` it answers and a string has nowhere to put the id. The block field
names are the Messages API's, so `body()` passes them through untouched.
Expand Down Expand Up @@ -347,6 +374,13 @@ until 2026-08, naming a tool no catalog contained (`llm.test.ts`, `agent.test.ts
parallelism because `reserve` DEBITS on the root's turnstile before the call. Three members
against a ceiling only one fits leave exactly one `ok`; that is asserted through the hive
rather than asserted about the ledger, because the ledger already promised it.
- **A member's throw is RECORDED, whatever it is.** `failureOf` reads it with `isThrownError` and
`stringField` from core, never `error instanceof Error` and `.message`: a member is an app's
action, so a `Proxy` makes `instanceof` run a `getPrototypeOf` trap, and a throw there takes
down the whole hive — the one outcome the three arms exist to prevent. `skipped` has two
reasons, because they are two facts: `SKIPPED_ABORTED` (a sibling failed under `'abort'`) and
`SKIPPED_NO_INPUT` (the split produced nothing at that index). One string for both sent a
caller to retry a tail that was never cut.
- **`onMemberError` is required.** `'abort'` stops and leaves the rest `skipped`; `'collect'`
harvests. Both are right for somebody, so neither may be inherited silently.
- `concurrency` defaults to 4 and `minMembers` to 2, and neither number is measured off any run —
Expand Down
218 changes: 218 additions & 0 deletions packages/ai/src/agent-transcript.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,218 @@
/**
* The transcript the next turn replays, and the one rule the Messages API enforces about it:
* every `tool_use` block an assistant turn carries must be answered by a `tool_result` in the very
* next message. A transcript that breaks it is a 400 — an `X_AI_PROVIDER_UNAVAILABLE` in place of
* a completed run — so it is asserted structurally here rather than trusted per case.
*/

import { describe, expect, test } from 'bun:test';
import { createContext, userActor } from '@ultimat3/core';
import { allow } from '@ultimat3/policy';
import { t } from '@ultimat3/schema';
import { agent } from './agent';
import { createGateway } from './gateway';
import { definePrompt, type Prompt } from './prompt';
import type { AiMessage, GenerateRequest, GenerateResult, Provider, TokenUsage } from './provider';
import { costOf, EchoProvider } from './provider';
import { configureAi } from './runtime';
import type { ProjectableAction } from './tools';

const Input = t.object({ orderId: t.string });
const Output = t.object({ answer: t.string });
const USAGE: TokenUsage = {
inputTokens: 12,
outputTokens: 8,
cacheReadTokens: 0,
cacheWriteTokens: 0,
};

type Turn = { readonly calls: readonly { name: string; input: Record<string, unknown> }[] };

function scripted(...turns: readonly Turn[]): { provider: Provider; seen: GenerateRequest[] } {
const seen: GenerateRequest[] = [];
const provider: Provider = {
name: 'scripted',
models: ['claude-opus-5'],
generate(request) {
const turn = turns[Math.min(seen.length, turns.length - 1)];
seen.push(request);
return Promise.resolve({
model: 'claude-opus-5',
text: '',
toolCalls: (turn?.calls ?? []).map((call, index) => ({
id: `call-${seen.length}-${index}`,
name: call.name,
input: call.input,
})),
stopReason: 'tool_use',
stopDetails: undefined,
usage: USAGE,
cost: costOf('claude-opus-5', USAGE),
} satisfies GenerateResult);
},
stream: (request) => new EchoProvider().stream(request),
};
return { provider, seen };
}

function lookupTool(): ProjectableAction {
return {
name: 'lookupOrder',
description: 'Look an order up',
mcp: { expose: true },
run: () => Promise.resolve({ status: 'shipped' }),
};
}

let seq = 0;
function promptFor(): Prompt<{ orderId: string }> {
seq += 1;
return definePrompt<{ orderId: string }>({
id: `transcript-${seq}`,
version: '1.0.0',
template: 'Resolve order {{orderId}}.',
});
}

function ctxAs(id: string) {
return createContext({ actor: userActor({ id }) });
}

const blocksOf = (message: AiMessage | undefined) =>
Array.isArray(message?.content) ? message.content : [];

/**
* The API's rule, executable: the ids a transcript asks about and never answers. Each `tool_use`
* is looked up in the message that FOLLOWS its own, because that is the only place the Messages
* API accepts the answer.
*/
function unansweredToolUses(messages: readonly AiMessage[]): readonly string[] {
const open: string[] = [];
for (const [index, message] of messages.entries()) {
const asked = blocksOf(message)
.filter((block) => block.type === 'tool_use')
.map((block) => (block.type === 'tool_use' ? block.id : ''));
if (asked.length === 0) continue;
const answered = new Set(
blocksOf(messages[index + 1])
.filter((block) => block.type === 'tool_result')
.map((block) => (block.type === 'tool_result' ? block.tool_use_id : '')),
);
open.push(...asked.filter((id) => !answered.has(id)));
}
return open;
}

describe('every tool_use the transcript replays is answered', () => {
// Parallel tool use: one turn asks for a tool AND answers. The answer was written before the
// tool result existed, so the loop keeps going — and the `respond` block it replays needs a
// `tool_result` of its own or the next request is a 400.
test('a turn that calls a tool AND respond leaves no tool_use unanswered', async () => {
const { provider, seen } = scripted(
{
calls: [
{ name: 'lookupOrder', input: { id: 'o-1' } },
{ name: 'respond', input: { answer: 'guessed' } },
],
},
{ calls: [{ name: 'respond', input: { answer: 'shipped' } }] },
);
configureAi({ gateway: createGateway({ providers: [provider] }) });

const support = agent({
input: Input,
output: Output,
prompt: promptFor(),
vars: ({ input }) => ({ orderId: input.orderId }),
tools: [lookupTool()],
policy: allow(),
}).named('parallelAgent');

// The run completes on the SECOND turn's answer: the speculative one was written without the
// tool result the same turn asked for.
expect(await support({ orderId: 'o-1' }, { ctx: ctxAs('user-7') })).toEqual({
answer: 'shipped',
});
const replayed = seen[1]?.messages ?? [];
expect(unansweredToolUses(replayed)).toEqual([]);
// The real tool's result is still first, and the rejected answer is flagged so the model does
// not read it as data.
const results = blocksOf(replayed[2]);
expect(results[0]).toMatchObject({ type: 'tool_result', tool_use_id: 'call-1-0' });
expect(results[1]).toMatchObject({
type: 'tool_result',
tool_use_id: 'call-1-1',
is_error: true,
});
});

// The same hole on the repair path: a `respond` whose input fails the output schema is replayed
// as a `tool_use`, and the correction that follows it has to be that call's `tool_result`.
test('a repair turn answers the respond call it is correcting', async () => {
const { provider, seen } = scripted(
{ calls: [{ name: 'respond', input: { answer: 42 } }] },
{ calls: [{ name: 'respond', input: { answer: 'shipped' } }] },
);
configureAi({ gateway: createGateway({ providers: [provider] }) });

const support = agent({
input: Input,
output: Output,
prompt: promptFor(),
vars: ({ input }) => ({ orderId: input.orderId }),
tools: [lookupTool()],
policy: allow(),
}).named('repairAgent');

expect(await support({ orderId: 'o-1' }, { ctx: ctxAs('user-7') })).toEqual({
answer: 'shipped',
});
const replayed = seen[1]?.messages ?? [];
expect(unansweredToolUses(replayed)).toEqual([]);
const correction = blocksOf(replayed[2])[0];
expect(correction).toMatchObject({ type: 'tool_result', tool_use_id: 'call-1-0' });
expect(correction && 'content' in correction ? correction.content : '').toContain(
'failed its schema',
);
});

// A model that answers in prose emits no `tool_use` at all, so the correction stays an ordinary
// user message — a `tool_result` naming nothing would be the 400 in the other direction.
test('a prose answer is corrected with a plain user message', async () => {
const seen: GenerateRequest[] = [];
const provider: Provider = {
name: 'prose',
models: ['claude-opus-5'],
generate(request) {
seen.push(request);
return Promise.resolve({
model: 'claude-opus-5',
text: seen.length === 1 ? '{"answer":42}' : '{"answer":"shipped"}',
toolCalls: [],
stopReason: 'end_turn',
stopDetails: undefined,
usage: USAGE,
cost: costOf('claude-opus-5', USAGE),
} satisfies GenerateResult);
},
stream: (request) => new EchoProvider().stream(request),
};
configureAi({ gateway: createGateway({ providers: [provider] }) });

const support = agent({
input: Input,
output: Output,
prompt: promptFor(),
vars: ({ input }) => ({ orderId: input.orderId }),
tools: [lookupTool()],
policy: allow(),
}).named('proseAgent');

expect(await support({ orderId: 'o-1' }, { ctx: ctxAs('user-7') })).toEqual({
answer: 'shipped',
});
const replayed = seen[1]?.messages ?? [];
expect(unansweredToolUses(replayed)).toEqual([]);
expect(replayed[2]?.content).toContain('failed its schema');
});
});
Loading