diff --git a/docs/telemetry-privacy.md b/docs/telemetry-privacy.md index 52bb98fbcb..60ecc7f851 100644 --- a/docs/telemetry-privacy.md +++ b/docs/telemetry-privacy.md @@ -231,6 +231,69 @@ Leave `conversationLogPath` empty (or omit it) to use the default `/conver - `redactEmails` (boolean): Redact email addresses (default: `true`) - `redactPersonalInfo` (boolean): Redact PII patterns (default: `true`) +#### Client Performance Telemetry + +LLxprt can optionally collect **local client-side performance telemetry** — +timing data for client phases (prepare, stream handling, Ink rendering, stdout +writes, finalization), provider/tool activity intervals, and operation lifecycle +metadata. This data is written to local JSONL files and is **never transmitted +externally**. + +Both keys are **disabled by default**. To enable: + +```json +{ + "telemetry": { + "perf": { + "enabled": true, + "memory": true + } + } +} +``` + +- `telemetry.perf.enabled` (boolean): Master switch for performance telemetry. Default: `false`. When `false`, no perf files are created, no observers are installed, and no memory ring is allocated. +- `telemetry.perf.memory` (boolean): Include memory trend data (RSS, heap, external, array buffers) in perf records. Default: `false`. **Effective only when `enabled` is `true`** — memory is gated by the master switch. When perf is enabled but memory is off, operation records omit the memory columns entirely (absent, not zero-filled). + +`telemetry.perf` is an **object**, not a boolean. Setting it to `true` or `false` +directly is invalid. + +When perf telemetry is enabled, data is persisted to local JSONL files and is +**never transmitted externally**. + +- **Location**: the perf directory is `/perf`, where the global + log dir is `Storage.getGlobalLogDir()` (resolved from `LLXPRT_LOG_HOME`, then + `LLXPRT_CONFIG_HOME`, then the platform default — see + [Application Directories](./reference/application-directories.md)). Files are + named `perf-YYYYMMDD-.jsonl` (one per writer per UTC day). +- **What is recorded**: each `operation` record carries identity/build fields + (`session_id`, `operation_id`, `runtime_id`, `project_hash`, `llxprt_version`, + `git_sha`, `runtime`, `platform`), the comparison dimensions (`provider`, + `model`, `render_mode`, terminal geometry), token counts + (`context_tokens`, `output_tokens`), direct client-phase timing + (`client_prepare_ms`, `stream_handler_ms`, `ink_render_ms`, + `stdout_write_sync_ms`, `client_finalize_ms`), provider/tool activity + intervals, the terminal `status`, and `concurrent_instances`. When + `telemetry.perf.memory` is on, `memory_sample` rows additionally carry RSS, + heap, external, and array-buffer bytes with `uptime_ms`. Prompt/response text + is **not** recorded. +- **Retention**: an eventual bound of **64 MiB / 128 artifacts** (JSONL files + + claim files) is enforced oldest-first. A genuinely-live writer — today's UTC + day-key with an mtime within the maintenance window — is never evicted, and a + non-stale run claim survives while it is active; both still count toward the + caps. This lets a long-running process converge to the bounds by evicting its + own older files while its current file stays safe. +- **Inspection and management** (interactive `/perf` subcommands): + - `/perf` — current-process snapshot (live samples, active operation) when + perf is active in this process; otherwise reports it is not active. + - `/perf inspect` — directory path, schema version, privacy/default-off + statement, file/record counts, and self-health (skipped/truncated lines, + last write error, evictions). + - `/perf report [--baseline ]` — grouped p50 metrics by build + and comparison dimensions, with optional matched-dimension delta. + - `/perf delete` — removes old/stale perf artifacts (respecting live writers + and active claims). + ### Environment Variables You can also control telemetry through environment variables: diff --git a/packages/cli/src/__tests__/cliSessionDispatch.characterization.test.tsx b/packages/cli/src/__tests__/cliSessionDispatch.characterization.test.tsx index 3becc9403b..002db6d35c 100644 --- a/packages/cli/src/__tests__/cliSessionDispatch.characterization.test.tsx +++ b/packages/cli/src/__tests__/cliSessionDispatch.characterization.test.tsx @@ -167,6 +167,7 @@ function createMinimalConfig(options: { refreshAuth: vi.fn(async () => {}), setEphemeralSetting: vi.fn(), getEphemeralSetting: vi.fn(() => undefined), + getTelemetrySettings: () => ({ perf: { enabled: false, memory: false } }), }; } diff --git a/packages/cli/src/cli.provider-init.test.ts b/packages/cli/src/cli.provider-init.test.ts index 3314f0e963..adffe745c2 100644 --- a/packages/cli/src/cli.provider-init.test.ts +++ b/packages/cli/src/cli.provider-init.test.ts @@ -265,6 +265,9 @@ describe('cli main provider initialization', () => { setTerminalBackground: vi.fn(), getPolicyEngine: vi.fn(() => null), + getTelemetrySettings: vi.fn(() => ({ + perf: { enabled: false, memory: false }, + })), } as unknown as Config; const { loadCliConfig } = await import('./config/config.js'); @@ -363,6 +366,9 @@ describe('cli main provider initialization', () => { getAgentClient, setTerminalBackground: vi.fn(), getPolicyEngine: vi.fn(() => null), + getTelemetrySettings: vi.fn(() => ({ + perf: { enabled: false, memory: false }, + })), } as unknown as Config; const resumeResult = makeResumeResult('restored user content'); @@ -495,6 +501,9 @@ describe('cli main provider initialization', () => { getAgentClient, setTerminalBackground: vi.fn(), getPolicyEngine: vi.fn(() => null), + getTelemetrySettings: vi.fn(() => ({ + perf: { enabled: false, memory: false }, + })), } as unknown as Config; const resumeResult = makeResumeResult('restored user content'); diff --git a/packages/cli/src/cli.startInteractiveUI.test.tsx b/packages/cli/src/cli.startInteractiveUI.test.tsx index 98d26353db..1579ae2b4b 100644 --- a/packages/cli/src/cli.startInteractiveUI.test.tsx +++ b/packages/cli/src/cli.startInteractiveUI.test.tsx @@ -143,6 +143,9 @@ describe('startInteractiveUI', () => { storage: {}, getDebugMode: () => false, getTerminalBackground: () => undefined, + // Perf disabled: buildAndStartPerfOwner reads getTelemetrySettings() and + // returns null (no perf owner) without touching any other runtime seam. + getTelemetrySettings: () => ({ perf: { enabled: false } }), } as Config; const mockAgent = { dispose: vi.fn().mockResolvedValue(undefined), diff --git a/packages/cli/src/cli.test.tsx b/packages/cli/src/cli.test.tsx index 8c1362fbcb..5a126e4709 100644 --- a/packages/cli/src/cli.test.tsx +++ b/packages/cli/src/cli.test.tsx @@ -80,9 +80,9 @@ void vi.mock('./ui/utils/terminalCapabilityManager.js', () => ({ void vi.mock('./config/config.js', () => ({ loadCliConfig: vi.fn().mockResolvedValue({ - getSandbox: vi.fn(() => false), - getQuestion: vi.fn(() => ''), - getProvider: vi.fn(() => undefined), + getSandbox: () => false, + getQuestion: () => '', + getProvider: () => undefined, } as unknown as Config), })); @@ -503,10 +503,7 @@ describe('cli.tsx main function', () => { getIdeClient: vi.fn(() => null), getListExtensions: vi.fn(() => false), getOutputFormat: vi.fn(() => OutputFormat.TEXT), - getToolRegistryInfo: vi.fn(() => ({ - registered: [], - unregistered: [], - })), + getToolRegistryInfo: vi.fn(() => ({ registered: [], unregistered: [] })), getSandbox: vi.fn(() => false), getModel: vi.fn(() => 'gemini-2.5-pro'), getProjectRoot: vi.fn(() => '/tmp/project'), @@ -527,6 +524,9 @@ describe('cli.tsx main function', () => { setTerminalBackground: vi.fn(), getTerminalBackground: vi.fn(() => undefined), getPolicyEngine: vi.fn(() => null), + getTelemetrySettings: vi.fn(() => ({ + perf: { enabled: false, memory: false }, + })), } as unknown as Config; const loadSettingsMock = loadSettings as Mock; @@ -590,7 +590,9 @@ describe('cli.tsx main function', () => { quiet: undefined, }); - const renderMock = vi.fn().mockReturnValue({ unmount: vi.fn() }); + const renderMock = vi + .fn() + .mockReturnValue({ clear: vi.fn(), unmount: vi.fn() }); __setRenderForTesting(renderMock); const originalIsTTY = process.stdin.isTTY; @@ -665,10 +667,7 @@ describe('cli.tsx main function', () => { getIdeClient: vi.fn(() => null), getListExtensions: vi.fn(() => false), getOutputFormat: vi.fn(() => OutputFormat.TEXT), - getToolRegistryInfo: vi.fn(() => ({ - registered: [], - unregistered: [], - })), + getToolRegistryInfo: vi.fn(() => ({ registered: [], unregistered: [] })), getSandbox: vi.fn(() => false), getModel: vi.fn(() => 'gemini-2.5-pro'), getProjectRoot: vi.fn(() => '/tmp/project'), @@ -689,6 +688,9 @@ describe('cli.tsx main function', () => { setTerminalBackground: vi.fn(), getTerminalBackground: vi.fn(() => undefined), getPolicyEngine: vi.fn(() => null), + getTelemetrySettings: vi.fn(() => ({ + perf: { enabled: false, memory: false }, + })), } as unknown as Config; const loadSettingsMock = loadSettings as Mock; diff --git a/packages/cli/src/config/configBuilder.ts b/packages/cli/src/config/configBuilder.ts index e1b4187386..7b382430a1 100644 --- a/packages/cli/src/config/configBuilder.ts +++ b/packages/cli/src/config/configBuilder.ts @@ -89,6 +89,7 @@ function buildTelemetryConfig(argv: CliArgs, settings: Settings) { enabled: argv.telemetry ?? telemetrySettings?.enabled, logPrompts: argv.telemetryLogPrompts ?? telemetrySettings?.logPrompts, outfile: argv.telemetryOutfile ?? telemetrySettings?.outfile, + perf: telemetrySettings?.perf, ...buildTelemetryRedactionConfig(telemetrySettings), }; } diff --git a/packages/cli/src/config/perfSettingsMerge.behavior.test.ts b/packages/cli/src/config/perfSettingsMerge.behavior.test.ts new file mode 100644 index 0000000000..3542a6bf58 --- /dev/null +++ b/packages/cli/src/config/perfSettingsMerge.behavior.test.ts @@ -0,0 +1,166 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + * + * Behavioral tests for the real settings merge pipeline as it applies to + * telemetry.perf. Uses production mergeSettings code (no mocks) to prove + * the actual precedence and merge semantics. + * + * EVIDENCE-AC2: persisted settings merge precedence for telemetry.perf. + */ + +import { describe, it, expect } from 'bun:test'; +import { mergeSettings } from './settingsMerge.js'; +import type { Settings } from './settingsSchema.js'; + +function emptySettings(): Settings { + return {} as Settings; +} + +describe('telemetry.perf — real mergeSettings behavior', () => { + describe('absent in all layers', () => { + it('produces no perf key in merged telemetry', () => { + const merged = mergeSettings( + emptySettings(), + emptySettings(), + emptySettings(), + emptySettings(), + true, + ); + expect(merged.telemetry.perf).toBeUndefined(); + }); + }); + + describe('user-only perf', () => { + it('user telemetry.perf.enabled flows through merge', () => { + const user = { + telemetry: { perf: { enabled: true } }, + } as Settings; + const merged = mergeSettings( + emptySettings(), + emptySettings(), + user, + emptySettings(), + true, + ); + expect(merged.telemetry.perf).toEqual({ enabled: true }); + }); + }); + + describe('workspace overrides user (higher precedence)', () => { + it('workspace telemetry.perf replaces user telemetry.perf (shallow merge at telemetry level)', () => { + // The established merge behavior is shallow-spread for the telemetry + // object section. This means a higher-precedence layer's perf object + // replaces the lower-precedence one entirely (documented, not a defect). + const user = { + telemetry: { perf: { enabled: true } }, + } as Settings; + const workspace = { + telemetry: { perf: { memory: true } }, + } as Settings; + const merged = mergeSettings( + emptySettings(), + emptySettings(), + user, + workspace, + true, + ); + // Shallow merge: workspace.perf replaces user.perf entirely + expect(merged.telemetry.perf).toEqual({ memory: true }); + }); + }); + + describe('both layers set perf.enabled', () => { + it('workspace perf.enabled wins over user perf.enabled', () => { + const user = { + telemetry: { perf: { enabled: false } }, + } as Settings; + const workspace = { + telemetry: { perf: { enabled: true } }, + } as Settings; + const merged = mergeSettings( + emptySettings(), + emptySettings(), + user, + workspace, + true, + ); + expect(merged.telemetry.perf?.enabled).toBe(true); + }); + }); + + describe('telemetry scalar fields still merge across layers', () => { + it('user telemetry.enabled and workspace telemetry.perf coexist', () => { + const user = { + telemetry: { enabled: true }, + } as Settings; + const workspace = { + telemetry: { perf: { enabled: true, memory: true } }, + } as Settings; + const merged = mergeSettings( + emptySettings(), + emptySettings(), + user, + workspace, + true, + ); + expect(merged.telemetry.enabled).toBe(true); + expect(merged.telemetry.perf).toEqual({ enabled: true, memory: true }); + }); + }); + + describe('untrusted workspace is ignored', () => { + it('workspace telemetry.perf is not applied when isTrusted=false', () => { + const workspace = { + telemetry: { perf: { enabled: true } }, + } as Settings; + const merged = mergeSettings( + emptySettings(), + emptySettings(), + emptySettings(), + workspace, + false, + ); + expect(merged.telemetry.perf).toBeUndefined(); + }); + }); + + describe('system layer (highest file precedence)', () => { + it('system telemetry.perf wins over user and workspace', () => { + const system = { + telemetry: { perf: { enabled: true, memory: true } }, + } as Settings; + const workspace = { + telemetry: { perf: { enabled: false } }, + } as Settings; + const merged = mergeSettings( + system, + emptySettings(), + emptySettings(), + workspace, + true, + ); + expect(merged.telemetry.perf).toEqual({ enabled: true, memory: true }); + }); + }); + + describe('system defaults layer', () => { + it('systemDefaults telemetry.perf is overridden by user telemetry.perf', () => { + const systemDefaults = { + telemetry: { perf: { enabled: true } }, + } as Settings; + const user = { + telemetry: { perf: { enabled: false, memory: false } }, + } as Settings; + const merged = mergeSettings( + emptySettings(), + systemDefaults, + user, + emptySettings(), + true, + ); + expect(merged.telemetry.perf).toEqual({ enabled: false, memory: false }); + }); + }); +}); diff --git a/packages/cli/src/config/perfSettingsValidation.behavior.test.ts b/packages/cli/src/config/perfSettingsValidation.behavior.test.ts new file mode 100644 index 0000000000..03cd91dc32 --- /dev/null +++ b/packages/cli/src/config/perfSettingsValidation.behavior.test.ts @@ -0,0 +1,147 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + * + * Behavioral tests for settings validation of telemetry.perf. + * Tests the actual Zod validation pipeline (no mocks) to prove + * that the persisted shape is accepted/rejected correctly. + * + * EVIDENCE-AC2: settings validation acceptance/rejection for the perf object. + */ + +import { describe, it, expect } from 'bun:test'; +import { validateSettings } from './settings-validation.js'; + +describe('settings validation — telemetry.perf shape', () => { + describe('accepted shapes', () => { + it('accepts telemetry.perf as an object with enabled and memory booleans', () => { + const result = validateSettings({ + telemetry: { + perf: { enabled: true, memory: true }, + }, + }); + expect(result.success).toBe(true); + }); + + it('accepts telemetry.perf with only enabled', () => { + const result = validateSettings({ + telemetry: { + perf: { enabled: true }, + }, + }); + expect(result.success).toBe(true); + }); + + it('accepts telemetry.perf with only memory', () => { + const result = validateSettings({ + telemetry: { + perf: { memory: true }, + }, + }); + expect(result.success).toBe(true); + }); + + it('accepts telemetry.perf as an empty object', () => { + const result = validateSettings({ + telemetry: { + perf: {}, + }, + }); + expect(result.success).toBe(true); + }); + + it('accepts telemetry with both false booleans', () => { + const result = validateSettings({ + telemetry: { + perf: { enabled: false, memory: false }, + }, + }); + expect(result.success).toBe(true); + }); + + it('accepts telemetry without perf alongside other telemetry fields', () => { + const result = validateSettings({ + telemetry: { + enabled: true, + logPrompts: true, + }, + }); + expect(result.success).toBe(true); + }); + }); + + describe('rejected shapes', () => { + it('rejects telemetry.perf as a boolean true (D2: perf is NOT a boolean)', () => { + const result = validateSettings({ + telemetry: { + perf: true, + }, + }); + expect(result.success).toBe(false); + }); + + it('rejects telemetry.perf as a boolean false (D2: perf is NOT a boolean)', () => { + const result = validateSettings({ + telemetry: { + perf: false, + }, + }); + expect(result.success).toBe(false); + }); + + it('rejects telemetry.perf.enabled as a non-boolean', () => { + const result = validateSettings({ + telemetry: { + perf: { enabled: 'yes' }, + }, + }); + expect(result.success).toBe(false); + }); + + it('rejects telemetry.perf.memory as a non-boolean', () => { + const result = validateSettings({ + telemetry: { + perf: { memory: 1 }, + }, + }); + expect(result.success).toBe(false); + }); + + it('rejects telemetry.perf with unknown properties (additionalProperties: false)', () => { + const result = validateSettings({ + telemetry: { + perf: { enabled: true, extra: 'field' }, + }, + }); + expect(result.success).toBe(false); + }); + + it('rejects telemetry.perf as a string', () => { + const result = validateSettings({ + telemetry: { + perf: 'enabled', + }, + }); + expect(result.success).toBe(false); + }); + + it('rejects telemetry.perf as a number', () => { + const result = validateSettings({ + telemetry: { + perf: 1, + }, + }); + expect(result.success).toBe(false); + }); + + it('rejects telemetry.perf as an array', () => { + const result = validateSettings({ + telemetry: { + perf: [true], + }, + }); + expect(result.success).toBe(false); + }); + }); +}); diff --git a/packages/cli/src/config/settingsSchema.ts b/packages/cli/src/config/settingsSchema.ts index e46cac6777..c3b1c4f75b 100644 --- a/packages/cli/src/config/settingsSchema.ts +++ b/packages/cli/src/config/settingsSchema.ts @@ -154,6 +154,26 @@ export const SETTINGS_SCHEMA_DEFINITIONS: Record< type: 'string', description: 'File path for writing telemetry output.', }, + perf: { + type: 'object', + description: + 'Client-side performance telemetry (local-only, default off). ' + + 'When enabled, timing and resource data is written to local perf files. ' + + 'memory requires enabled to be true.', + additionalProperties: false, + properties: { + enabled: { + type: 'boolean', + description: + 'Master switch for local client performance telemetry. Default false.', + }, + memory: { + type: 'boolean', + description: + 'Include memory trend data in perf records. Effective only when enabled is true. Default false.', + }, + }, + }, }, }, SubagentDefinition: { diff --git a/packages/cli/src/services/BuiltinCommandLoader.test.ts b/packages/cli/src/services/BuiltinCommandLoader.test.ts index 55394600d8..78f635ae2c 100644 --- a/packages/cli/src/services/BuiltinCommandLoader.test.ts +++ b/packages/cli/src/services/BuiltinCommandLoader.test.ts @@ -127,6 +127,7 @@ describe('BuiltinCommandLoader', () => { getSkillManager: vi.fn().mockReturnValue({ getAllSkills: vi.fn().mockReturnValue([]), }), + getProjectTempDir: () => '/tmp/llxprt-test-project', } as unknown as Config; restoreCommandMock.mockReturnValue({ @@ -247,6 +248,7 @@ describe('BuiltinCommandLoader profile', () => { getSkillManager: vi.fn().mockReturnValue({ getAllSkills: vi.fn().mockReturnValue([]), }), + getProjectTempDir: () => '/tmp/llxprt-test-project', } as unknown as Config; }); diff --git a/packages/cli/src/services/BuiltinCommandLoader.ts b/packages/cli/src/services/BuiltinCommandLoader.ts index 13d84dd500..09ced535c4 100644 --- a/packages/cli/src/services/BuiltinCommandLoader.ts +++ b/packages/cli/src/services/BuiltinCommandLoader.ts @@ -4,6 +4,7 @@ * SPDX-License-Identifier: Apache-2.0 */ +import { join } from 'node:path'; import { isDevelopment } from '../utils/installationInfo.js'; import type { SlashCommand, @@ -69,6 +70,7 @@ import { todoCommand } from '../ui/commands/todoCommand.js'; import { setupCommand } from '../ui/commands/setupCommand.js'; import { tasksCommands } from '../ui/commands/tasksCommand.js'; import { hooksCommand } from '../ui/commands/hooksCommand.js'; +import { createPerfCommand } from '../ui/commands/perfCommand.js'; /** * @plan PLAN-20260214-SESSIONBROWSER.P21 */ @@ -191,6 +193,12 @@ export class BuiltinCommandLoader implements ICommandLoader { * @plan PLAN-20260214-SESSIONBROWSER.P21 */ continueCommand, + createPerfCommand({ + snapshotCapability: this.config?.getPerfSnapshotCapability?.() ?? null, + tokenUsageDir: this.config + ? join(this.config.getProjectTempDir(), 'token-usage') + : undefined, + }), ]; return allDefinitions.filter((cmd): cmd is SlashCommand => cmd !== null); diff --git a/packages/cli/src/services/cliCommandApiMap.ts b/packages/cli/src/services/cliCommandApiMap.ts new file mode 100644 index 0000000000..4dfc77b48b --- /dev/null +++ b/packages/cli/src/services/cliCommandApiMap.ts @@ -0,0 +1,55 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * CLI-owned command-API boundary extensions (issue #3167 / P13). + * + * The canonical {@link COMMAND_API_MAP} in `@vybestack/llxprt-code-agents` + * classifies every touchpoint that has an agents/core runtime or durable + * app-service surface. Some commands live ENTIRELY in the CLI package and have + * no agents or core runtime dependency — they are pure CLI-local telemetry/UI + * operations. Those are classified here rather than in the agents map so the + * agents package stays agent-neutral. + * + * The completeness test combines this array with the agents COMMAND_API_MAP to + * validate a single unified boundary map with uniqueness and orphan checks. + */ + +/** + * CLI-local command-API entries for commands that have no agents/core surface. + * These are `cli-local`: pure UI/telemetry operations with no live Agent method + * and no durable app-service function. + * + * `/perf` and its subcommands read/write telemetry data files and display live + * process snapshots. They never invoke an Agent method or mutate durable + * app-service state, so they are CLI-local — not runtime, not subpath. + */ +export const CLI_COMMAND_API_EXTENSIONS = [ + { + command: '/perf', + kind: 'cli-local', + target: 'perf snapshot (UI)', + note: 'Live process perf snapshot; CLI-local telemetry display', + }, + { + command: '/perf inspect', + kind: 'cli-local', + target: 'perf inspect (UI)', + note: 'Perf data location and sample counts; CLI-local telemetry file read', + }, + { + command: '/perf report', + kind: 'cli-local', + target: 'perf report (UI)', + note: 'Longitudinal perf trends; CLI-local telemetry file read', + }, + { + command: '/perf delete', + kind: 'cli-local', + target: 'perf delete (UI)', + note: 'Delete perf data files; CLI-local telemetry file operation', + }, +] as const; diff --git a/packages/cli/src/services/commandApiMapCompleteness.test.ts b/packages/cli/src/services/commandApiMapCompleteness.test.ts index 6e09c36dca..0aaea69b7a 100644 --- a/packages/cli/src/services/commandApiMapCompleteness.test.ts +++ b/packages/cli/src/services/commandApiMapCompleteness.test.ts @@ -9,20 +9,36 @@ * @requirement:REQ-021 * * Command-map completeness test (#2203 / REQ-021). Every registered CLI slash - * command (top-level and sub-command) must appear in COMMAND_API_MAP or be - * excluded via the CONFIG_GATED_COMMANDS set (commands not loaded with null - * config). This prevents future drift between the command registry and the - * classification map. + * command (top-level and sub-command) must appear in the combined + * command→API map (the agents-canonical COMMAND_API_MAP plus CLI-owned + * extensions for commands with no agents/core surface) or be excluded via + * the CONFIG_GATED_COMMANDS set (commands not loaded with null config). This + * prevents future drift between the command registry and the classification + * map. * - * The COMMAND_API_MAP classifies each command as runtime / subpath / cli-local + * The combined map classifies each command as runtime / subpath / cli-local * and is the canonical source of truth for the runtime-vs-app-service boundary. + * CLI-local commands that have no agents/core dependency (e.g. `/perf` and its + * subcommands) are owned in the CLI package via CLI_COMMAND_API_EXTENSIONS so + * the agents package stays agent-neutral. */ import { describe, it, expect } from 'bun:test'; import { COMMAND_API_MAP } from '@vybestack/llxprt-code-agents/app-service.js'; +import { CLI_COMMAND_API_EXTENSIONS } from './cliCommandApiMap.js'; import type { SlashCommand } from '../ui/commands/types.js'; import { BuiltinCommandLoader } from './BuiltinCommandLoader.js'; +/** + * The unified boundary map: the agents-canonical COMMAND_API_MAP augmented + * with CLI-owned extensions for commands that have no agents/core surface. + * Every completeness, orphan, and uniqueness check below validates this + * combined map so there is a single source of truth at the test boundary. + */ +const COMBINED_COMMAND_API_MAP: ReadonlyArray< + (typeof COMMAND_API_MAP)[number] +> = [...COMMAND_API_MAP, ...CLI_COMMAND_API_EXTENSIONS]; + /** * Config-gated commands that are not loaded when config is null. These are * intentionally excluded from the registered-command completeness check @@ -40,7 +56,7 @@ const CONFIG_GATED_COMMANDS: readonly string[] = [ /** * Known subcommands of config-gated commands. These are not loaded by * BuiltinCommandLoader(null), so they are verified explicitly here to - * ensure they have COMMAND_API_MAP entries. + * ensure they have map entries. */ const GATED_SUBCOMMANDS: readonly string[] = [ '/hooks list', @@ -51,10 +67,10 @@ const GATED_SUBCOMMANDS: readonly string[] = [ ]; /** - * Conceptual entries in COMMAND_API_MAP that do not correspond to literal slash - * commands but are required by the boundary test (app-service functions invoked - * via dialogs or internal actions, not via /command). These are intentionally - * excluded from the reverse orphan check. + * Conceptual entries in the combined map that do not correspond to literal + * slash commands but are required by the boundary test (app-service functions + * invoked via dialogs or internal actions, not via /command). These are + * intentionally excluded from the reverse orphan check. */ const CONCEPTUAL_COMMANDS: readonly string[] = [ '/mcp add', @@ -66,11 +82,13 @@ const CONCEPTUAL_COMMANDS: readonly string[] = [ /** * Returns true when the given command path is covered by at least one entry in - * COMMAND_API_MAP. A command is covered if there is an entry whose command + * the combined map. A command is covered if there is an entry whose command * field exactly matches the path. */ function isCommandMapped(commandPath: string): boolean { - return COMMAND_API_MAP.some((entry) => entry.command === commandPath); + return COMBINED_COMMAND_API_MAP.some( + (entry) => entry.command === commandPath, + ); } /** @@ -112,14 +130,14 @@ describe('Command-map completeness (#2203 / REQ-021)', () => { expect(checkablePaths.length).toBeGreaterThan(0); }); - it('every registered command has a COMMAND_API_MAP entry', () => { + it('every registered command has a combined-map entry', () => { const unmapped = checkablePaths.filter((p) => !isCommandMapped(p)); expect(unmapped).toStrictEqual([]); }); - it('no orphaned slash-command entries exist in COMMAND_API_MAP', () => { - const slashEntries = COMMAND_API_MAP.map((e) => e.command).filter((cmd) => - cmd.startsWith('/'), + it('no orphaned slash-command entries exist in the combined map', () => { + const slashEntries = COMBINED_COMMAND_API_MAP.map((e) => e.command).filter( + (cmd) => cmd.startsWith('/'), ); const knownPaths = new Set([ ...checkablePaths, @@ -131,14 +149,16 @@ describe('Command-map completeness (#2203 / REQ-021)', () => { expect(orphans).toStrictEqual([]); }); - it('no two COMMAND_API_MAP entries share the same command string', () => { - const names = COMMAND_API_MAP.map((e) => e.command); + it('no two combined-map entries share the same command string', () => { + const names = COMBINED_COMMAND_API_MAP.map((e) => e.command); const duplicates = names.filter((name, idx) => names.indexOf(name) !== idx); expect(duplicates).toHaveLength(0); }); it('every subpath entry targets the pinned specifier with a named export', () => { - const subpathEntries = COMMAND_API_MAP.filter((e) => e.kind === 'subpath'); + const subpathEntries = COMBINED_COMMAND_API_MAP.filter( + (e) => e.kind === 'subpath', + ); expect(subpathEntries.length).toBeGreaterThan(0); for (const entry of subpathEntries) { expect(entry.target).toBe('@vybestack/llxprt-code-agents/app-service.js'); @@ -147,7 +167,7 @@ describe('Command-map completeness (#2203 / REQ-021)', () => { } }); - it('config-gated commands and their subcommands are tracked in COMMAND_API_MAP', () => { + it('config-gated commands and their subcommands are tracked in the combined map', () => { for (const gated of CONFIG_GATED_COMMANDS) { expect(isCommandMapped(gated)).toBe(true); } @@ -155,4 +175,17 @@ describe('Command-map completeness (#2203 / REQ-021)', () => { expect(isCommandMapped(sub)).toBe(true); } }); + + it('CLI extensions do not duplicate agents-map command strings', () => { + const agentsCommands = new Set(COMMAND_API_MAP.map((e) => e.command)); + for (const ext of CLI_COMMAND_API_EXTENSIONS) { + expect(agentsCommands.has(ext.command)).toBe(false); + } + }); + + it('every CLI extension is classified cli-local', () => { + for (const ext of CLI_COMMAND_API_EXTENSIONS) { + expect(ext.kind).toBe('cli-local'); + } + }); }); diff --git a/packages/cli/src/session/buildPerfOwner.behavior.test.ts b/packages/cli/src/session/buildPerfOwner.behavior.test.ts new file mode 100644 index 0000000000..0355247d3f --- /dev/null +++ b/packages/cli/src/session/buildPerfOwner.behavior.test.ts @@ -0,0 +1,176 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * P12 behavioral tests for buildAndStartPerfOwner disabled path (Item 5). + * + * Calls the REAL buildAndStartPerfOwner with perf disabled. Uses lazy object + * methods (that throw if called) to prove ONLY getTelemetrySettings() is read + * on the disabled path. Also proves no perf directory/artifacts/observers and + * no timers/timing/memory APIs. + * + * The production buildAndStartPerfOwner reads config/agent lazily through + * getters — but because config/agent arguments are evaluated before entry, + * the test passes lazy object methods that prove all non-setting methods + * are untouched. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import { promises as fsp } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { + buildAndStartPerfOwner, + type PerfOwnerConfigCapability, + type PerfOwnerAgentCapability, +} from './interactiveUI.js'; +import { LoadedSettings } from '../config/settings.js'; +import { + getInteractiveStdoutObserver, + getInteractiveRenderObserver, + setInteractiveStdoutObserver, + setInteractiveRenderObserver, +} from '../ui/inkRenderOptions.js'; +import { + getPerfPhaseObserver, + setPerfPhaseObserver, +} from '@vybestack/llxprt-code-telemetry/perf/perfPhaseObserver.js'; + +let dir: string; + +beforeEach(() => { + dir = fs.mkdtempSync(join(tmpdir(), 'perf-disabled-')); + setInteractiveStdoutObserver(null); + setInteractiveRenderObserver(null); + setPerfPhaseObserver(null); +}); + +afterEach(async () => { + setInteractiveStdoutObserver(null); + setInteractiveRenderObserver(null); + setPerfPhaseObserver(null); + try { + await fsp.rm(dir, { recursive: true, force: true }); + } catch { + // ignore + } +}); + +/** + * Creates a PerfOwnerConfigCapability where every method EXCEPT + * getTelemetrySettings throws if called. This proves the disabled path + * touches only getTelemetrySettings(). + */ +function makeThrowingConfig(): PerfOwnerConfigCapability & { + telemetryCallCount: () => number; +} { + let telemetryCalls = 0; + return { + getTelemetrySettings() { + telemetryCalls++; + // Perf disabled: telemetry settings without perf.enabled. + return { perf: { enabled: false } }; + }, + getSessionId() { + throw new Error('getSessionId should not be called when perf disabled'); + }, + getProjectRoot() { + throw new Error('getProjectRoot should not be called when perf disabled'); + }, + getScreenReader() { + throw new Error( + 'getScreenReader should not be called when perf disabled', + ); + }, + telemetryCallCount: () => telemetryCalls, + }; +} + +/** + * Creates a PerfOwnerAgentCapability where every method throws if called. + */ +function makeThrowingAgent(): PerfOwnerAgentCapability { + return { + getRuntimeId() { + throw new Error('getRuntimeId should not be called when perf disabled'); + }, + getProvider() { + throw new Error('getProvider should not be called when perf disabled'); + }, + getModel() { + throw new Error('getModel should not be called when perf disabled'); + }, + }; +} + +function makeSettings(): LoadedSettings { + return new LoadedSettings( + { path: '', settings: {} }, + { path: '', settings: {} }, + { path: '', settings: {} }, + { path: '', settings: {} }, + true, + ); +} + +describe('buildAndStartPerfOwner — disabled path (Item 5)', () => { + it('returns null and reads only getTelemetrySettings', async () => { + const config = makeThrowingConfig(); + const agent = makeThrowingAgent(); + const settings = makeSettings(); + + const owner = await buildAndStartPerfOwner( + config, + agent, + settings, + '0.0.0-test', + ); + + // Returns null — disabled. + expect(owner).toBe(null); + + // Only getTelemetrySettings was called (exactly once). + expect(config.telemetryCallCount()).toBe(1); + }); + + it('installs no observers and allocates no timers', async () => { + const config = makeThrowingConfig(); + const agent = makeThrowingAgent(); + + await buildAndStartPerfOwner(config, agent, makeSettings(), 'test'); + + // No observers installed. + expect(getInteractiveStdoutObserver()).toBe(null); + expect(getInteractiveRenderObserver()).toBe(null); + expect(getPerfPhaseObserver()).toBe(null); + }); + + it('does not mutate process.stdout or process.stderr', async () => { + const config = makeThrowingConfig(); + const agent = makeThrowingAgent(); + + const stdoutWrite = process.stdout.write; + const stderrWrite = process.stderr.write; + + await buildAndStartPerfOwner(config, agent, makeSettings(), 'test'); + + // No mutation of stdout/stderr write methods. + expect(process.stdout.write).toBe(stdoutWrite); + expect(process.stderr.write).toBe(stderrWrite); + }); +}); + +describe('buildAndStartPerfOwner — platform includes arch (Item 6)', () => { + it('the resolved platform string includes process.platform-process.arch', async () => { + // Import the resolver directly to prove the platform format. + const { resolvePlatformArch } = await import( + '../ui/hooks/perf/interactivePerfRuntime.js' + ); + const platform = resolvePlatformArch(); + expect(platform).toBe(`${process.platform}-${process.arch}`); + }); +}); diff --git a/packages/cli/src/session/interactiveUI.startup.transaction.behavior.test.ts b/packages/cli/src/session/interactiveUI.startup.transaction.behavior.test.ts new file mode 100644 index 0000000000..29c08d290f --- /dev/null +++ b/packages/cli/src/session/interactiveUI.startup.transaction.behavior.test.ts @@ -0,0 +1,528 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Finding B behavioral tests — fully transactional interactive startup. + * + * Proves that every fallible stage after a perf owner successfully starts runs + * inside ONE transaction. Any failure preserves the primary error first, + * independently disposes the owner (clears observers, removes claim, clears + * timer), clears/unmounts any produced instance, disables staged mouse, and + * restores terminal protocols. Stages before mouse activation do NOT falsely + * disable unstaged mouse. Uses a REAL perf owner so observer/claim/timer + * cleanup is behavioral evidence (not mock-theater). + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import { promises as fsp } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import * as crypto from 'node:crypto'; +import type { Config } from '@vybestack/llxprt-code-core'; +import type { Agent } from '@vybestack/llxprt-code-agents'; +import type { LoadedSettings } from '../config/settings.js'; +import { + commitInteractiveStartup, + __resetInteractiveUIStateForTesting, + type InteractiveStartupPorts, +} from './interactiveUI.js'; +import { createInteractivePerfRuntime } from '../ui/hooks/perf/interactivePerfRuntime.js'; +import type { OperationIdentitySnapshot } from '../ui/hooks/agentStream/operationLifecycle.js'; +import { + setInteractiveStdoutObserver, + setInteractiveRenderObserver, + getInteractiveStdoutObserver, + getInteractiveRenderObserver, +} from '../ui/inkRenderOptions.js'; +import { + getPerfPhaseObserver, + setPerfPhaseObserver, +} from '@vybestack/llxprt-code-telemetry/perf/perfPhaseObserver.js'; +import type { + PerfScheduler, + PerfTimerHandle, +} from '@vybestack/llxprt-code-telemetry/perf/index.js'; + +let dir: string; + +function fixtureIdentity(): OperationIdentitySnapshot { + return { + session_id: 'sess-tx', + runtime_id: 'rt-tx', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'hash-tx', + llxprt_version: '0.11.0', + git_sha: 'tx1234', + runtime: 'bun-1.3.14', + platform: 'darwin-arm64', + provider: 'test-provider', + model: 'test-model', + terminal_cols: 80, + terminal_rows: 24, + render_mode: 'incremental', + }; +} + +class CountingScheduler implements PerfScheduler { + setIntervalCount = 0; + clearCount = 0; + + setInterval(_callback: () => Promise, _ms: number): PerfTimerHandle { + this.setIntervalCount++; + return { + unref: () => {}, + clear: () => { + this.clearCount++; + }, + }; + } +} + +const testConfig = { + getProjectRoot: () => '/test', + getDebugMode: () => false, +} as unknown as Config; + +const testAgent = {} as unknown as Agent; +const testSettings = {} as unknown as LoadedSettings; + +/** + * Base noop ports. Every port returns a minimal stub or is a no-op. Each test + * overrides exactly ONE port to throw, proving the failure at that stage is + * transactionally rolled back. + */ +function noopPorts(): InteractiveStartupPorts { + return { + renderOptions: (() => ({ + alternateBuffer: false, + incrementalRendering: false, + stdout: { columns: 80, rows: 24 }, + })) as never, + buildUiRuntime: (() => ({ + shell: { getTerminalBackground: () => '' }, + })) as never, + buildSlashRuntime: (() => ({})) as never, + debugAppend: () => {}, + setupTerminal: () => {}, + isMouseEnabled: () => false, + render: (() => ({ clear: () => {}, unmount: () => {} })) as never, + registerSync: () => {}, + setupLifecycle: async () => {}, + }; +} + +/** Creates and starts a real perf owner for the test. */ +async function makeStartedOwner(scheduler: CountingScheduler) { + const owner = createInteractivePerfRuntime({ + enabled: true, + memoryEnabled: false, + perfDir: dir, + runUuid: crypto.randomUUID(), + identityProvider: { snapshot: () => fixtureIdentity() }, + __schedulerForTesting: scheduler, + }); + expect(owner).not.toBe(null); + await owner!.start(); + return owner!; +} + +beforeEach(() => { + dir = fs.mkdtempSync(join(tmpdir(), 'perf-tx-startup-')); + setInteractiveStdoutObserver(null); + setInteractiveRenderObserver(null); + setPerfPhaseObserver(null); + __resetInteractiveUIStateForTesting(); +}); + +afterEach(async () => { + __resetInteractiveUIStateForTesting(); + setInteractiveStdoutObserver(null); + setInteractiveRenderObserver(null); + setPerfPhaseObserver(null); + try { + await fsp.rm(dir, { recursive: true, force: true }); + } catch { + // ignore + } +}); + +/** + * Asserts the perf owner was fully disposed after a failed startup: observers + * cleared, timer cleared, claim removed. + */ +function assertOwnerDisposed( + owner: { registry: unknown }, + scheduler: CountingScheduler, +) { + expect(getInteractiveStdoutObserver()).toBe(null); + expect(getInteractiveRenderObserver()).toBe(null); + expect(getPerfPhaseObserver()).toBe(null); + expect(scheduler.clearCount).toBe(1); + + const files = fs.existsSync(dir) ? fs.readdirSync(dir) : []; + expect(files.some((f) => f.endsWith('.claim'))).toBe(false); +} + +/** + * Extracts all error messages from an Error or AggregateError (flattened). + */ +function errorMessages(err: unknown): string[] { + if (err instanceof AggregateError) { + return (err.errors as unknown[]).flatMap((e) => errorMessages(e)); + } + if (err instanceof Error) return [err.message]; + return [String(err)]; +} + +describe('Finding B — transactional interactive startup', () => { + it('render-options failure: owner disposed, no mouse staged, primary error preserved', async () => { + const scheduler = new CountingScheduler(); + const owner = await makeStartedOwner(scheduler); + const ports = { + ...noopPorts(), + renderOptions: (() => { + throw new Error('render-options-boom'); + }) as never, + }; + + let caught: unknown = null; + try { + await commitInteractiveStartup({ + config: testConfig, + agent: testAgent, + settings: testSettings, + perfOwner: owner, + version: 'test', + startupWarnings: [], + ports, + }); + } catch (err) { + caught = err; + } + expect(caught).not.toBe(null); + expect( + errorMessages(caught).some((m) => m.includes('render-options-boom')), + ).toBe(true); + assertOwnerDisposed(owner, scheduler); + }); + + it('ui-runtime failure: owner disposed, no mouse staged', async () => { + const scheduler = new CountingScheduler(); + const owner = await makeStartedOwner(scheduler); + const ports = { + ...noopPorts(), + buildUiRuntime: (() => { + throw new Error('ui-runtime-boom'); + }) as never, + }; + + let caught: unknown = null; + try { + await commitInteractiveStartup({ + config: testConfig, + agent: testAgent, + settings: testSettings, + perfOwner: owner, + version: 'test', + startupWarnings: [], + ports, + }); + } catch (err) { + caught = err; + } + expect( + errorMessages(caught).some((m) => m.includes('ui-runtime-boom')), + ).toBe(true); + assertOwnerDisposed(owner, scheduler); + }); + + it('slash-runtime failure: owner disposed, no mouse staged', async () => { + const scheduler = new CountingScheduler(); + const owner = await makeStartedOwner(scheduler); + const ports = { + ...noopPorts(), + buildSlashRuntime: (() => { + throw new Error('slash-runtime-boom'); + }) as never, + }; + + let caught: unknown = null; + try { + await commitInteractiveStartup({ + config: testConfig, + agent: testAgent, + settings: testSettings, + perfOwner: owner, + version: 'test', + startupWarnings: [], + ports, + }); + } catch (err) { + caught = err; + } + expect( + errorMessages(caught).some((m) => m.includes('slash-runtime-boom')), + ).toBe(true); + assertOwnerDisposed(owner, scheduler); + }); + + it('debug-append failure: owner disposed, no mouse staged', async () => { + const scheduler = new CountingScheduler(); + const owner = await makeStartedOwner(scheduler); + const ports = { + ...noopPorts(), + debugAppend: () => { + throw new Error('debug-boom'); + }, + }; + + let caught: unknown = null; + try { + await commitInteractiveStartup({ + config: testConfig, + agent: testAgent, + settings: testSettings, + perfOwner: owner, + version: 'test', + startupWarnings: [], + ports, + }); + } catch (err) { + caught = err; + } + expect(errorMessages(caught).some((m) => m.includes('debug-boom'))).toBe( + true, + ); + assertOwnerDisposed(owner, scheduler); + }); + + it('terminal-setup failure: owner disposed, mouse correctly staged for rollback', async () => { + const scheduler = new CountingScheduler(); + const owner = await makeStartedOwner(scheduler); + const ports = { + ...noopPorts(), + // Mouse is enabled (isMouseEnabled returns true), then setupTerminal throws. + isMouseEnabled: () => true, + setupTerminal: () => { + throw new Error('terminal-setup-boom'); + }, + }; + + let caught: unknown = null; + try { + await commitInteractiveStartup({ + config: testConfig, + agent: testAgent, + settings: testSettings, + perfOwner: owner, + version: 'test', + startupWarnings: [], + ports, + }); + } catch (err) { + caught = err; + } + expect( + errorMessages(caught).some((m) => m.includes('terminal-setup-boom')), + ).toBe(true); + assertOwnerDisposed(owner, scheduler); + }); + + it('render failure: owner disposed, instance not produced, primary error preserved', async () => { + const scheduler = new CountingScheduler(); + const owner = await makeStartedOwner(scheduler); + const ports = { + ...noopPorts(), + render: (() => { + throw new Error('render-boom'); + }) as never, + }; + + let caught: unknown = null; + try { + await commitInteractiveStartup({ + config: testConfig, + agent: testAgent, + settings: testSettings, + perfOwner: owner, + version: 'test', + startupWarnings: [], + ports, + }); + } catch (err) { + caught = err; + } + expect(errorMessages(caught).some((m) => m.includes('render-boom'))).toBe( + true, + ); + assertOwnerDisposed(owner, scheduler); + }); + + it('sync-cleanup registration failure remains retryable on the next startup', async () => { + let registrationAttempts = 0; + const failingPorts = { + ...noopPorts(), + registerSync: () => { + registrationAttempts++; + throw new Error('register-sync-boom'); + }, + }; + + await expect( + commitInteractiveStartup({ + config: testConfig, + agent: testAgent, + settings: testSettings, + perfOwner: null, + version: 'test', + startupWarnings: [], + ports: failingPorts, + }), + ).rejects.toThrow('register-sync-boom'); + + const succeedingPorts = { + ...noopPorts(), + registerSync: () => { + registrationAttempts++; + }, + }; + await commitInteractiveStartup({ + config: testConfig, + agent: testAgent, + settings: testSettings, + perfOwner: null, + version: 'test', + startupWarnings: [], + ports: succeedingPorts, + }); + + expect(registrationAttempts).toBe(2); + }); + + it('setup failure: owner disposed, instance cleared, primary error preserved', async () => { + const scheduler = new CountingScheduler(); + const owner = await makeStartedOwner(scheduler); + + let setupCallCount = 0; + const fakeInstance = { clear: () => {}, unmount: () => {} }; + const ports = { + ...noopPorts(), + render: (() => fakeInstance) as never, + setupLifecycle: async () => { + setupCallCount++; + throw new Error('setup-boom'); + }, + }; + + let caught: unknown = null; + try { + await commitInteractiveStartup({ + config: testConfig, + agent: testAgent, + settings: testSettings, + perfOwner: owner, + version: 'test', + startupWarnings: [], + ports, + }); + } catch (err) { + caught = err; + } + expect(setupCallCount).toBe(1); + expect(errorMessages(caught).some((m) => m.includes('setup-boom'))).toBe( + true, + ); + assertOwnerDisposed(owner, scheduler); + }); + + it('primary-error ordering: primary error first, cleanup errors aggregate after', async () => { + const scheduler = new CountingScheduler(); + const owner = await makeStartedOwner(scheduler); + // Make the owner's dispose throw by corrupting it after start. + const originalDispose = owner.dispose.bind(owner); + owner.dispose = async () => { + throw new Error('dispose-boom'); + }; + + const ports = { + ...noopPorts(), + renderOptions: (() => { + throw new Error('primary-boom'); + }) as never, + }; + + let caught: unknown = null; + try { + await commitInteractiveStartup({ + config: testConfig, + agent: testAgent, + settings: testSettings, + perfOwner: owner, + version: 'test', + startupWarnings: [], + ports, + }); + } catch (err) { + caught = err; + } + try { + expect(caught).toBeInstanceOf(AggregateError); + const agg = caught as AggregateError; + const msgs = errorMessages(agg); + // Primary error is first. + expect(msgs[0]).toBe('primary-boom'); + // Cleanup error follows. + expect(msgs.some((m) => m.includes('dispose-boom'))).toBe(true); + } finally { + // Restore and dispose for test cleanup. The throwing override prevented + // disposed from being set, so call the real dispose to tear down the + // owner's registry observers, sink/retention claim, and timer. + owner.dispose = originalDispose; + await originalDispose(); + } + }); + + it('exactly-once cleanup: module refs cleared so global cleanup is a no-op', async () => { + const scheduler = new CountingScheduler(); + const owner = await makeStartedOwner(scheduler); + const fakeInstance = { clear: () => {}, unmount: () => {} }; + const ports = { + ...noopPorts(), + render: (() => fakeInstance) as never, + setupLifecycle: async () => { + throw new Error('setup-boom-for-once'); + }, + }; + + let caught: unknown = null; + try { + await commitInteractiveStartup({ + config: testConfig, + agent: testAgent, + settings: testSettings, + perfOwner: owner, + version: 'test', + startupWarnings: [], + ports, + }); + } catch (err) { + caught = err; + } + expect(caught).not.toBe(null); + assertOwnerDisposed(owner, scheduler); + + // The transactional catch atomically cleared module refs via + // captureAndClearTrackedInstanceAndOwner. Calling replacePreviousInstanceAndOwner + // now must be a no-op (nothing to dispose — refs already cleared). + // We verify this does NOT throw (no double-dispose of the already-disposed owner). + const { replacePreviousInstanceAndOwner } = await import( + './interactiveUI.js' + ); + await replacePreviousInstanceAndOwner(); + }); +}); diff --git a/packages/cli/src/session/interactiveUI.tsx b/packages/cli/src/session/interactiveUI.tsx index 43b6666824..0b293e9ef8 100644 --- a/packages/cli/src/session/interactiveUI.tsx +++ b/packages/cli/src/session/interactiveUI.tsx @@ -23,22 +23,44 @@ import { type IContent, type LockHandle, type MessageBus, + type TelemetrySettings, writeToStdout, } from '@vybestack/llxprt-code-core'; import { debugLogger } from '@vybestack/llxprt-code-telemetry'; import { getCliVersion } from '../utils/version.js'; -import { enableMouseEvents } from '../ui/utils/mouse.js'; +import { enableMouseEvents, disableMouseEvents } from '../ui/utils/mouse.js'; import { restoreTerminalProtocolsSync } from '../ui/utils/terminalProtocolCleanup.js'; import { checkForUpdates } from '../ui/utils/updateCheck.js'; import { handleAutoUpdate } from '../utils/handleAutoUpdate.js'; import { SettingsContext } from '../ui/contexts/SettingsContext.js'; import { inkRenderOptions } from '../ui/inkRenderOptions.js'; +import { + resolvePerfSettings, + getProjectHash, +} from '@vybestack/llxprt-code-core'; +import { + createInteractivePerfRuntime, + createIdentityProviderFromGetters, + resolveRenderMode, + resolveRuntimeVersion, + resolvePlatformArch, +} from '../ui/hooks/perf/interactivePerfRuntime.js'; +import type { InteractivePerfRuntime } from '../ui/hooks/perf/interactivePerfRuntime.js'; +import { getGitCommitInfo } from '../utils/gitCommitInfo.js'; import { isMouseEventsEnabled } from '../ui/mouseEventsEnabled.js'; import { computeTerminalTitle } from '../utils/windowTitle.js'; import { StreamingState } from '../ui/types.js'; import { registerCleanup, registerSyncCleanup } from '../utils/cleanup.js'; import { appendInteractiveUiDebug } from './debugLog.js'; import { mouseEventsExitHandler } from './terminalCleanup.js'; +import { + cleanupInstanceAndOwner, + rollbackInteractiveFailure, + type InteractiveInstanceCapability, + type InteractiveOwnerCapability, + type InteractiveMouseTeardown, + type InteractiveTerminalRestore, +} from './interactiveUiLifecycle.js'; import type { Agent } from '@vybestack/llxprt-code-agents'; import { buildUiRuntimeFromSource, @@ -56,7 +78,7 @@ import { * repeated calls simply update this reference — the single registered * callback always tears down whichever instance is current. */ -let latestInstance: ReturnType | undefined; +let latestInstance: InteractiveInstanceCapability | undefined; /** * Idempotent flag so the cleanup callback is registered at most once per @@ -93,6 +115,14 @@ function handleError(error: Error, errorInfo: ErrorInfo) { } } +/** + * Module-level reference to the latest interactive perf runtime owner. + * Mirrors the latestInstance pattern: startInteractiveUI may be called more + * than once in a long-lived process, so we track the latest owner and dispose + * it in the single registered cleanup callback. + */ +let latestPerfOwner: InteractiveOwnerCapability | null = null; + /** * Module-level guard ensuring the title-reset exit listener is registered at * most once per process. setWindowTitle is called on every interactive @@ -107,6 +137,7 @@ function resetTitleExitHandler() { export function __resetInteractiveUIStateForTesting() { latestInstance = undefined; + latestPerfOwner = null; cleanupRegistered = false; titleResetExitListenerRegistered = false; syncCleanupRegistered = false; @@ -115,6 +146,21 @@ export function __resetInteractiveUIStateForTesting() { process.off('exit', restoreTerminalProtocolsSync); } +/** + * Narrow test-only seam to install a tracked latest instance and perf owner so + * a behavior test can drive the ACTUAL exported + * {@link replacePreviousInstanceAndOwner} against real tracked state. Does not + * reset state without cleanup in production paths — only tests call this to + * stage ownership before invoking the real replacement routine. + */ +export function __setTrackedInstanceAndOwnerForTesting( + instance: InteractiveInstanceCapability | undefined, + owner: InteractiveOwnerCapability | null, +): void { + latestInstance = instance; + latestPerfOwner = owner; +} + export function setWindowTitle(title: string, settings: LoadedSettings) { if (settings.merged.ui.hideWindowTitle !== true) { // Initial state before React loop starts @@ -139,9 +185,169 @@ export function setWindowTitle(title: string, settings: LoadedSettings) { } /** - * @plan:PLAN-20260211-SESSIONRECORDING.P26 - * @pseudocode recording-integration.md lines 115-132 + * Narrow Pick-style config capability for {@link buildAndStartPerfOwner}. + * Exposes only the methods the composition reads so a Bun behavior test can + * call the real composition function with a minimal instrumented config. + */ +export interface PerfOwnerConfigCapability { + getTelemetrySettings(): TelemetrySettings; + getSessionId(): string; + getProjectRoot(): string; + getScreenReader(): boolean; +} + +/** + * Narrow Pick-style agent capability for {@link buildAndStartPerfOwner}. + * Exposes only the methods the composition reads so a Bun behavior test can + * instrument provider/model/runtimeId getters and prove they are read fresh + * at each operation boundary. + */ +export interface PerfOwnerAgentCapability { + getRuntimeId(): string; + getProvider(): string; + getModel(): string; +} + +/** + * P12: Constructs and starts the interactive perf runtime owner from real + * runtime/config/build APIs. Returns null when perf is disabled (before any + * construction — zero side effects). Extracted from startInteractiveUI to + * keep the composition function within the max-lines-per-function limit. + * + * Accepts a narrow Pick-style capability so a Bun behavior test can call the + * real composition function with perf disabled and instrument every + * identity/hash/provider/model/timing/memory/timer-related seam. + */ +export async function buildAndStartPerfOwner( + config: PerfOwnerConfigCapability, + agent: PerfOwnerAgentCapability, + settings: LoadedSettings, + version: string, +): Promise { + const perfSettings = resolvePerfSettings(config.getTelemetrySettings()); + if (!perfSettings.enabled) return null; + const owner = createInteractivePerfRuntime({ + enabled: true, + memoryEnabled: perfSettings.memory, + identityProvider: createIdentityProviderFromGetters( + { + sessionId: config.getSessionId(), + runtimeId: agent.getRuntimeId(), + projectHash: getProjectHash(config.getProjectRoot()), + cliVersion: version, + gitSha: getGitCommitInfo(), + runtime: resolveRuntimeVersion(), + platform: resolvePlatformArch(), + }, + { + provider: () => agent.getProvider(), + model: () => agent.getModel(), + terminalCols: () => extractTerminalCols(process.stdout), + terminalRows: () => extractTerminalRows(process.stdout), + renderMode: () => + resolveRenderMode( + config.getScreenReader(), + settings.merged.ui.useAlternateBuffer === true && + !config.getScreenReader(), + settings.merged.ui.useAlternateBuffer === true && + !config.getScreenReader() && + settings.merged.ui.incrementalRendering !== false, + ), + }, + ), + }); + if (owner === null) { + throw new Error( + 'buildAndStartPerfOwner: createInteractivePerfRuntime returned null ' + + 'despite enabled=true (impossible state)', + ); + } + await owner.start(); + return owner; +} + +/** + * Reads stdout columns safely. process.stdout.columns is typed as `number` + * but can be undefined when stdout is not a TTY; the optional-property + * parameter type avoids an unnecessary-condition lint without a type assertion. */ +function extractTerminalCols(stream: { columns?: number }): number { + return stream.columns ?? 0; +} + +function extractTerminalRows(stream: { rows?: number }): number { + return stream.rows ?? 0; +} + +/** + * Builds the root JSX element tree for Ink render. Extracted from + * startInteractiveUI to keep the composition function within the + * max-lines-per-function limit. + */ +function buildRenderElement( + uiRuntime: ReturnType, + slashCommandRuntime: ReturnType, + agent: Agent, + settings: LoadedSettings, + startupWarnings: string[], + version: string, + runtimeMessageBus: MessageBus | undefined, + recordingIntegration: RecordingIntegration | undefined, + resumedHistory: IContent[] | undefined, + initialRecordingService: SessionRecordingService | undefined, + initialLockHandle: LockHandle | null | undefined, + suppressStartupWelcome: boolean | undefined, + perfOwner: InteractivePerfRuntime | null, +): React.ReactElement { + return ( + + + + + + + + ); +} + +/** + * Enables mouse events and registers terminal-protocol exit handlers. + * Idempotent across repeated startInteractiveUI calls. Extracted to keep + * startInteractiveUI within the max-lines-per-function limit. + */ +function setupTerminalExitHandlers( + renderOptions: ReturnType, + settings: LoadedSettings, +): void { + const mouseEventsEnabled = isMouseEventsEnabled(renderOptions, settings); + if (mouseEventsEnabled) { + enableMouseEvents(); + // mouseEventsExitHandler is module-level; process.off+on keeps exactly one + // listener across repeated calls. process.on('exit') avoids deadlocking on + // waitUntilExit during runExitCleanup (fixes #959). + process.off('exit', mouseEventsExitHandler); + process.on('exit', mouseEventsExitHandler); + } + process.off('exit', restoreTerminalProtocolsSync); + process.on('exit', restoreTerminalProtocolsSync); +} + export async function startInteractiveUI( config: Config, agent: Agent, @@ -162,99 +368,236 @@ export async function startInteractiveUI( ); setWindowTitle(basename(workspaceRoot), settings); - const renderOptions = inkRenderOptions(config, settings); - const uiRuntime = buildUiRuntimeFromSource(config); - const slashCommandRuntime = buildSlashCommandRuntime(config); - appendInteractiveUiDebug( - `renderOptions alternateBuffer=${String(renderOptions.alternateBuffer)} incrementalRendering=${String(renderOptions.incrementalRendering)} stdoutColumns=${String(renderOptions.stdout?.columns)} stdoutRows=${String(renderOptions.stdout?.rows)}`, + // Deterministic pre-start replacement: tear down any previous instance and + // perf owner BEFORE constructing/starting a new owner. This prevents + // observer-conflict: a new owner's installObservers() must not collide with + // a previous owner's still-installed observers. + await replacePreviousInstanceAndOwner(); + + const perfOwner = await buildAndStartPerfOwner( + config, + agent, + settings, + version, ); - const mouseEventsEnabled = isMouseEventsEnabled(renderOptions, settings); - if (mouseEventsEnabled) { - enableMouseEvents(); - // Register the mouse-events teardown idempotently. startInteractiveUI may - // be called more than once in a long-lived process (e.g. tests); a bare - // process.on('exit', ...) with an inline arrow would accumulate a new - // listener (and a fresh closure) on every call. mouseEventsExitHandler is - // module-level, so process.off removes the exact prior registration (a - // no-op on the first call) and process.on re-adds the single listener. - // process.on('exit') is used instead of registerCleanup because - // registerCleanup includes instance.waitUntilExit() which would deadlock - // on quit. The 'exit' event fires synchronously during process.exit() - // (fixes #959). - process.off('exit', mouseEventsExitHandler); - process.on('exit', mouseEventsExitHandler); - } - // Register the exit listener idempotently: startInteractiveUI may be called - // more than once in a long-lived process (e.g. tests), and a bare - // process.on('exit', ...) would accumulate duplicate listeners that each - // re-run the (idempotent) terminal-protocol restoration. process.off first - // (a no-op when not yet registered) keeps registration to exactly one - // listener across calls while preserving the registerSyncCleanup path. - process.off('exit', restoreTerminalProtocolsSync); - process.on('exit', restoreTerminalProtocolsSync); + return commitInteractiveStartup({ + config, + agent, + settings, + perfOwner, + version, + startupWarnings, + runtimeMessageBus, + recordingIntegration, + resumedHistory, + initialRecordingService, + initialLockHandle, + suppressStartupWelcome, + }); +} + +/** + * Builds the mouse/terminal teardown capabilities for a rollback path. Mouse + * disable uses the raw disableMouseEvents (not the swallowing exit handler) so + * a failure surfaces rather than being silently swallowed. Shared by the + * render-failure and setup-failure rollback paths. + */ +function buildTerminalRollbackCapabilities(mouseEventsEnabled: boolean): { + mouse: InteractiveMouseTeardown | null; + restore: InteractiveTerminalRestore; +} { + const mouse: InteractiveMouseTeardown | null = mouseEventsEnabled + ? { + disable: disableMouseEvents, + removeListener: () => process.off('exit', mouseEventsExitHandler), + } + : null; + const restore: InteractiveTerminalRestore = { + restore: restoreTerminalProtocolsSync, + removeListener: () => process.off('exit', restoreTerminalProtocolsSync), + }; + return { mouse, restore }; +} + +/** + * Injectable startup stage ports. Each port defaults to the real production + * function; tests override individual stages to inject failures at meaningful + * pre-render boundaries without mock-theater reimplementation. Narrow + * package-private seam introduced for {@link commitInteractiveStartup}. + */ +export interface InteractiveStartupPorts { + readonly renderOptions: typeof inkRenderOptions; + readonly buildUiRuntime: typeof buildUiRuntimeFromSource; + readonly buildSlashRuntime: typeof buildSlashCommandRuntime; + readonly debugAppend: (line: string) => void; + readonly setupTerminal: typeof setupTerminalExitHandlers; + readonly isMouseEnabled: typeof isMouseEventsEnabled; + readonly render: ( + node: React.ReactElement, + options: ReturnType, + ) => InteractiveInstanceCapability; + readonly registerSync: typeof registerSyncCleanup; + readonly setupLifecycle: typeof setupInstanceLifecycle; +} + +/** + * Default production ports. `render` wraps the module-level `render` variable + * (which `__setRenderForTesting` can override) so both the test seam and the + * explicit port override work. + */ +const defaultStartupPorts: InteractiveStartupPorts = { + renderOptions: inkRenderOptions, + buildUiRuntime: buildUiRuntimeFromSource, + buildSlashRuntime: buildSlashCommandRuntime, + debugAppend: appendInteractiveUiDebug, + setupTerminal: setupTerminalExitHandlers, + isMouseEnabled: isMouseEventsEnabled, + render: (node, options) => render(node, options), + registerSync: registerSyncCleanup, + setupLifecycle: setupInstanceLifecycle, +}; + +/** + * Transaction state tracking staged resources for rollback. `mouseStaged` is + * false until {@link setupTerminalExitHandlers} runs and mouse is confirmed + * enabled; stages before mouse activation must NOT falsely disable unstaged + * mouse. `instance` is undefined until render succeeds. + */ +interface StartupTransactionState { + readonly owner: InteractiveOwnerCapability | null; + mouseStaged: boolean; + instance: InteractiveInstanceCapability | undefined; +} + +/** + * Arguments for {@link commitInteractiveStartup}. Every fallible stage after a + * perf owner successfully starts runs inside this one transaction. + */ +export interface CommitInteractiveStartupArgs { + readonly config: Config; + readonly agent: Agent; + readonly settings: LoadedSettings; + readonly perfOwner: InteractivePerfRuntime | null; + readonly version: string; + readonly startupWarnings: string[]; + readonly ports?: Partial; + readonly runtimeMessageBus?: MessageBus; + readonly recordingIntegration?: RecordingIntegration; + readonly resumedHistory?: IContent[]; + readonly initialRecordingService?: SessionRecordingService; + readonly initialLockHandle?: LockHandle | null | undefined; + readonly suppressStartupWelcome?: boolean; +} + +/** + * Runs every fallible stage after a perf owner successfully starts as ONE + * transaction: inkRenderOptions, buildUiRuntimeFromSource, + * buildSlashCommandRuntime, debug append, terminal/mouse staging, render, + * sync-cleanup registration, and setupInstanceLifecycle. + * + * On ANY failure the single transactional rollback: preserves the primary + * failure first, atomically clears tracked module refs (so no later global + * cleanup double-disposes), independently disposes the owner, clears/unmounts + * any produced instance, disables staged mouse state + removes the listener, + * and restores terminal protocols + removes the listener. Stages before mouse + * activation do NOT falsely disable unstaged mouse. Internal cleanup errors + * aggregate after the primary failure via {@link rollbackInteractiveFailure}. + * + * No nested/double rollback — one explicit try/catch with transaction state. + */ +export async function commitInteractiveStartup( + args: CommitInteractiveStartupArgs, +): Promise { + const ports: InteractiveStartupPorts = { + ...defaultStartupPorts, + ...args.ports, + }; + const state: StartupTransactionState = { + owner: args.perfOwner, + mouseStaged: false, + instance: undefined, + }; - let instance: ReturnType; try { - instance = render( - - - - - - - , + const renderOptions = ports.renderOptions(args.config, args.settings); + const uiRuntime = ports.buildUiRuntime(args.config); + const slashCommandRuntime = ports.buildSlashRuntime( + args.config, + args.perfOwner?.snapshotCapability ?? null, + ); + ports.debugAppend( + `renderOptions alternateBuffer=${String(renderOptions.alternateBuffer)} incrementalRendering=${String(renderOptions.incrementalRendering)} stdoutColumns=${String(renderOptions.stdout?.columns)} stdoutRows=${String(renderOptions.stdout?.rows)}`, + ); + // Compute mouseStaged BEFORE setupTerminal so a terminal-setup failure + // correctly rolls back staged mouse. Stages before this point (render- + // options, runtime, slash-runtime, debug) leave mouseStaged=false so + // unstaged mouse is NOT falsely disabled. + state.mouseStaged = ports.isMouseEnabled(renderOptions, args.settings); + ports.setupTerminal(renderOptions, args.settings); + + state.instance = ports.render( + buildRenderElement( + uiRuntime, + slashCommandRuntime, + args.agent, + args.settings, + args.startupWarnings, + args.version, + args.runtimeMessageBus, + args.recordingIntegration, + args.resumedHistory, + args.initialRecordingService, + args.initialLockHandle, + args.suppressStartupWelcome, + args.perfOwner, + ), renderOptions, ); - } catch (error) { - if (mouseEventsEnabled) { - mouseEventsExitHandler(); - process.off('exit', mouseEventsExitHandler); + + // Sync-cleanup registration (guarded against duplicate registration). + if (!syncCleanupRegistered) { + ports.registerSync(restoreTerminalProtocolsSync); + syncCleanupRegistered = true; } - restoreTerminalProtocolsSync(); - process.off('exit', restoreTerminalProtocolsSync); - throw error; - } - appendInteractiveUiDebug('render returned'); - - // Also register the synchronous restoration for the runExitCleanup() path - // (non-interactive sessions call runExitCleanup before process.exit, where - // process 'exit' listeners have not yet fired). registerSyncCleanup appends - // to a module-level array without dedup, so guard with syncCleanupRegistered - // to avoid accumulating duplicate entries across repeated startInteractiveUI - // calls (e.g. tests). restoreTerminalProtocolsSync is idempotent (guarded - // by isTTY + writes only disable sequences), so running it both here and via - // process.on('exit') is harmless. - if (!syncCleanupRegistered) { - syncCleanupRegistered = true; - registerSyncCleanup(restoreTerminalProtocolsSync); - } - setupInstanceLifecycle(instance, settings, { - projectRoot: config.getProjectRoot(), - debugMode: config.getDebugMode(), - }); + await ports.setupLifecycle( + state.instance, + args.settings, + { + projectRoot: args.config.getProjectRoot(), + debugMode: args.config.getDebugMode(), + }, + args.perfOwner, + ); + + return state.instance; + } catch (primaryError) { + // Single transactional rollback. Atomically clear tracked module refs so + // the registered global cleanup cannot double-dispose the same + // instance/owner. Then tear down staged instance/owner/mouse/terminal + // independently. The primary failure is preserved first; every cleanup + // error aggregates behind it. Stages before mouse activation leave + // mouseStaged=false so unstaged mouse is NOT falsely disabled. + captureAndClearTrackedInstanceAndOwner(); + const { mouse, restore } = buildTerminalRollbackCapabilities( + state.mouseStaged, + ); + return rollbackInteractiveFailure(primaryError, { + instance: state.instance, + owner: state.owner, + mouse, + restore, + }); + } } -function setupInstanceLifecycle( - instance: ReturnType, +async function setupInstanceLifecycle( + instance: InteractiveInstanceCapability, settings: LoadedSettings, runtimeScalars: { projectRoot: string; debugMode: boolean }, -): void { + perfOwner: InteractivePerfRuntime | null, +): Promise { checkForUpdates(settings) .then((info) => { handleAutoUpdate(info, settings, runtimeScalars.projectRoot); @@ -266,23 +609,59 @@ function setupInstanceLifecycle( } }); - // Track the latest instance so the single registered cleanup callback tears - // down whichever instance is current (not a stale closure from an earlier - // call). The callback is registered at most once per process. + // Prior instance/owner cleanup has ALREADY been done by + // replacePreviousInstanceAndOwner() before buildAndStartPerfOwner(). Here + // we only track the new instance and register the cleanup callback. + latestInstance = instance; + latestPerfOwner = perfOwner; if (!cleanupRegistered) { cleanupRegistered = true; + // Registered global cleanup uses the SAME capture+clear+dispose helper as + // replacePreviousInstanceAndOwner: capture and clear module refs BEFORE + // disposal so exactly-once is guaranteed even when disposal throws. If a + // replacement already captured+cleared, this callback finds empty slots + // and is a no-op; if this callback runs first, the replacement is a no-op. registerCleanup(async () => { - const current = latestInstance; - if (!current) { - return; - } - // Unmount immediately rather than awaiting waitUntilExit(). During - // shutdown (e.g. runExitCleanup from the non-interactive path), the Ink - // instance may never naturally exit, and awaiting it would deadlock - // runExitCleanup indefinitely, blocking process.exit. - current.clear(); - current.unmount(); + const { instance, owner } = captureAndClearTrackedInstanceAndOwner(); + await cleanupInstanceAndOwner(instance, owner); }); } } + +/** + * Atomically captures the currently tracked instance/owner and clears the + * module-level references. Used by the registered global cleanup, + * {@link replacePreviousInstanceAndOwner}, and the setup-failure transactional + * catch so all three paths share one exactly-once disposal point: the + * capture+clear happens before any disposal attempt, so even when disposal + * throws the slots are already empty and no second path can re-dispose the + * same instance/owner. + */ +function captureAndClearTrackedInstanceAndOwner(): { + instance: InteractiveInstanceCapability | undefined; + owner: InteractiveOwnerCapability | null; +} { + const instance = latestInstance; + const owner = latestPerfOwner; + latestInstance = undefined; + latestPerfOwner = null; + return { instance, owner }; +} + +/** + * Deterministic pre-start replacement: tears down any previous interactive + * instance and perf owner BEFORE a new owner is constructed and started. This + * prevents observer-conflict: a new owner's installObservers() must not + * collide with a previous owner's still-installed observers. + * + * Delegates to {@link captureAndClearTrackedInstanceAndOwner} + + * {@link cleanupInstanceAndOwner} so clear, unmount, and dispose run + * independently and internal errors surface as one Error or AggregateError. + * Tracking is cleared BEFORE cleanup so the slots are reclaimable even when + * cleanup throws. + */ +export async function replacePreviousInstanceAndOwner(): Promise { + const { instance, owner } = captureAndClearTrackedInstanceAndOwner(); + await cleanupInstanceAndOwner(instance, owner); +} diff --git a/packages/cli/src/session/interactiveUiLifecycle.ts b/packages/cli/src/session/interactiveUiLifecycle.ts new file mode 100644 index 0000000000..67f5b932e8 --- /dev/null +++ b/packages/cli/src/session/interactiveUiLifecycle.ts @@ -0,0 +1,176 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Production lifecycle helpers for the interactive UI instance and perf owner. + * + * Centralizes the clear/unmount/dispose teardown and the render/setup-failure + * rollback so {@link interactiveUI.tsx} and its behavior tests exercise ONE + * shared routine rather than mirrored copies. Every cleanup step runs + * independently — a throw in one step NEVER prevents a later step from running. + * Internal errors are collected and surfaced (never swallowed); when a primary + * failure exists it is preserved first and every cleanup error aggregated + * after it. + */ + +/** + * Structural capability for a rendered Ink instance. Ink's `render()` return + * value satisfies this structurally (`{ clear, unmount }`). + */ +export interface InteractiveInstanceCapability { + clear(): void; + unmount(): void; +} + +/** + * Structural capability for the interactive perf owner. The + * `InteractivePerfRuntime` satisfies this structurally (`{ dispose }`). + */ +export interface InteractiveOwnerCapability { + dispose(): Promise; +} + +/** + * Shared instance + owner teardown used by both public cleanup functions. + * Runs `instance.clear()`, `instance.unmount()`, and `owner.dispose()` in + * order, each independently of the others — a throw in one step NEVER prevents + * a later step from running. Every error is pushed into `errors` (never + * swallowed) so the caller can aggregate them identically. + */ +async function teardownInstanceAndOwner( + instance: InteractiveInstanceCapability | undefined, + owner: InteractiveOwnerCapability | null, + errors: unknown[], +): Promise { + if (instance !== undefined) { + try { + instance.clear(); + } catch (err) { + errors.push(err); + } + try { + instance.unmount(); + } catch (err) { + errors.push(err); + } + } + if (owner !== null) { + try { + await owner.dispose(); + } catch (err) { + errors.push(err); + } + } +} + +/** + * Runs `instance.clear()`, `instance.unmount()`, and `owner.dispose()` in + * order, each independently of the others. A throw in one step NEVER prevents + * a later step from running. Collects every error and throws a single `Error` + * (one error) or `AggregateError` (many); resolves cleanly when none throw. + * + * Shared by the pre-start previous-instance/owner replacement and the + * registered global cleanup so both paths behave identically. + */ +export async function cleanupInstanceAndOwner( + instance: InteractiveInstanceCapability | undefined, + owner: InteractiveOwnerCapability | null, +): Promise { + const errors: unknown[] = []; + await teardownInstanceAndOwner(instance, owner, errors); + if (errors.length === 1) { + throw errors[0]; + } + if (errors.length > 1) { + throw new AggregateError( + errors, + 'interactive instance/owner cleanup failed', + ); + } +} + +/** + * Mouse-events teardown capability for the rollback path. Holds the RAW + * disable function (not the swallowing exit handler) so failures surface + * rather than being silently swallowed, plus the listener-removal callback. + */ +export interface InteractiveMouseTeardown { + readonly disable: () => void; + readonly removeListener: () => void; +} + +/** + * Terminal protocol restore capability for the rollback path. Holds the + * restore function plus the listener-removal callback. + */ +export interface InteractiveTerminalRestore { + readonly restore: () => void; + readonly removeListener: () => void; +} + +/** + * Capabilities the rollback helper tears down. `instance` is the rendered Ink + * instance (present on setup failure; `undefined` on render failure where no + * instance was produced). `mouse` is `null` when mouse events were not + * enabled. `owner` is `null` when perf is disabled. + */ +export interface InteractiveRollbackCapabilities { + readonly instance: InteractiveInstanceCapability | undefined; + readonly owner: InteractiveOwnerCapability | null; + readonly mouse: InteractiveMouseTeardown | null; + readonly restore: InteractiveTerminalRestore; +} + +/** + * Interactive-failure rollback used by BOTH the render-failure path and the + * post-render setup-failure transactional catch. Attempts, each independently: + * instance.clear + instance.unmount (when an instance exists), owner.dispose, + * mouse disable + mouse listener removal (when enabled), and terminal restore + * + restore listener removal. The `primaryError` is always preserved first; + * every cleanup error is aggregated after it via `AggregateError` (or the + * primary error is thrown alone when no cleanup step failed). Never swallows + * internal errors. + */ +export async function rollbackInteractiveFailure( + primaryError: unknown, + capabilities: InteractiveRollbackCapabilities, +): Promise { + const errors: unknown[] = []; + await teardownInstanceAndOwner( + capabilities.instance, + capabilities.owner, + errors, + ); + if (capabilities.mouse !== null) { + try { + capabilities.mouse.disable(); + } catch (err) { + errors.push(err); + } + try { + capabilities.mouse.removeListener(); + } catch (err) { + errors.push(err); + } + } + try { + capabilities.restore.restore(); + } catch (err) { + errors.push(err); + } + try { + capabilities.restore.removeListener(); + } catch (err) { + errors.push(err); + } + if (errors.length === 0) { + throw primaryError; + } + throw new AggregateError( + [primaryError, ...errors], + 'interactive failure and one or more cleanup steps also failed', + ); +} diff --git a/packages/cli/src/ui/App.tsx b/packages/cli/src/ui/App.tsx index d3ebfb2020..6b149e7eff 100644 --- a/packages/cli/src/ui/App.tsx +++ b/packages/cli/src/ui/App.tsx @@ -15,6 +15,8 @@ import type { import type { SlashCommandRuntime, UiRuntime } from './cliUiRuntime.js'; import type { Agent } from '@vybestack/llxprt-code-agents'; import type { LoadedSettings } from '../config/settings.js'; +import type { OperationLifecycleRegistry } from './hooks/agentStream/operationLifecycle.js'; +import type { MemoryTelemetryController } from './hooks/memoryTrend/memoryTelemetry.js'; import { KeypressProvider } from './contexts/KeypressContext.js'; import { MouseProvider } from './contexts/MouseContext.js'; import { SessionStatsProvider } from './contexts/SessionContext.js'; @@ -51,6 +53,10 @@ interface AppProps { /** @plan:PLAN-20260214-SESSIONBROWSER.P23 */ initialLockHandle?: LockHandle | null; suppressStartupWelcome?: boolean; + /** P12: optional perf operation lifecycle registry (perf enabled only). */ + operationLifecycle?: OperationLifecycleRegistry; + /** P12: optional memory telemetry controller (perf+memory enabled only). */ + memoryController?: MemoryTelemetryController; } /** diff --git a/packages/cli/src/ui/AppContainerRuntime.tsx b/packages/cli/src/ui/AppContainerRuntime.tsx index 63e2d790c0..3e6dfec8df 100644 --- a/packages/cli/src/ui/AppContainerRuntime.tsx +++ b/packages/cli/src/ui/AppContainerRuntime.tsx @@ -17,6 +17,8 @@ import type { SlashCommandRuntime, UiRuntime } from './cliUiRuntime.js'; import type { Agent } from '@vybestack/llxprt-code-agents'; import type { LoadedSettings } from '../config/settings.js'; import type { AppState, AppAction } from './reducers/appReducer.js'; +import type { OperationLifecycleRegistry } from './hooks/agentStream/operationLifecycle.js'; +import type { MemoryTelemetryController } from './hooks/memoryTrend/memoryTelemetry.js'; import { UIStateProvider } from './contexts/UIStateContext.js'; import { UIActionsProvider } from './contexts/UIActionsContext.js'; import { DefaultAppLayout } from './layouts/DefaultAppLayout.js'; @@ -62,6 +64,10 @@ export interface AppContainerRuntimeProps { /** @plan:PLAN-20260214-SESSIONBROWSER.P23 */ initialLockHandle?: LockHandle | null; suppressStartupWelcome?: boolean; + /** P12: optional perf operation lifecycle registry (perf enabled only). */ + operationLifecycle?: OperationLifecycleRegistry; + /** P12: optional memory telemetry controller (perf+memory enabled only). */ + memoryController?: MemoryTelemetryController; } type HookResults = { @@ -456,15 +462,16 @@ export const AppContainerRuntime = (props: AppContainerRuntimeProps) => { setLlxprtMdFileCount: bootstrap.setLlxprtMdFileCount, suppressStartupWelcome: props.suppressStartupWelcome, }); - const input = useAppInput( - buildInputParams( + const input = useAppInput({ + ...buildInputParams( bootstrap, dialogs, props.appState, props.appDispatch, props.slashCommandRuntime, ), - ); + operationLifecycle: props.operationLifecycle, + }); const layout = useAppLayout(buildLayoutParams(bootstrap, dialogs, input)); useUnconfiguredProviderGuidance({ hasActiveProvider: diff --git a/packages/cli/src/ui/cliUiRuntime.ts b/packages/cli/src/ui/cliUiRuntime.ts index 67774e71fa..70fc307d12 100644 --- a/packages/cli/src/ui/cliUiRuntime.ts +++ b/packages/cli/src/ui/cliUiRuntime.ts @@ -48,6 +48,7 @@ import type { import type { GitHubBrokerClient } from '@vybestack/llxprt-code-tools'; import type { EventEmitter } from 'node:events'; import { AppEvent, appEvents, type AppEvents } from '../utils/events.js'; +import type { PerfSnapshotCapability } from './commands/perfCommand.js'; export interface RefreshMemoryResult { memoryContent: string; @@ -432,6 +433,13 @@ export interface AppStateRuntime { */ getGitHubBrokerClient(): GitHubBrokerClient | undefined; updateSystemInstructionIfInitialized(): void | Promise; + /** + * Returns the owned perf snapshot capability for the bare `/perf` live view, + * or null when perf telemetry is not active. P12 wires this from the + * interactive perf runtime owner. Optional: callers without a perf owner + * (disabled/default-off) return null or omit this entirely. + */ + getPerfSnapshotCapability?(): PerfSnapshotCapability | null; } /** @@ -798,6 +806,7 @@ export type SlashCommandRuntime = CliUiRuntime; */ export function buildSlashCommandRuntime( source: UiRuntimeBareSource, + perfSnapshotCapability?: PerfSnapshotCapability | null, ): CliUiRuntime { // Non-slice members must be destructured out and re-attached explicitly. // The spread below only flattens capability SLICE OBJECTS; a member whose @@ -813,6 +822,9 @@ export function buildSlashCommandRuntime( ...Object.values(capabilities), { storage }, getRunImageOperation !== undefined ? { getRunImageOperation } : {}, + perfSnapshotCapability !== undefined && perfSnapshotCapability !== null + ? { getPerfSnapshotCapability: () => perfSnapshotCapability } + : {}, ); } diff --git a/packages/cli/src/ui/commands/perfCommand.behavior.test.ts b/packages/cli/src/ui/commands/perfCommand.behavior.test.ts new file mode 100644 index 0000000000..085e24cbf8 --- /dev/null +++ b/packages/cli/src/ui/commands/perfCommand.behavior.test.ts @@ -0,0 +1,565 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import { promises as fs } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { + createPerfCommand, + type PerfSnapshotCapability, + type PerfOperations, +} from './perfCommand.js'; +import type { MessageActionReturn } from './types.js'; +import { BuiltinCommandLoader } from '../../services/BuiltinCommandLoader.js'; +import type { PerfOperationRecord } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; + +function makeOperation( + overrides: Partial = {}, +): PerfOperationRecord { + return { + schema_version: 1, + record_type: 'operation', + ts: '2026-01-01T00:00:00.000Z', + session_id: 'sess-1', + operation_id: 'op-1', + runtime_id: 'rt-1', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'hash-1', + llxprt_version: '0.10.0', + git_sha: 'abc1234', + runtime: 'cli', + platform: 'darwin', + provider: 'test-provider', + model: 'test-model', + context_tokens: 1000, + output_tokens: 500, + terminal_cols: 80, + terminal_rows: 24, + render_mode: 'ink', + concurrent_instances: 1, + status: 'completed', + client_prepare_ms: 10, + stream_handler_ms: 100, + ink_render_ms: 5, + ink_render_count: 3, + stdout_bytes: 1024, + stdout_write_calls: 5, + stdout_write_sync_ms: 2, + client_finalize_ms: 8, + provider_attempts: 1, + provider_attempt_sum_ms: 200, + provider_union_ms: 200, + tool_calls: 2, + tool_call_sum_ms: 50, + tool_union_ms: 50, + agent_activity_union_ms: 250, + operation_elapsed_ms: 1000, + approval_wait_ms: 0, + unclassified_elapsed_ms: 0, + session_operation_index: 0, + uptime_ms: 5000, + ...overrides, + }; +} + +async function makeTempDir(): Promise { + const dir = join( + tmpdir(), + `perf-cmd-${Date.now()}-${Math.random().toString(36).slice(2)}`, + ); + await fs.mkdir(dir, { recursive: true }); + return dir; +} + +async function writeJsonl( + dir: string, + name: string, + lines: string[], +): Promise { + const content = lines.join('\n'); + await fs.writeFile( + join(dir, name), + content + (content.endsWith('\n') ? '' : '\n'), + 'utf8', + ); +} + +function isMessage(result: unknown): result is MessageActionReturn { + return ( + typeof result === 'object' && + result !== null && + 'type' in result && + (result as { type: string }).type === 'message' + ); +} + +describe('PerfCommand (P11, AC-9)', () => { + let dir: string; + + beforeEach(async () => { + dir = await makeTempDir(); + }); + + afterEach(async () => { + await fs.rm(dir, { recursive: true, force: true }); + }); + + // --- /perf inspect --- + + it('/perf inspect returns directory info and counts', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + ]); + + const cmd = createPerfCommand({ perfDir: dir }); + const inspectSub = cmd.subCommands!.find((s) => s.name === 'inspect')!; + const result = await inspectSub.action!({} as never, ''); + + expect(isMessage(result)).toBe(true); + const msg = result as MessageActionReturn; + expect(msg.messageType).toBe('info'); + expect(msg.content).toContain('Perf Inspect'); + expect(msg.content).toContain(dir); + expect(msg.content).toContain('operations: 1'); + }); + + it('/perf inspect on empty directory shows zero counts', async () => { + const cmd = createPerfCommand({ perfDir: dir }); + const inspectSub = cmd.subCommands!.find((s) => s.name === 'inspect')!; + const result = await inspectSub.action!({} as never, ''); + + expect(isMessage(result)).toBe(true); + const msg = result as MessageActionReturn; + expect(msg.content).toContain('Owned JSONL files: 0'); + expect(msg.content).toContain('operations: 0'); + }); + + // --- /perf report --- + + it('/perf report returns longitudinal report', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation({ operation_elapsed_ms: 1000 })), + ]); + + const cmd = createPerfCommand({ perfDir: dir }); + const reportSub = cmd.subCommands!.find((s) => s.name === 'report')!; + const result = await reportSub.action!({} as never, ''); + + expect(isMessage(result)).toBe(true); + const msg = result as MessageActionReturn; + expect(msg.content).toContain('Perf Report'); + expect(msg.content).toContain('operation_elapsed_ms'); + expect(msg.content).toContain('p50=1000'); + }); + + it('/perf report without baseline has no delta text', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + ]); + + const cmd = createPerfCommand({ perfDir: dir }); + const reportSub = cmd.subCommands!.find((s) => s.name === 'report')!; + const result = await reportSub.action!({} as never, ''); + + const msg = result as MessageActionReturn; + expect(msg.content).not.toContain('Baseline:'); + }); + + it('/perf report --baseline VERSION parses and shows deltas', async () => { + await writeJsonl(dir, 'perf-20260101-base.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.10.0', + git_sha: 'aaa', + operation_elapsed_ms: 1000, + }), + ), + ]); + await writeJsonl(dir, 'perf-20260102-new.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.11.0', + git_sha: 'bbb', + operation_elapsed_ms: 2000, + }), + ), + ]); + + const cmd = createPerfCommand({ perfDir: dir }); + const reportSub = cmd.subCommands!.find((s) => s.name === 'report')!; + const result = await reportSub.action!({} as never, '--baseline 0.10.0'); + + const msg = result as MessageActionReturn; + expect(msg.messageType).toBe('info'); + expect(msg.content).toContain('Baseline: 0.10.0 (matched)'); + expect(msg.content).toContain('delta'); + }); + + it('/perf report --baseline SHA parses correctly', async () => { + await writeJsonl(dir, 'perf-20260101-base.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.10.0', + git_sha: 'abc1234', + operation_elapsed_ms: 1000, + }), + ), + ]); + + const cmd = createPerfCommand({ perfDir: dir }); + const reportSub = cmd.subCommands!.find((s) => s.name === 'report')!; + const result = await reportSub.action!({} as never, '--baseline abc1234'); + + const msg = result as MessageActionReturn; + expect(msg.content).toContain('Baseline: abc1234 (matched)'); + }); + + it('/perf report --baseline (no value) rejects with useful error', async () => { + const cmd = createPerfCommand({ perfDir: dir }); + const reportSub = cmd.subCommands!.find((s) => s.name === 'report')!; + const result = await reportSub.action!({} as never, '--baseline'); + + const msg = result as MessageActionReturn; + expect(msg.messageType).toBe('error'); + expect(msg.content).toContain('--baseline requires a value'); + }); + + it('/perf report --baseline --other rejects malformed', async () => { + const cmd = createPerfCommand({ perfDir: dir }); + const reportSub = cmd.subCommands!.find((s) => s.name === 'report')!; + const result = await reportSub.action!({} as never, '--baseline --other'); + + const msg = result as MessageActionReturn; + expect(msg.messageType).toBe('error'); + expect(msg.content).toContain('--baseline requires a value'); + }); + + it('/perf report with unexpected argument rejects', async () => { + const cmd = createPerfCommand({ perfDir: dir }); + const reportSub = cmd.subCommands!.find((s) => s.name === 'report')!; + const result = await reportSub.action!({} as never, 'unexpected-arg'); + + const msg = result as MessageActionReturn; + expect(msg.messageType).toBe('error'); + expect(msg.content).toContain("unexpected argument 'unexpected-arg'"); + }); + + // --- /perf delete --- + + it('/perf delete removes stale perf files', async () => { + // Old file (eligible for deletion) + await writeJsonl(dir, 'perf-20250101-old.jsonl', [ + JSON.stringify(makeOperation()), + ]); + // Stale claim (set mtime well in the past, past the 180s lease) + await fs.writeFile(join(dir, 'stale.claim'), ''); + const staleTime = new Date(Date.now() - 300_000); // 5 min ago + await fs.utimes(join(dir, 'stale.claim'), staleTime, staleTime); + + const cmd = createPerfCommand({ perfDir: dir }); + const deleteSub = cmd.subCommands!.find((s) => s.name === 'delete')!; + const result = await deleteSub.action!({} as never, ''); + + const msg = result as MessageActionReturn; + expect(msg.messageType).toBe('info'); + expect(msg.content).toContain('Perf Delete'); + expect(msg.content).toContain('Deleted: 2 file(s)'); + + // Verify files are gone + const remaining = await fs.readdir(dir); + expect(remaining).toHaveLength(0); + }); + + it('/perf delete does not delete active writer files', async () => { + const dayKey = (() => { + const d = new Date(); + const y = d.getUTCFullYear(); + const m = String(d.getUTCMonth() + 1).padStart(2, '0'); + const day = String(d.getUTCDate()).padStart(2, '0'); + return `${y}${m}${day}`; + })(); + + // Today's file with recent mtime (active writer) + await writeJsonl(dir, `perf-${dayKey}-active.jsonl`, [ + JSON.stringify(makeOperation()), + ]); + + const cmd = createPerfCommand({ perfDir: dir }); + const deleteSub = cmd.subCommands!.find((s) => s.name === 'delete')!; + const result = await deleteSub.action!({} as never, ''); + + const msg = result as MessageActionReturn; + expect(msg.content).toContain('Protected (live):'); + + // File still exists + const remaining = await fs.readdir(dir); + expect(remaining).toContain(`perf-${dayKey}-active.jsonl`); + }); + + // --- /perf (no args) --- + + it('/perf with no snapshot capability says unavailable honestly', async () => { + const cmd = createPerfCommand({ perfDir: dir }); + const result = await cmd.action!({} as never, ''); + + expect(isMessage(result)).toBe(true); + const msg = result as MessageActionReturn; + expect(msg.messageType).toBe('info'); + expect(msg.content).toContain('not active'); + }); + + it('/perf with snapshot capability shows current process snapshot', async () => { + const capability: PerfSnapshotCapability = { + getMemorySnapshot: () => [ + { + rss: 50_000_000, + heapUsed: 20_000_000, + external: 5_000_000, + arrayBuffers: 1_000_000, + uptimeMs: 60_000, + msSinceLastOperation: 5_000, + timestampMs: Date.now(), + }, + ], + getActiveOperationSummary: () => ({ + provider: 'test-provider', + model: 'test-model', + elapsedMs: 12_000, + }), + getSelfHealth: () => ({ lastWriteErrorCode: null, evictionCount: 0 }), + }; + + const cmd = createPerfCommand({ + perfDir: dir, + snapshotCapability: capability, + }); + const result = await cmd.action!({} as never, ''); + + expect(isMessage(result)).toBe(true); + const msg = result as MessageActionReturn; + expect(msg.messageType).toBe('info'); + expect(msg.content).toContain('Perf Snapshot'); + expect(msg.content).toContain('Active operation'); + expect(msg.content).toContain('Memory samples: 1'); + }); + + it('/perf trend scales a large negative RSS delta by magnitude (not raw bytes)', async () => { + // Two samples with a large negative RSS delta (~-10 MiB). The old + // formatBytes treated negatives as < 1024 (raw B); the fix scales the + // absolute value while preserving the sign. + const firstRss = 50_000_000; + const secondRss = 39_510_016; // delta ≈ -10_489_984 bytes ≈ -10.0 MiB + const firstUptime = 10_000; + const secondUptime = 70_000; + const capability: PerfSnapshotCapability = { + getMemorySnapshot: () => [ + { + rss: firstRss, + heapUsed: 20_000_000, + external: 5_000_000, + arrayBuffers: 1_000_000, + uptimeMs: firstUptime, + msSinceLastOperation: 5_000, + timestampMs: Date.now(), + }, + { + rss: secondRss, + heapUsed: 18_000_000, + external: 4_000_000, + arrayBuffers: 900_000, + uptimeMs: secondUptime, + msSinceLastOperation: 2_000, + timestampMs: Date.now(), + }, + ], + getActiveOperationSummary: () => null, + getSelfHealth: () => ({ lastWriteErrorCode: null, evictionCount: 0 }), + }; + + const cmd = createPerfCommand({ + perfDir: dir, + snapshotCapability: capability, + }); + const result = await cmd.action!({} as never, ''); + const msg = result as MessageActionReturn; + + // The trend line must contain the scaled unit (MiB) with a leading minus, + // not the raw byte count. + expect(msg.content).toContain('trend:'); + expect(msg.content).toContain('-10.0 MiB'); + expect(msg.content).not.toContain('-10489984 B'); + }); + it('/perf with null snapshot capability says unavailable', async () => { + const cmd = createPerfCommand({ perfDir: dir, snapshotCapability: null }); + const result = await cmd.action!({} as never, ''); + + const msg = result as MessageActionReturn; + expect(msg.content).toContain('not active'); + }); + + it('/perf with unknown subcommand returns error', async () => { + const cmd = createPerfCommand({ perfDir: dir }); + const result = await cmd.action!({} as never, 'unknownsub'); + + const msg = result as MessageActionReturn; + expect(msg.messageType).toBe('error'); + expect(msg.content).toContain('Unknown subcommand'); + }); + + // --- P11: errno I/O failures → user error; non-errno → reject --- + + function errnoOperations( + method: 'inspect' | 'report' | 'delete', + err: Error & { code?: string }, + ): PerfOperations { + const thrower = async (): Promise => { + throw err; + }; + const noop = async (): Promise => { + throw new Error('should not be called'); + }; + return { + inspect: method === 'inspect' ? thrower : (noop as never), + report: method === 'report' ? thrower : (noop as never), + delete: method === 'delete' ? thrower : (noop as never), + }; + } + + it('/perf inspect converts an errno I/O failure to a user error message', async () => { + const err = new Error('permission denied') as NodeJS.ErrnoException; + err.code = 'EACCES'; + const cmd = createPerfCommand({ + perfDir: dir, + operations: errnoOperations('inspect', err), + }); + const inspectSub = cmd.subCommands!.find((s) => s.name === 'inspect')!; + const result = await inspectSub.action!({} as never, ''); + + const msg = result as MessageActionReturn; + expect(msg.messageType).toBe('error'); + expect(msg.content).toContain('Failed to inspect perf data'); + expect(msg.content).toContain('permission denied'); + }); + + it('/perf report converts an errno I/O failure to a user error message', async () => { + const err = new Error('read-only') as NodeJS.ErrnoException; + err.code = 'EROFS'; + const cmd = createPerfCommand({ + perfDir: dir, + operations: errnoOperations('report', err), + }); + const reportSub = cmd.subCommands!.find((s) => s.name === 'report')!; + const result = await reportSub.action!({} as never, ''); + + const msg = result as MessageActionReturn; + expect(msg.messageType).toBe('error'); + expect(msg.content).toContain('Failed to generate perf report'); + }); + + it('/perf delete converts an errno I/O failure to a user error message', async () => { + const err = new Error('no space') as NodeJS.ErrnoException; + err.code = 'ENOSPC'; + const cmd = createPerfCommand({ + perfDir: dir, + operations: errnoOperations('delete', err), + }); + const deleteSub = cmd.subCommands!.find((s) => s.name === 'delete')!; + const result = await deleteSub.action!({} as never, ''); + + const msg = result as MessageActionReturn; + expect(msg.messageType).toBe('error'); + expect(msg.content).toContain('Failed to delete perf data'); + }); + + it('/perf inspect allows a non-errno internal error to reject', async () => { + const cmd = createPerfCommand({ + perfDir: dir, + operations: errnoOperations('inspect', new TypeError('internal bug')), + }); + const inspectSub = cmd.subCommands!.find((s) => s.name === 'inspect')!; + await expect(inspectSub.action!({} as never, '')).rejects.toThrow( + TypeError, + ); + }); + + it('/perf report allows a non-errno internal error to reject', async () => { + const cmd = createPerfCommand({ + perfDir: dir, + operations: errnoOperations('report', new RangeError('internal bug')), + }); + const reportSub = cmd.subCommands!.find((s) => s.name === 'report')!; + await expect(reportSub.action!({} as never, '')).rejects.toThrow( + RangeError, + ); + }); + + it('/perf delete allows a non-errno internal error to reject', async () => { + const cmd = createPerfCommand({ + perfDir: dir, + operations: errnoOperations('delete', new TypeError('internal bug')), + }); + const deleteSub = cmd.subCommands!.find((s) => s.name === 'delete')!; + await expect(deleteSub.action!({} as never, '')).rejects.toThrow(TypeError); + }); + + // --- Loader registration --- + + it('perf command is registered in BuiltinCommandLoader', () => { + const loader = new BuiltinCommandLoader(null); + const commands = loader.loadCommandsSync(); + const perf = commands.find((c) => c.name === 'perf'); + + expect(perf).toBeDefined(); + expect(perf!.kind).toBe('built-in' as never); + expect(perf!.description).toContain('Performance telemetry'); + expect(perf!.subCommands).toBeDefined(); + expect(perf!.subCommands!.map((s) => s.name)).toContain('inspect'); + expect(perf!.subCommands!.map((s) => s.name)).toContain('report'); + expect(perf!.subCommands!.map((s) => s.name)).toContain('delete'); + }); + + // --- P12: /perf factory wiring with owned snapshot capability --- + + it('BuiltinCommandLoader uses injected snapshot capability when available', async () => { + const capability: PerfSnapshotCapability = { + getMemorySnapshot: () => [], + getActiveOperationSummary: () => ({ + provider: 'openai', + model: 'gpt-4o', + elapsedMs: 1234, + }), + getSelfHealth: () => ({ lastWriteErrorCode: null, evictionCount: 0 }), + }; + // Minimal config: truthy so ideCommand/restoreCommand are evaluated, but + // they short-circuit to null when their accessors return undefined/false. + const loader = new BuiltinCommandLoader({ + getPerfSnapshotCapability: () => capability, + getProjectTempDir: () => join(tmpdir(), 'token-usage'), + getIdeClient: () => undefined, + getCheckpointingEnabled: () => false, + getEnableHooksUI: () => false, + isSkillsSupportEnabled: () => false, + } as never); + const commands = loader.loadCommandsSync(); + const perf = commands.find((c) => c.name === 'perf')!; + const result = (await perf.action!({} as never, '')) as MessageActionReturn; + expect(result.type).toBe('message'); + expect(result.content).toContain('Active operation'); + expect(result.content).toContain('openai'); + expect(result.content).toContain('gpt-4o'); + }); + + it('BuiltinCommandLoader without snapshot capability says not active', async () => { + const loader = new BuiltinCommandLoader(null); + const commands = loader.loadCommandsSync(); + const perf = commands.find((c) => c.name === 'perf')!; + const result = (await perf.action!({} as never, '')) as MessageActionReturn; + expect(result.type).toBe('message'); + expect(result.content).toContain('not active'); + }); +}); diff --git a/packages/cli/src/ui/commands/perfCommand.ts b/packages/cli/src/ui/commands/perfCommand.ts new file mode 100644 index 0000000000..c58a5c5d85 --- /dev/null +++ b/packages/cli/src/ui/commands/perfCommand.ts @@ -0,0 +1,438 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * `/perf` slash command (P11, issue #3167). + * + * Subcommands: + * /perf inspect — where data lives, what fields, sample counts + * /perf report — longitudinal buildReport() output (optional --baseline) + * /perf delete — remove all perf files (with live-writer safety) + * + * No args (bare `/perf`) produces a snapshot of THIS process (live MemoryRing + * + active operation summary) through an injected `PerfSnapshotCapability`. + * No unowned global singleton — P12 wires the production capability. When + * unavailable/disabled, the snapshot says so honestly. + * + * inspect / report / delete operate on the canonical global log/perf directory + * via `Storage.getGlobalLogDir()/perf`, but accept an injected dir/deps for + * behavioral tests. + */ + +import { join } from 'node:path'; +import type { + MessageActionReturn, + SlashCommand, + CommandContext, +} from './types.js'; +import { CommandKind } from './types.js'; +import { Storage } from '@vybestack/llxprt-code-settings'; +import { + perfInspect, + formatInspect, + buildReport, + formatReport, + perfDelete, + formatDeleteResult, +} from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import type { + PerfInspectResult, + ReportResult, + ReportSelfHealth, + PerfDeleteResult, + PerfDeleteOptions, +} from '@vybestack/llxprt-code-telemetry/perf/index.js'; + +// --------------------------------------------------------------------------- +// Perf snapshot capability (injected runtime — NOT a global singleton) +// --------------------------------------------------------------------------- + +/** + * Injectable perf directory operations. Defaults to the real telemetry + * functions; tests pass a custom implementation to assert error handling + * without global patching. There is no global singleton. + */ +export interface PerfOperations { + inspect(dir: string): Promise; + report( + dir: string, + baseline?: string, + selfHealth?: Partial, + tokenUsageDir?: string, + ): Promise; + delete(options: PerfDeleteOptions): Promise; +} + +/** Default operations backed by the real telemetry functions. */ +const defaultOperations: PerfOperations = { + inspect: (dir) => perfInspect(dir), + report: (dir, baseline, selfHealth, tokenUsageDir) => + buildReport(dir, baseline, selfHealth, tokenUsageDir), + delete: (options) => perfDelete(options), +}; + +/** + * Returns true for a genuine external errno I/O failure (has a Node errno + * code). Non-errno internal/programming errors return false so callers can + * allow them to reject rather than swallowing them as user-facing messages. + */ +function isErrnoError(err: unknown): err is NodeJS.ErrnoException { + return ( + err instanceof Error && + typeof (err as NodeJS.ErrnoException).code === 'string' + ); +} + +/** + * A single live memory sample from the current process ring. + */ +export interface PerfSnapshotSample { + readonly rss: number; + readonly heapUsed: number; + readonly external: number; + readonly arrayBuffers: number; + readonly uptimeMs: number; + readonly msSinceLastOperation: number; + readonly timestampMs: number; +} + +/** + * Active-process self-health exposed for the `/perf report` view. + * `lastWriteErrorCode` is null when the last write succeeded; a string errno + * when it failed. `evictionCount` is 0 when no evictions occurred. These are + * known values — the report distinguishes them from `undefined` (unavailable), + * which occurs when no active runtime capability exists. + */ +export interface PerfSelfHealth { + readonly lastWriteErrorCode: string | null; + readonly evictionCount: number; +} + +/** + * A snapshot of the current process for the bare `/perf` view and report + * self-health. P12 constructs and injects this when perf telemetry is enabled. + */ +export interface PerfSnapshotCapability { + /** Returns the live memory ring samples (oldest→newest), or null if unavailable. */ + getMemorySnapshot(): readonly PerfSnapshotSample[] | null; + /** Returns a summary of the active operation, or null if none. */ + getActiveOperationSummary(): { + readonly provider: string; + readonly model: string; + readonly elapsedMs: number; + } | null; + /** Returns active-process self-health (known null/0 values). */ + getSelfHealth(): PerfSelfHealth; +} + +export interface PerfCommandOptions { + /** Injected live snapshot capability. When null/undefined, bare /perf says unavailable. */ + readonly snapshotCapability?: PerfSnapshotCapability | null; + /** Override the perf directory for tests. Defaults to Storage.getGlobalLogDir()/perf. */ + readonly perfDir?: string; + /** + * Token-usage JSONL directory for the read-time continuation join. When + * provided, /perf report streams and aggregates token rows by operation id. + * Production wires join(config.getProjectTempDir(), 'token-usage'). + */ + readonly tokenUsageDir?: string; + /** Injectable directory operations. Defaults to the real telemetry functions. */ + readonly operations?: PerfOperations; +} + +// --------------------------------------------------------------------------- +// Argument parsing +// --------------------------------------------------------------------------- + +interface ParsedReportArgs { + readonly baseline: string | undefined; + readonly error: string | undefined; +} + +/** + * Parses `/perf report` arguments. Accepts `--baseline ` (exact version + * or sha). Rejects malformed args with a useful error message. + */ +function parseReportArgs(args: string): ParsedReportArgs { + const tokens = args + .trim() + .split(/\s+/) + .filter((t) => t.length > 0); + + let baseline: string | undefined; + let i = 0; + while (i < tokens.length) { + const token = tokens[i]; + if (token === '--baseline') { + i++; + if (i >= tokens.length) { + return { + baseline: undefined, + error: '--baseline requires a value (version or git sha)', + }; + } + const value = tokens[i]; + if (value.startsWith('--')) { + return { + baseline: undefined, + error: `--baseline requires a value, got flag '${value}'`, + }; + } + baseline = value; + i++; + } else { + return { + baseline: undefined, + error: `unexpected argument '${token}'. Usage: /perf report [--baseline ]`, + }; + } + } + + return { baseline, error: undefined }; +} + +// --------------------------------------------------------------------------- +// Helpers +// --------------------------------------------------------------------------- + +function messageInfo(content: string): MessageActionReturn { + return { type: 'message', messageType: 'info', content }; +} + +function messageError(content: string): MessageActionReturn { + return { type: 'message', messageType: 'error', content }; +} + +function getDefaultPerfDir(): string { + return join(Storage.getGlobalLogDir(), 'perf'); +} + +function formatBytes(bytes: number): string { + if (bytes === 0) return '0 B'; + const sign = bytes < 0 ? '-' : ''; + const abs = Math.abs(bytes); + if (abs < 1024) return `${sign}${abs} B`; + if (abs < 1024 * 1024) return `${sign}${(abs / 1024).toFixed(1)} KiB`; + return `${sign}${(abs / (1024 * 1024)).toFixed(1)} MiB`; +} + +// --------------------------------------------------------------------------- +// Live snapshot formatter +// --------------------------------------------------------------------------- + +function formatSnapshot( + capability: PerfSnapshotCapability, +): MessageActionReturn { + const memSamples = capability.getMemorySnapshot(); + const activeOp = capability.getActiveOperationSummary(); + const lines: string[] = []; + + lines.push('Perf Snapshot (this process)'); + lines.push('============================'); + lines.push(''); + + if (activeOp !== null) { + lines.push('Active operation:'); + lines.push(` provider: ${activeOp.provider}`); + lines.push(` model: ${activeOp.model}`); + lines.push(` elapsed: ${activeOp.elapsedMs} ms`); + lines.push(''); + } else { + lines.push('No active operation.'); + lines.push(''); + } + + if (memSamples !== null && memSamples.length > 0) { + const latest = memSamples[memSamples.length - 1]; + const first = memSamples[0]; + lines.push(`Memory samples: ${memSamples.length}`); + lines.push(` latest rss: ${formatBytes(latest.rss)}`); + lines.push(` latest heap: ${formatBytes(latest.heapUsed)}`); + lines.push(` uptime: ${(latest.uptimeMs / 1000).toFixed(1)}s`); + lines.push( + ` idle: ${(latest.msSinceLastOperation / 1000).toFixed(1)}s since last operation`, + ); + if (memSamples.length >= 2) { + const rssDelta = latest.rss - first.rss; + const uptimeDelta = latest.uptimeMs - first.uptimeMs; + if (uptimeDelta > 0) { + const perMin = (rssDelta / uptimeDelta) * 60_000; + lines.push( + ` trend: ${rssDelta >= 0 ? '+' : ''}${formatBytes(rssDelta)} over ${(uptimeDelta / 1000).toFixed(1)}s (${perMin >= 0 ? '+' : ''}${formatBytes(perMin)}/min)`, + ); + } + } + } else { + lines.push('No memory samples collected.'); + } + + return messageInfo(lines.join('\n')); +} + +// --------------------------------------------------------------------------- +// Command factory +// --------------------------------------------------------------------------- + +/** + * Creates the inspect subcommand for the perf directory. + * + * Genuine external errno I/O failures are converted to a user-facing error + * message; non-errno internal/programming errors are allowed to reject so they + * surface as bugs rather than being swallowed. + */ +function createInspectSubCommand( + perfDir: string, + operations: PerfOperations, +): SlashCommand { + return { + name: 'inspect', + description: 'Show where perf data lives and sample counts', + kind: CommandKind.BUILT_IN, + action: async (): Promise => { + try { + const result = await operations.inspect(perfDir); + return messageInfo(formatInspect(result)); + } catch (err) { + if (isErrnoError(err)) { + return messageError(`Failed to inspect perf data: ${err.message}`); + } + throw err; + } + }, + }; +} + +/** + * Creates the report subcommand for the perf directory. + * + * Passes active self-health (from the snapshot capability when available) and + * the token-usage directory to telemetry buildReport so the report reflects + * live sink/retention state and the read-time continuation join. When the + * capability is unavailable, self-health is undefined (formatted as + * "unavailable" rather than falsely claiming null/0). + * + * Genuine external errno I/O failures are converted to a user-facing error + * message; non-errno internal/programming errors are allowed to reject. + */ +function createReportSubCommand( + perfDir: string, + operations: PerfOperations, + getSelfHealth: () => PerfSelfHealth | null, + tokenUsageDir: string | undefined, +): SlashCommand { + return { + name: 'report', + description: + 'Show longitudinal perf trends (optionally vs a --baseline version or sha)', + kind: CommandKind.BUILT_IN, + action: async ( + _ctx: CommandContext, + reportArgs: string, + ): Promise => { + const parsed = parseReportArgs(reportArgs); + if (parsed.error !== undefined) { + return messageError(parsed.error); + } + try { + const selfHealth = getSelfHealth() ?? undefined; + const result = await operations.report( + perfDir, + parsed.baseline, + selfHealth, + tokenUsageDir, + ); + return messageInfo(formatReport(result)); + } catch (err) { + if (isErrnoError(err)) { + return messageError(`Failed to generate perf report: ${err.message}`); + } + throw err; + } + }, + }; +} + +/** + * Creates the delete subcommand for the perf directory. + * + * Genuine external errno I/O failures are converted to a user-facing error + * message; non-errno internal/programming errors are allowed to reject. + */ +function createDeleteSubCommand( + perfDir: string, + operations: PerfOperations, +): SlashCommand { + return { + name: 'delete', + description: 'Delete perf data files (respects active writers)', + kind: CommandKind.BUILT_IN, + action: async (): Promise => { + try { + const result = await operations.delete({ dir: perfDir }); + return messageInfo(formatDeleteResult(result)); + } catch (err) { + if (isErrnoError(err)) { + return messageError(`Failed to delete perf data: ${err.message}`); + } + throw err; + } + }, + }; +} + +/** + * Creates the `/perf` slash command. Accepts an optional snapshot capability, + * perf directory override, token-usage directory, and injectable operations + * for testing. When no snapshot capability is provided, bare `/perf` reports + * that perf telemetry is not active. `/perf report` still works on stored + * data with unavailable self-health and the configured token-usage directory. + */ +export function createPerfCommand( + options: PerfCommandOptions = {}, +): SlashCommand { + const perfDir = options.perfDir ?? getDefaultPerfDir(); + const snapshotCapability = options.snapshotCapability ?? null; + const operations = options.operations ?? defaultOperations; + const tokenUsageDir = options.tokenUsageDir; + const getSelfHealth = (): PerfSelfHealth | null => + snapshotCapability?.getSelfHealth() ?? null; + + return { + name: 'perf', + description: + 'Performance telemetry: inspect, report, delete, or snapshot this process', + kind: CommandKind.BUILT_IN, + action: async ( + _context: CommandContext, + args: string, + ): Promise => { + const trimmed = args.trim(); + if (trimmed === '') { + if (snapshotCapability === null) { + return messageInfo( + 'Perf telemetry is not active in this process. Use /perf inspect to view stored data, or /perf report for trends.', + ); + } + return formatSnapshot(snapshotCapability); + } + return messageError( + `Unknown subcommand. Usage: /perf [inspect|report|delete]`, + ); + }, + subCommands: [ + createInspectSubCommand(perfDir, operations), + createReportSubCommand(perfDir, operations, getSelfHealth, tokenUsageDir), + createDeleteSubCommand(perfDir, operations), + ], + }; +} + +/** + * Default `/perf` command instance registered in BuiltinCommandLoader. + * The production snapshot capability is wired by BuiltinCommandLoader via + * `createPerfCommand({ snapshotCapability })`. + */ +export const perfCommand: SlashCommand = createPerfCommand(); diff --git a/packages/cli/src/ui/commands/perfCommand.wiring.behavior.test.ts b/packages/cli/src/ui/commands/perfCommand.wiring.behavior.test.ts new file mode 100644 index 0000000000..d0ef77a71a --- /dev/null +++ b/packages/cli/src/ui/commands/perfCommand.wiring.behavior.test.ts @@ -0,0 +1,388 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Finding C behavioral tests — production /perf report wiring. + * + * Proves: (1) the default production operation invocation receives the exact + * perf dir, baseline, self-health, and token-usage directory; (2) inactive + * health formats unavailable rather than falsely claiming null/0; (3) active + * clean health is known null/0; (4) active errors/evictions propagate; (5) + * continuation token rows in a real project token-usage directory affect the + * production command report (not helper-only). + * + * Uses REAL files and the REAL production buildReport (default operations) for + * formatting + token-join evidence, plus a capturing operations port for wiring + * verification. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import { promises as fs } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { + createPerfCommand, + type PerfSnapshotCapability, + type PerfOperations, + type PerfSelfHealth, +} from './perfCommand.js'; +import type { MessageActionReturn } from './types.js'; +import type { + ReportResult, + ReportSelfHealth, +} from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import type { PerfOperationRecord } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; + +function makeOperation( + overrides: Partial = {}, +): PerfOperationRecord { + return { + schema_version: 1, + record_type: 'operation', + ts: '2026-01-01T00:00:00.000Z', + session_id: 'sess-1', + operation_id: 'sess-1#agentic-loop#aaaa', + runtime_id: 'rt-1', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'hash-1', + llxprt_version: '0.10.0', + git_sha: 'abc1234', + runtime: 'cli', + platform: 'darwin', + provider: 'test-provider', + model: 'test-model', + context_tokens: 1000, + output_tokens: 500, + terminal_cols: 80, + terminal_rows: 24, + render_mode: 'ink', + concurrent_instances: 1, + status: 'completed', + client_prepare_ms: 10, + stream_handler_ms: 100, + ink_render_ms: 5, + ink_render_count: 3, + stdout_bytes: 1024, + stdout_write_calls: 5, + stdout_write_sync_ms: 2, + client_finalize_ms: 8, + provider_attempts: 1, + provider_attempt_sum_ms: 200, + provider_union_ms: 200, + tool_calls: 2, + tool_call_sum_ms: 50, + tool_union_ms: 50, + agent_activity_union_ms: 250, + operation_elapsed_ms: 1000, + approval_wait_ms: 0, + unclassified_elapsed_ms: 0, + session_operation_index: 0, + uptime_ms: 5000, + ...overrides, + }; +} + +async function makeTempDir(prefix: string): Promise { + const dir = join( + tmpdir(), + `${prefix}-${Date.now()}-${Math.random().toString(36).slice(2)}`, + ); + await fs.mkdir(dir, { recursive: true }); + return dir; +} + +async function writeJsonl( + dir: string, + name: string, + lines: string[], +): Promise { + const content = lines.join('\n'); + await fs.writeFile( + join(dir, name), + content + (content.endsWith('\n') ? '' : '\n'), + 'utf8', + ); +} + +function isMessage(result: unknown): result is MessageActionReturn { + return ( + typeof result === 'object' && + result !== null && + 'type' in result && + (result as { type: string }).type === 'message' + ); +} + +function emptyReportResult(): ReportResult { + return { + groups: [], + counts: { + files: 0, + bytes: 0, + parsed: 0, + malformed: 0, + futureVersion: 0, + unversioned: 0, + truncated: 0, + blank: 0, + }, + selfHealth: { + skipped: 0, + truncated: 0, + lastWriteErrorCode: undefined, + evictionCount: undefined, + }, + baseline: null, + }; +} + +describe('Finding C — production /perf report wiring', () => { + let perfDir: string; + let tokenDir: string; + + beforeEach(async () => { + perfDir = await makeTempDir('perf-wire'); + tokenDir = await makeTempDir('token-wire'); + }); + + afterEach(async () => { + await Promise.all([ + fs.rm(perfDir, { recursive: true, force: true }), + fs.rm(tokenDir, { recursive: true, force: true }), + ]); + }); + + // --- (1) Default operation invocation receives exact params --- + + it('report subcommand passes exact perf dir, baseline, self-health, and token directory', async () => { + let captured: { + dir: string; + baseline: string | undefined; + selfHealth: Partial | undefined; + tokenUsageDir: string | undefined; + } | null = null; + + const capturingOps: PerfOperations = { + inspect: async () => { + throw new Error('not used'); + }, + report: async ( + dir: string, + baseline?: string, + selfHealth?: Partial, + tokenUsageDir?: string, + ): Promise => { + captured = { dir, baseline, selfHealth, tokenUsageDir }; + return emptyReportResult(); + }, + delete: async () => { + throw new Error('not used'); + }, + }; + + const capability: PerfSnapshotCapability = { + getMemorySnapshot: () => null, + getActiveOperationSummary: () => null, + getSelfHealth: (): PerfSelfHealth => ({ + lastWriteErrorCode: null, + evictionCount: 0, + }), + }; + + const cmd = createPerfCommand({ + perfDir, + operations: capturingOps, + snapshotCapability: capability, + tokenUsageDir: tokenDir, + }); + const reportSub = cmd.subCommands!.find((s) => s.name === 'report')!; + await reportSub.action!({} as never, '--baseline 0.10.0'); + + expect(captured).not.toBe(null); + const c = captured!; + expect(c.dir).toBe(perfDir); + expect(c.baseline).toBe('0.10.0'); + expect(c.selfHealth).toEqual({ + lastWriteErrorCode: null, + evictionCount: 0, + }); + expect(c.tokenUsageDir).toBe(tokenDir); + }); + + // --- (2) Inactive health formats unavailable (not null/0) --- + + it('inactive report (no snapshot capability) formats self-health as unavailable', async () => { + await writeJsonl(perfDir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + ]); + + // No snapshotCapability: self-health is undefined → "unavailable". + const cmd = createPerfCommand({ perfDir, tokenUsageDir: tokenDir }); + const reportSub = cmd.subCommands!.find((s) => s.name === 'report')!; + const result = await reportSub.action!({} as never, ''); + + expect(isMessage(result)).toBe(true); + const msg = result as MessageActionReturn; + expect(msg.content).toContain('Self-health:'); + // Undefined lastWriteErrorCode → "unavailable", NOT "none". + expect(msg.content).toContain('last write error: unavailable'); + // Undefined evictionCount → "unavailable", NOT "0". + expect(msg.content).toContain('evictions: unavailable'); + }); + + // --- (3) Active clean health is known null/0 --- + + it('active clean health formats as known null/0 (none and 0)', async () => { + await writeJsonl(perfDir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + ]); + + const capability: PerfSnapshotCapability = { + getMemorySnapshot: () => null, + getActiveOperationSummary: () => null, + getSelfHealth: () => ({ lastWriteErrorCode: null, evictionCount: 0 }), + }; + + const cmd = createPerfCommand({ + perfDir, + snapshotCapability: capability, + tokenUsageDir: tokenDir, + }); + const reportSub = cmd.subCommands!.find((s) => s.name === 'report')!; + const result = await reportSub.action!({} as never, ''); + + const msg = result as MessageActionReturn; + expect(msg.content).toContain('last write error: none'); + expect(msg.content).toContain('evictions: 0'); + }); + + // --- (4) Active errors/evictions propagate --- + + it('active errors/evictions propagate to the report self-health', async () => { + await writeJsonl(perfDir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + ]); + + const capability: PerfSnapshotCapability = { + getMemorySnapshot: () => null, + getActiveOperationSummary: () => null, + getSelfHealth: () => ({ + lastWriteErrorCode: 'ENOSPC', + evictionCount: 7, + }), + }; + + const cmd = createPerfCommand({ + perfDir, + snapshotCapability: capability, + tokenUsageDir: tokenDir, + }); + const reportSub = cmd.subCommands!.find((s) => s.name === 'report')!; + const result = await reportSub.action!({} as never, ''); + + const msg = result as MessageActionReturn; + expect(msg.content).toContain('last write error: ENOSPC'); + expect(msg.content).toContain('evictions: 7'); + }); + + // --- (5) Continuation token rows affect the production command report --- + + it('continuation token rows in a real token-usage directory affect the production report', async () => { + const opId = 'sess-1#agentic-loop#aaaa'; + await writeJsonl(perfDir, 'perf-20260101-run1.jsonl', [ + JSON.stringify( + makeOperation({ + operation_id: opId, + context_tokens: 1000, + output_tokens: 500, + }), + ), + ]); + + // Token usage: initial + 2 continuations. + await writeJsonl(tokenDir, 'usage.jsonl', [ + JSON.stringify({ + prompt_id: opId, + actual_prompt_tokens: 1000, + output_tokens: 100, + }), + JSON.stringify({ + prompt_id: `${opId}#continuation#1`, + actual_prompt_tokens: 2000, + output_tokens: 200, + }), + JSON.stringify({ + prompt_id: `${opId}#continuation#2`, + actual_prompt_tokens: 3000, + output_tokens: 300, + }), + ]); + + // Use the DEFAULT production operations (real buildReport) with a real + // token-usage directory. This is NOT a helper-only test. + const cmd = createPerfCommand({ perfDir, tokenUsageDir: tokenDir }); + const reportSub = cmd.subCommands!.find((s) => s.name === 'report')!; + const result = await reportSub.action!({} as never, ''); + + const msg = result as MessageActionReturn; + // Joined: 1000 + 2000 + 3000 = 6000 (replaces persisted 1000). + expect(msg.content).toContain('context_tokens: p50=6000'); + // Joined output: 100 + 200 + 300 = 600 (replaces persisted 500). + expect(msg.content).toContain('output_tokens: p50=600'); + }); + + // --- (6) Token-usage directory omitted keeps persisted totals --- + + it('report without tokenUsageDir keeps persisted perf token totals', async () => { + await writeJsonl(perfDir, 'perf-20260101-run1.jsonl', [ + JSON.stringify( + makeOperation({ context_tokens: 1234, output_tokens: 567 }), + ), + ]); + + const cmd = createPerfCommand({ perfDir }); + const reportSub = cmd.subCommands!.find((s) => s.name === 'report')!; + const result = await reportSub.action!({} as never, ''); + + const msg = result as MessageActionReturn; + expect(msg.content).toContain('context_tokens: p50=1234'); + expect(msg.content).toContain('output_tokens: p50=567'); + }); + + // --- (7) Self-health is null when capability is null (not passed at all) --- + + it('report without snapshotCapability passes undefined self-health to operations', async () => { + let reportWasCalled = false; + let capturedSelfHealth: Partial | null = null; + + const capturingOps: PerfOperations = { + inspect: async () => { + throw new Error('not used'); + }, + report: async ( + _dir: string, + _baseline?: string, + selfHealth?: Partial, + ): Promise => { + reportWasCalled = true; + capturedSelfHealth = selfHealth ?? null; + return emptyReportResult(); + }, + delete: async () => { + throw new Error('not used'); + }, + }; + + const cmd = createPerfCommand({ perfDir, operations: capturingOps }); + const reportSub = cmd.subCommands!.find((s) => s.name === 'report')!; + await reportSub.action!({} as never, ''); + + // No capability → self-health is undefined (unavailable). + expect(reportWasCalled).toBe(true); + expect(capturedSelfHealth).toBe(null); + }); +}); diff --git a/packages/cli/src/ui/containers/AppContainer/hooks/useAppBootstrap.ts b/packages/cli/src/ui/containers/AppContainer/hooks/useAppBootstrap.ts index 6e6c1eb184..7f553f427e 100644 --- a/packages/cli/src/ui/containers/AppContainer/hooks/useAppBootstrap.ts +++ b/packages/cli/src/ui/containers/AppContainer/hooks/useAppBootstrap.ts @@ -38,6 +38,7 @@ import { useSessionInitialization } from './useSessionInitialization.js'; import { useTokenMetricsTracking } from './useTokenMetricsTracking.js'; import { registerCleanup } from '../../../../utils/cleanup.js'; import type { Agent } from '@vybestack/llxprt-code-agents'; +import type { MemoryTelemetryController } from '../../../hooks/memoryTrend/memoryTelemetry.js'; import type { LoadedSettings } from '../../../../config/settings.js'; import type { HistoryItem } from '../../../types.js'; import type { @@ -62,6 +63,8 @@ export interface AppBootstrapProps { recordingIntegration?: RecordingIntegration; initialRecordingService?: SessionRecordingService; initialLockHandle?: LockHandle | null; + /** P12: optional memory telemetry controller (perf+memory enabled only). */ + memoryController?: MemoryTelemetryController; } export interface AppBootstrapResult { @@ -163,7 +166,7 @@ function useBootstrapHistory(props: AppBootstrapProps) { loadHistory, resumedHistory, }); - useMemoryMonitor({ addItem }); + useMemoryMonitor({ addItem, memoryController: props.memoryController }); return { runtime, isFocused, diff --git a/packages/cli/src/ui/containers/AppContainer/hooks/useAppInput.ts b/packages/cli/src/ui/containers/AppContainer/hooks/useAppInput.ts index 4a3c088dbd..5e1a54ca4b 100644 --- a/packages/cli/src/ui/containers/AppContainer/hooks/useAppInput.ts +++ b/packages/cli/src/ui/containers/AppContainer/hooks/useAppInput.ts @@ -7,6 +7,7 @@ import type React from 'react'; import { useCallback, useMemo, useRef } from 'react'; import { useAgentStream } from '../../../hooks/agentStream/index.js'; +import type { OperationLifecycleRegistry } from '../../../hooks/agentStream/operationLifecycle.js'; import { useAutoAcceptIndicator } from '../../../hooks/useAutoAcceptIndicator.js'; import { useLoadingIndicator } from '../../../hooks/useLoadingIndicator.js'; import { useSlashCommandProcessor } from '../../../hooks/slashCommandProcessor.js'; @@ -100,6 +101,8 @@ export interface AppInputParams { // Direct appState: AppState; appDispatch: React.Dispatch; + /** P12: optional perf operation lifecycle registry (perf enabled only). */ + operationLifecycle?: OperationLifecycleRegistry; } function useInputCoreCallbacks(p: AppInputParams) { @@ -354,6 +357,7 @@ function useInputStreamSetup( runtimeMessageBus, p.subagentManager, removeItems, + p.operationLifecycle, ); return { ...bufferSetup, agentStreamResult }; } diff --git a/packages/cli/src/ui/hooks/agentStream/__tests__/lifecyclePerfFixtures.ts b/packages/cli/src/ui/hooks/agentStream/__tests__/lifecyclePerfFixtures.ts new file mode 100644 index 0000000000..34dd734a7a --- /dev/null +++ b/packages/cli/src/ui/hooks/agentStream/__tests__/lifecyclePerfFixtures.ts @@ -0,0 +1,288 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Shared behavioral fixtures for useSubmitQuery operation-lifecycle integration + * tests. Provides a real PerfSink / PerfRetention / OperationLifecycleRegistry + * writing to temp files, and reads records back through the real tolerant + * reader. Each call to {@link createLifecyclePerfHarness} returns an isolated + * context so multiple test files can import this module without sharing state. + */ + +import { vi } from 'bun:test'; +import type React from 'react'; +import type { UseSubmitQueryDeps } from '../useSubmitQuery.js'; +import { StreamingState, type HistoryItemWithoutId } from '../../../types.js'; +import type { Agent } from '@vybestack/llxprt-code-agents'; +import { + type AgentClientContract, + type RecordingIntegration, +} from '@vybestack/llxprt-code-core'; +import { + PerfSink, + PerfRetention, +} from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import { readPerfRecords } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import type { PerfOperationRecord } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import { createStreamRuntimeForTest } from './streamRuntimeTestHelper.js'; +import { PendingResponseBuffer } from '../pendingResponseBuffer.js'; +import { + OperationLifecycleRegistry, + type OperationIdentitySnapshot, + type OperationIdentityProvider, +} from '../operationLifecycle.js'; +import { mkdtemp, rm, readdir } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +export function fixtureIdentity(): OperationIdentitySnapshot { + return { + session_id: 'test-session', + runtime_id: 'runtime-uuid', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'proj-hash', + llxprt_version: '0.0.0-test', + git_sha: 'deadbeef', + runtime: 'bun', + platform: 'darwin', + provider: 'test-provider', + model: 'test-model', + terminal_cols: 80, + terminal_rows: 24, + render_mode: 'ink', + }; +} + +export function createMockAgentClient(): AgentClientContract { + return { + getCurrentSequenceModel: () => 'test-model', + getChat: () => + ({ + recordCompletedToolCalls: vi.fn(), + }) as never, + } as unknown as AgentClientContract; +} + +export interface LifecycleDeps { + abortControllerRef: React.MutableRefObject; + runStreamRef: React.MutableRefObject< + | (( + message: unknown, + signal: AbortSignal, + promptId: string, + ) => Promise) + | null + >; + pendingResponse: PendingResponseBuffer; + setIsRespondingCalls: boolean[]; +} + +export function createLifecycleDeps( + options?: Partial, +): LifecycleDeps { + const setIsRespondingCalls: boolean[] = []; + return { + abortControllerRef: + options?.abortControllerRef ?? + ({ current: null as AbortController | null } as never), + runStreamRef: options?.runStreamRef ?? ({ current: null } as never), + pendingResponse: + options?.pendingResponse ?? new PendingResponseBuffer(undefined), + setIsRespondingCalls, + }; +} + +export function buildLifecycleHookDeps( + deps: LifecycleDeps, + registry: OperationLifecycleRegistry, +): UseSubmitQueryDeps { + return { + runtime: createStreamRuntimeForTest(), + agent: createMockAgentClient() as unknown as Agent, + addItem: vi.fn().mockReturnValue(1), + settings: {} as never, + onDebugMessage: vi.fn(), + onCancelSubmit: vi.fn(), + setTurnCancelled: vi.fn(), + onAuthError: vi.fn(), + sanitizeContent: (text: string) => ({ text, blocked: false }), + flushPendingHistoryItem: vi.fn(), + pendingResponse: deps.pendingResponse, + pendingHistoryItemRef: { + current: null, + } as React.MutableRefObject, + thinkingBlocksRef: { current: [] }, + turnCancelledRef: { current: false }, + queuedSubmissionsRef: { current: [] }, + drainSuppressedRef: { current: false }, + enqueueSubmission: vi.fn(), + enqueueSubmissionFirst: vi.fn(), + requeueSubmission: vi.fn(), + dequeueSubmission: vi.fn(), + clearSubmissions: vi.fn(), + tryReserveDrain: vi.fn().mockReturnValue(true), + releaseDrain: vi.fn(), + setPendingHistoryItem: vi.fn(), + setIsResponding: vi.fn((value: unknown) => { + if (typeof value === 'boolean') deps.setIsRespondingCalls.push(value); + }) as never, + setInitError: vi.fn(), + setThought: vi.fn(), + setLastAgentActivityTime: vi.fn(), + scheduleToolCalls: vi.fn(), + abortActiveStream: vi.fn(), + handleShellCommand: vi.fn().mockReturnValue(false), + handleSlashCommand: vi.fn().mockResolvedValue(false), + logger: null, + shellModeActive: false, + loopDetectedRef: { current: false }, + lastProfileNameRef: { current: undefined }, + lastModelInfoRef: { current: null }, + lastModelIdentityRef: { current: null }, + abortControllerRef: deps.abortControllerRef, + runStreamRef: deps.runStreamRef, + submitQueryRef: { current: null }, + isResponding: false, + streamingState: StreamingState.Idle, + recordingIntegration: { + flushAtTurnBoundary: vi.fn(), + } as unknown as RecordingIntegration, + operationLifecycle: registry, + }; +} + +export interface LifecyclePerfHarness { + readonly registry: OperationLifecycleRegistry; + setup: () => Promise; + drainAndRead: () => Promise; + cleanup: () => Promise; +} + +interface LifecyclePerfHarnessState { + perfDir: string; + sink: PerfSink | null; + retention: PerfRetention | null; + registry: OperationLifecycleRegistry | null; + sinkDisposed: boolean; +} + +async function disposeHarnessSink( + state: LifecyclePerfHarnessState, + errors: unknown[], +): Promise { + if (state.sink === null || state.sinkDisposed) return; + try { + await state.sink.dispose(); + state.sinkDisposed = true; + } catch (error) { + errors.push(error); + } +} + +async function removeHarnessDirectory( + state: LifecyclePerfHarnessState, + errors: unknown[], +): Promise { + if (state.perfDir === '') return; + try { + await rm(state.perfDir, { recursive: true, force: true }); + } catch (error) { + errors.push(error); + } +} + +async function setupHarness(state: LifecyclePerfHarnessState): Promise { + state.perfDir = await mkdtemp(join(tmpdir(), 'perf-lifecycle-')); + try { + const runUuid = crypto.randomUUID(); + state.retention = new PerfRetention({ dir: state.perfDir, runUuid }); + state.sink = new PerfSink({ + dir: state.perfDir, + runUuid, + retention: state.retention, + }); + await state.sink.start(); + const provider: OperationIdentityProvider = { + snapshot: () => fixtureIdentity(), + }; + state.registry = new OperationLifecycleRegistry({ + identityProvider: provider, + sink: state.sink, + retention: state.retention, + }); + } catch (error) { + const cleanupErrors: unknown[] = []; + await disposeHarnessSink(state, cleanupErrors); + await removeHarnessDirectory(state, cleanupErrors); + if (cleanupErrors.length > 0) { + throw new AggregateError( + [error, ...cleanupErrors], + 'setup partial-failure cleanup also failed', + ); + } + throw error; + } +} + +async function readHarnessRecords( + state: LifecyclePerfHarnessState, +): Promise { + const records: PerfOperationRecord[] = []; + for (const name of await readdir(state.perfDir)) { + if (!name.endsWith('.jsonl')) continue; + const result = await readPerfRecords(join(state.perfDir, name)); + for (const record of result.records) { + if (record.record_type === 'operation') { + records.push(record); + } + } + } + return records; +} + +async function drainAndReadHarness( + state: LifecyclePerfHarnessState, +): Promise { + if (state.registry === null || state.sink === null) { + throw new Error('LifecyclePerfHarness.setup() must be called first'); + } + await state.registry.drain(); + await state.sink.dispose(); + state.sinkDisposed = true; + return readHarnessRecords(state); +} + +async function cleanupHarness(state: LifecyclePerfHarnessState): Promise { + const errors: unknown[] = []; + await disposeHarnessSink(state, errors); + await removeHarnessDirectory(state, errors); + if (errors.length === 1) throw errors[0]; + if (errors.length > 1) { + throw new AggregateError(errors, 'lifecycle perf harness cleanup failed'); + } +} + +export function createLifecyclePerfHarness(): LifecyclePerfHarness { + const state: LifecyclePerfHarnessState = { + perfDir: '', + sink: null, + retention: null, + registry: null, + sinkDisposed: false, + }; + return { + get registry(): OperationLifecycleRegistry { + if (state.registry === null) { + throw new Error('LifecyclePerfHarness.setup() must be called first'); + } + return state.registry; + }, + setup: () => setupHarness(state), + drainAndRead: () => drainAndReadHarness(state), + cleanup: () => cleanupHarness(state), + }; +} diff --git a/packages/cli/src/ui/hooks/agentStream/__tests__/overheadHarness.useSubmitQuery.test.tsx b/packages/cli/src/ui/hooks/agentStream/__tests__/overheadHarness.useSubmitQuery.test.tsx new file mode 100644 index 0000000000..eefec4c7ff --- /dev/null +++ b/packages/cli/src/ui/hooks/agentStream/__tests__/overheadHarness.useSubmitQuery.test.tsx @@ -0,0 +1,504 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +// Enable React's act() environment so hook state updates are flushed. +( + globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean } +).IS_REACT_ACT_ENVIRONMENT = true; + +/** + * P12 overhead harness — REAL useSubmitQuery fixture-stream overhead + * (EVIDENCE-AC12, Item 8). + * + * Both ENABLED and DISABLED scenarios consume the SAME deterministic fixture + * async stream workload through REAL useSubmitQuery orchestration. + * + * What is REAL in enabled mode: + * - useSubmitQuery (the actual hook with all turn-path control flow) + * - createInteractivePerfRuntime owner (single owner) with owner.start() so + * observer installation (stdout/render/phase) is genuinely performed — + * matching these comments rather than merely asserting it. + * - OperationLifecycleRegistry (begin/finalise/superseded sweep) + * - PerfSink + PerfRetention (real filesystem writes, claim lifecycle, + * maintenance timer cleared on dispose) + * - Record reader (tolerant JSONL parser) + * + * Ownership / leak discipline: + * - After the enabled workload, the owner is disposed BEFORE the disabled + * workload in deterministic order: observers null, claim removed, timer + * cleared. The owner is retained for afterEach emergency cleanup and + * disposed before the perf directory is removed. + * - The disabled workload runs with operationLifecycle undefined and no + * installed observers. It is asserted to add NO JSONL rows/artifacts by + * diffing the on-disk file set — not by inspecting a local empty array. + * - Both renderHook harnesses are unmounted under act(). + * + * What is the EXTERNAL deterministic fixture: + * - runStreamRef.current: a deterministic async generator that yields N + * content chunks per turn and resolves. Both enabled and disabled paths + * consume the exact same generator function with the same chunk count. + * - prepareQueryForAgent/prepareTurnForQuery: stubbed to always proceed + * (same for both paths). + * + * Prints p50/p95/p99 + delta (evidence). Asserts counts, status, schema + * validity, and cleanup only — no timing threshold. + */ + +import { describe, it, expect, vi, beforeEach, afterEach } from 'bun:test'; +import { act } from 'react'; +import { renderHook } from '../../../../test-utils/render.js'; +import { useSubmitQuery, type UseSubmitQueryDeps } from '../useSubmitQuery.js'; +import { StreamingState, type HistoryItemWithoutId } from '../../../types.js'; +import type { + AgentClientContract, + RecordingIntegration, +} from '@vybestack/llxprt-code-core'; +import type { Agent } from '@vybestack/llxprt-code-agents'; +import { readPerfRecords } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import type { PerfOperationRecord } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import type { PerfScheduler } from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import { createStreamRuntimeForTest } from './streamRuntimeTestHelper.js'; +import { PendingResponseBuffer } from '../pendingResponseBuffer.js'; +import { + type OperationIdentitySnapshot, + type OperationIdentityProvider, +} from '../operationLifecycle.js'; +import { + createInteractivePerfRuntime, + type InteractivePerfRuntime, +} from '../../perf/interactivePerfRuntime.js'; +import { + getInteractiveStdoutObserver, + getInteractiveRenderObserver, + setInteractiveStdoutObserver, + setInteractiveRenderObserver, +} from '../../../inkRenderOptions.js'; +import { + getPerfPhaseObserver, + setPerfPhaseObserver, +} from '@vybestack/llxprt-code-telemetry/perf/perfPhaseObserver.js'; +import { mkdtemp, rm, readdir } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +// ─── Module mocks (same as useSubmitQuery.lifecycle) ──────────────────────── + +let shouldProceedValue = true; +let queryToSendValue: string | null = 'test-query'; +let prepareQueryReject: unknown | null = null; +let prepareTurnReject: unknown | null = null; + +void vi.mock('../useStreamEventHandlers.js', () => ({ + useStreamEventHandlers: () => ({ + processStreamEvent: vi.fn(), + displayUserMessage: vi.fn(), + prepareQueryForAgent: vi.fn().mockImplementation(() => { + if (prepareQueryReject !== null) { + return Promise.reject(prepareQueryReject); + } + return Promise.resolve({ + queryToSend: queryToSendValue, + shouldProceed: shouldProceedValue, + }); + }), + handleLoopDetectedEvent: vi.fn(), + }), +})); + +void vi.mock('../../../contexts/SessionContext.js', () => ({ + useSessionStats: () => ({ + startNewPrompt: vi.fn(), + getPromptCount: () => 0, + }), +})); + +void vi.mock('../turnPreparation.js', () => ({ + prepareTurnForQuery: vi.fn().mockImplementation(() => { + if (prepareTurnReject !== null) { + return Promise.reject(prepareTurnReject); + } + return Promise.resolve(undefined); + }), +})); + +void vi.mock('../streamUtils.js', () => ({ + handleSubmissionError: vi.fn(), + processSlashCommandResult: vi.fn(), +})); + +void vi.mock('../agentEventDispatcher.js', () => ({ + dispatchAgentEvent: vi.fn(() => ({ agentMessageBuffer: '' })), +})); + +// ─── Deterministic fixture stream ─────────────────────────────────────────── + +/** + * Deterministic async generator that yields CHUNKS_PER_TURN content chunks + * then resolves. Both enabled and disabled scenarios consume this exact same + * workload. The yield timing is deterministic (microtask gap per chunk). + */ +async function* deterministicFixtureStream( + _msg: unknown, + _signal: AbortSignal, + _promptId: string, +): AsyncGenerator { + const CHUNKS = 10; + for (let i = 0; i < CHUNKS; i++) { + yield; + // Deterministic microtask gap. + await Promise.resolve(); + } +} + +/** + * Creates a deterministic runStream function that consumes the fixture stream. + * Returns a promise that resolves after the stream completes. Both enabled + * and disabled paths use this exact function. + */ +function makeDeterministicRunStream(): ( + msg: unknown, + signal: AbortSignal, + promptId: string, +) => Promise { + return async (_msg, _signal, _promptId) => { + for await (const _chunk of deterministicFixtureStream( + _msg, + _signal, + _promptId, + )) { + // The fixture stream just yields — the real runStream would dispatch + // events. Here we prove the useSubmitQuery orchestration path + // (begin → prepare → send → finalise) is exercised with the same + // workload in both modes. + } + }; +} + +// ─── Identity fixture ─────────────────────────────────────────────────────── + +function fixtureIdentity(): OperationIdentitySnapshot { + return { + session_id: 'sess-overhead', + runtime_id: 'rt-overhead', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'proj-hash', + llxprt_version: '0.11.0', + git_sha: 'abc1234', + runtime: 'bun', + platform: `${process.platform}-${process.arch}`, + provider: 'test-provider', + model: 'test-model', + terminal_cols: 80, + terminal_rows: 24, + render_mode: 'incremental', + }; +} + +/** + * Scheduler that counts timer clear() calls so the test can prove the owner's + * retention maintenance interval is cancelled on dispose (no dangling timer). + */ +class CountingScheduler implements PerfScheduler { + clearCount = 0; + setInterval(_callback: () => Promise, _ms: number) { + return { + unref() {}, + // Arrow captures the lexical `this` (the instance) without aliasing it. + clear: () => { + this.clearCount += 1; + }, + }; + } +} + +// ─── Harness ──────────────────────────────────────────────────────────────── + +const TURNS = 15; + +let perfDir: string; +let owner: InteractivePerfRuntime | null = null; +let scheduler: CountingScheduler; + +function createMockAgentClient(): AgentClientContract { + return { + getCurrentSequenceModel: () => 'test-model', + getChat: () => + ({ + recordCompletedToolCalls: vi.fn(), + }) as never, + } as unknown as AgentClientContract; +} + +function renderUseSubmitQuery(opts: { + runStream: ReturnType; + operationLifecycle: InteractivePerfRuntime['registry'] | undefined; +}): { + result: { current: ReturnType }; + unmount: () => void; +} { + const hookDeps: UseSubmitQueryDeps = { + runtime: createStreamRuntimeForTest(), + agent: createMockAgentClient() as unknown as Agent, + addItem: vi.fn().mockReturnValue(1), + settings: {} as never, + onDebugMessage: vi.fn(), + onCancelSubmit: vi.fn(), + setTurnCancelled: vi.fn(), + onAuthError: vi.fn(), + sanitizeContent: (text: string) => ({ text, blocked: false }), + flushPendingHistoryItem: vi.fn(), + pendingResponse: new PendingResponseBuffer(undefined), + pendingHistoryItemRef: { + current: null, + } as React.MutableRefObject, + thinkingBlocksRef: { current: [] }, + turnCancelledRef: { current: false }, + queuedSubmissionsRef: { current: [] }, + drainSuppressedRef: { current: false }, + enqueueSubmission: vi.fn(), + enqueueSubmissionFirst: vi.fn(), + requeueSubmission: vi.fn(), + dequeueSubmission: vi.fn(), + clearSubmissions: vi.fn(), + tryReserveDrain: vi.fn().mockReturnValue(true), + releaseDrain: vi.fn(), + setPendingHistoryItem: vi.fn(), + setIsResponding: vi.fn(), + setInitError: vi.fn(), + setThought: vi.fn(), + setLastAgentActivityTime: vi.fn(), + scheduleToolCalls: vi.fn(), + abortActiveStream: vi.fn(), + handleShellCommand: vi.fn().mockReturnValue(false), + handleSlashCommand: vi.fn().mockResolvedValue(false), + logger: null, + shellModeActive: false, + loopDetectedRef: { current: false }, + lastProfileNameRef: { current: undefined }, + lastModelInfoRef: { current: null }, + lastModelIdentityRef: { current: null }, + abortControllerRef: { current: null }, + runStreamRef: { current: opts.runStream }, + submitQueryRef: { current: null }, + isResponding: false, + streamingState: StreamingState.Idle, + recordingIntegration: { + flushAtTurnBoundary: vi.fn(), + } as unknown as RecordingIntegration, + operationLifecycle: opts.operationLifecycle, + }; + + return renderHook(() => useSubmitQuery(hookDeps)); +} + +async function runScenario( + operationLifecycle: InteractivePerfRuntime['registry'] | undefined, + turns: number, +): Promise<{ perOpMs: number[] }> { + const runStream = makeDeterministicRunStream(); + const { result, unmount } = renderUseSubmitQuery({ + runStream, + operationLifecycle, + }); + + const perOpMs: number[] = []; + + try { + for (let i = 0; i < turns; i++) { + const t0 = performance.now(); + await act(async () => { + await result.current.submitQuery( + `turn-${i}`, + undefined, + `sess-overhead#agentic-loop#turn-${i}`, + ); + }); + perOpMs.push(performance.now() - t0); + } + } finally { + // Unmount the renderHook harness under act (renderHook wraps unmount in act). + unmount(); + } + + return { perOpMs }; +} + +/** + * Reads every operation record from all JSONL files in the perf directory. + */ +async function readOperationRecords( + dir: string, +): Promise { + const records: PerfOperationRecord[] = []; + const names = await readdir(dir); + for (const name of names) { + if (!name.endsWith('.jsonl')) continue; + const result = await readPerfRecords(join(dir, name)); + for (const r of result.records) { + if (r.record_type === 'operation') { + records.push(r); + } + } + } + return records; +} + +function percentile(sorted: number[], p: number): number { + if (sorted.length === 0) return 0; + const idx = Math.min( + sorted.length - 1, + Math.ceil((p / 100) * sorted.length) - 1, + ); + return sorted[Math.max(0, idx)]; +} + +// ─── Tests ────────────────────────────────────────────────────────────────── + +describe('overhead harness — real useSubmitQuery fixture-stream overhead (Item 8)', () => { + beforeEach(async () => { + prepareQueryReject = null; + prepareTurnReject = null; + shouldProceedValue = true; + queryToSendValue = 'test-query'; + setInteractiveStdoutObserver(null); + setInteractiveRenderObserver(null); + setPerfPhaseObserver(null); + + perfDir = await mkdtemp(join(tmpdir(), 'perf-overhead-')); + scheduler = new CountingScheduler(); + owner = createInteractivePerfRuntime({ + enabled: true, + memoryEnabled: false, + perfDir, + identityProvider: { + snapshot: () => fixtureIdentity(), + } satisfies OperationIdentityProvider, + runUuid: 'overhead-enabled', + __schedulerForTesting: scheduler, + }); + // Real owner start: installs observers (stdout/render/phase), creates the + // claim, and starts the retention maintenance timer — genuinely, not just + // claimed in a comment. + await owner!.start(); + }); + + afterEach(async () => { + // Emergency cleanup: dispose any owner still alive (e.g. a failed test) + // BEFORE removing the perf directory. + if (owner !== null) { + try { + await owner.dispose(); + } catch { + // best-effort + } + owner = null; + } + setInteractiveStdoutObserver(null); + setInteractiveRenderObserver(null); + setPerfPhaseObserver(null); + try { + await rm(perfDir, { recursive: true, force: true }); + } catch { + // best-effort + } + }); + + it('prints p50/p95/p99 + delta; enabled produces N records, disabled zero artifacts', async () => { + // --- ENABLED scenario: real useSubmitQuery + real owner/registry/sink --- + expect(getInteractiveStdoutObserver()).toBe(owner!.registry); + expect(getInteractiveRenderObserver()).toBe(owner!.registry); + expect(getPerfPhaseObserver()).toBe(owner!.registry); + + const enabledResult = await runScenario(owner!.registry, TURNS); + + // Deterministically drain pending writes through the real registry. + await owner!.registry.drain(); + + // Read enabled records from disk. + const enabledRecords = await readOperationRecords(perfDir); + + // --- Dispose owner BEFORE the disabled workload, in deterministic order --- + await owner!.dispose(); + owner = null; + + // Real disposal: observers cleared, timer cancelled, claim removed. + expect(getInteractiveStdoutObserver()).toBe(null); + expect(getInteractiveRenderObserver()).toBe(null); + expect(getPerfPhaseObserver()).toBe(null); + expect(scheduler.clearCount).toBeGreaterThanOrEqual(1); + const filesAfterEnabled = await readdir(perfDir); + expect(filesAfterEnabled.some((f) => f.endsWith('.claim'))).toBe(false); + + // Snapshot the artifact set produced by the enabled workload. + const artifactsAfterEnabled = new Set(filesAfterEnabled); + + // --- DISABLED scenario: same fixture streams, NO operationLifecycle --- + const disabledResult = await runScenario(undefined, TURNS); + + // No operationLifecycle ⇒ no installed observers remain. + expect(getInteractiveStdoutObserver()).toBe(null); + expect(getInteractiveRenderObserver()).toBe(null); + expect(getPerfPhaseObserver()).toBe(null); + + // The disabled workload must not add any new JSONL rows or artifacts: + // diff the on-disk file set rather than inspecting a local empty array. + const filesAfterDisabled = await readdir(perfDir); + expect(filesAfterDisabled.sort()).toEqual( + [...artifactsAfterEnabled].sort(), + ); + + // --- PRINT evidence --- + const enabledSorted = [...enabledResult.perOpMs].sort((a, b) => a - b); + const disabledSorted = [...disabledResult.perOpMs].sort((a, b) => a - b); + const ep50 = percentile(enabledSorted, 50); + const ep95 = percentile(enabledSorted, 95); + const ep99 = percentile(enabledSorted, 99); + const dp50 = percentile(disabledSorted, 50); + const dp95 = percentile(disabledSorted, 95); + const dp99 = percentile(disabledSorted, 99); + + process.stdout.write(` +=== P12 Overhead Harness (REAL useSubmitQuery) === +Turns per scenario: ${TURNS} +ENABLED p50=${ep50.toFixed(4)}ms p95=${ep95.toFixed(4)}ms p99=${ep99.toFixed(4)}ms +DISABLED p50=${dp50.toFixed(4)}ms p95=${dp95.toFixed(4)}ms p99=${dp99.toFixed(4)}ms +DELTA p50=${(ep50 - dp50).toFixed(4)}ms p95=${(ep95 - dp95).toFixed(4)}ms p99=${(ep99 - dp99).toFixed(4)}ms +=== End overhead evidence === +`); + + // --- ASSERT stable invariants --- + + // 1. Enabled ⇒ exactly TURNS operation records on disk. + expect(enabledRecords.length).toBe(TURNS); + + // 2. All enabled records have valid schema (status, operation_id, index). + for (const rec of enabledRecords) { + expect(rec.record_type).toBe('operation'); + expect(rec.status).toBe('completed'); + expect(rec.operation_id).toContain('sess-overhead#agentic-loop#turn-'); + expect(Number.isFinite(rec.session_operation_index)).toBe(true); + } + + // 3. Session operation indices are 0..TURNS-1 (monotonic). + const indices = enabledRecords + .map((r) => r.session_operation_index) + .sort((a, b) => a - b); + for (let i = 0; i < TURNS; i++) { + expect(indices[i]).toBe(i); + } + + // 4. Disabled ⇒ no new perf records (real on-disk diff proven above). + + // 5. All per-op measurements are finite. + for (const ms of enabledResult.perOpMs) { + expect(Number.isFinite(ms)).toBe(true); + } + for (const ms of disabledResult.perOpMs) { + expect(Number.isFinite(ms)).toBe(true); + } + }); +}); diff --git a/packages/cli/src/ui/hooks/agentStream/__tests__/useAgentEventStream.defaultoff.p07.bun.tsx b/packages/cli/src/ui/hooks/agentStream/__tests__/useAgentEventStream.defaultoff.p07.bun.tsx new file mode 100644 index 0000000000..737acee971 --- /dev/null +++ b/packages/cli/src/ui/hooks/agentStream/__tests__/useAgentEventStream.defaultoff.p07.bun.tsx @@ -0,0 +1,233 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +// Enable React's act() environment so hook state updates are flushed. +( + globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean } +).IS_REACT_ACT_ENVIRONMENT = true; + +/** + * P07 default-off event-dispatch behavioral tests (issue #3167). + * + * useAgentEventStream routes each AgentEvent to the React state handler + * (processAgentEvent). When a perf observer (onAgentEventObserved) is present, + * it ALSO measures synchronous dispatch and invokes the observer OUTSIDE the + * generic catch (D8: a perf-callback throw rejects the stream). When the + * observer is ABSENT (perf disabled — the default), there must be NO timing + * work per event and NO sample allocation. + * + * These tests exercise the REAL useAgentEventStream through the REAL event + * iteration loop (a lightweight fake Agent yielding canned AgentEvents). They + * inject a package-private monotonic-clock seam to prove the absent-observer + * path performs zero timing calls. No mock theater around the dispatch logic. + */ + +import { describe, it, expect, beforeEach, afterEach, vi } from 'bun:test'; +import { renderHook } from '../../../../test-utils/render.js'; +import { act } from 'react'; +import type { AgentEvent, Agent } from '@vybestack/llxprt-code-agents'; +import type { ContentBlock, IContent } from '@vybestack/llxprt-code-core'; +import type { + AgentEventRouter, + UseAgentEventStreamReturn, +} from '../useAgentEventStream.js'; +import { + useAgentEventStream, + __setMonotonicClockForTesting, +} from '../useAgentEventStream.js'; +import { createFakeAgent } from './helpers/createFakeAgent.js'; + +beforeEach(() => { + __setMonotonicClockForTesting(null); +}); + +afterEach(() => { + __setMonotonicClockForTesting(null); +}); + +function makeCountingClock(): { + clock: () => number; + calls: () => number; +} { + let calls = 0; + let t = 0; + return { + clock: () => { + calls += 1; + t += 1; + return t; + }, + calls: () => calls, + }; +} + +function renderStreamHook( + agent: Agent, + onAgentEventObserved?: ( + event: AgentEvent, + signal: AbortSignal, + handlerMs: number, + ) => void, +): { + result: { + current: { + runStream: UseAgentEventStreamReturn['runStream']; + }; + }; + unmount: () => void; + processAgentEventRef: React.MutableRefObject; +} { + const processAgentEventRef: React.MutableRefObject = + { current: null }; + const { result, unmount } = renderHook(() => + useAgentEventStream({ + agent, + addItem: vi.fn(), + processAgentEventRef, + flushPendingHistoryItem: vi.fn(), + clearPendingHistoryItem: vi.fn(), + performMemoryRefresh: vi.fn().mockResolvedValue(undefined), + onAgentEventObserved, + }), + ); + return { result, unmount, processAgentEventRef }; +} + +describe('useAgentEventStream default-off event dispatch (P07)', () => { + it('absent observer performs NO monotonic-clock calls per event', async () => { + const events: AgentEvent[] = [ + { type: 'text', text: 'a' }, + { type: 'text', text: 'b' }, + { type: 'text', text: 'c' }, + { type: 'done', reason: 'stop' }, + ]; + const agent = createFakeAgent(events); + const { clock, calls } = makeCountingClock(); + __setMonotonicClockForTesting(clock); + + const routed: AgentEvent[] = []; + const { result, unmount, processAgentEventRef } = renderStreamHook(agent); + processAgentEventRef.current = (event: AgentEvent) => routed.push(event); + + const controller = new AbortController(); + await act(async () => { + await result.current.runStream( + 'hi' as string | ContentBlock[] | IContent, + controller.signal, + 'prompt-defaultoff', + ); + }); + + // Events were still routed (proving the loop ran). + expect(routed).toHaveLength(4); + // NO timing work happened because no observer was supplied. + expect(calls()).toBe(0); + unmount(); + }); + + it('absent observer still continues after an ordinary handler error', async () => { + const events: AgentEvent[] = [ + { type: 'text', text: 'before-error' }, + { type: 'text', text: 'throws' }, + { type: 'text', text: 'after-error' }, + { type: 'done', reason: 'stop' }, + ]; + const agent = createFakeAgent(events); + const { clock, calls } = makeCountingClock(); + __setMonotonicClockForTesting(clock); + + const routed: string[] = []; + const { result, unmount, processAgentEventRef } = renderStreamHook(agent); + processAgentEventRef.current = (event: AgentEvent) => { + if (event.type === 'text' && event.text === 'throws') { + throw new Error('ordinary handler error'); + } + routed.push(event.type === 'text' ? event.text : event.type); + }; + + const controller = new AbortController(); + await act(async () => { + await result.current.runStream( + 'hi' as string | ContentBlock[] | IContent, + controller.signal, + 'prompt-handler-error', + ); + }); + + // The bad event was swallowed; subsequent events still arrived. + expect(routed).toEqual(['before-error', 'after-error', 'done']); + // Still no timing work with an absent observer. + expect(calls()).toBe(0); + unmount(); + }); + + it('present observer measures dispatch and is invoked (clock IS used)', async () => { + const events: AgentEvent[] = [ + { type: 'text', text: 'x' }, + { type: 'done', reason: 'stop' }, + ]; + const agent = createFakeAgent(events); + const { clock, calls } = makeCountingClock(); + __setMonotonicClockForTesting(clock); + + const observed: Array<{ event: string; handlerMs: number }> = []; + const { result, unmount, processAgentEventRef } = renderStreamHook( + agent, + (event, _signal, handlerMs) => { + observed.push({ event: event.type, handlerMs }); + }, + ); + processAgentEventRef.current = () => {}; + + const controller = new AbortController(); + await act(async () => { + await result.current.runStream( + 'hi' as string | ContentBlock[] | IContent, + controller.signal, + 'prompt-observer-present', + ); + }); + + // The observer fired once per event, AFTER the timing measurement. + expect(observed).toHaveLength(2); + expect(observed.map((o) => o.event)).toEqual(['text', 'done']); + // Two clock calls per observed event (start + end). + expect(calls()).toBe(4); + // handlerMs is the synchronous dispatch delta (positive because the + // counting clock advances by 1 each call). + expect(observed[0].handlerMs).toBeGreaterThan(0); + unmount(); + }); + + it('present observer that throws rejects the stream (fail-fast)', async () => { + const events: AgentEvent[] = [ + { type: 'text', text: 'first' }, + { type: 'text', text: 'second' }, + { type: 'done', reason: 'stop' }, + ]; + const agent = createFakeAgent(events); + + const { result, unmount, processAgentEventRef } = renderStreamHook( + agent, + () => { + throw new Error('perf observer internal error'); + }, + ); + processAgentEventRef.current = () => {}; + + const controller = new AbortController(); + await act(async () => { + await expect( + result.current.runStream( + 'hi' as string | ContentBlock[] | IContent, + controller.signal, + 'prompt-observer-throw', + ), + ).rejects.toThrow('perf observer internal error'); + }); + unmount(); + }); +}); diff --git a/packages/cli/src/ui/hooks/agentStream/__tests__/useSubmitQuery.lifecycle.cancellation.bun.tsx b/packages/cli/src/ui/hooks/agentStream/__tests__/useSubmitQuery.lifecycle.cancellation.bun.tsx new file mode 100644 index 0000000000..146b5eaaf6 --- /dev/null +++ b/packages/cli/src/ui/hooks/agentStream/__tests__/useSubmitQuery.lifecycle.cancellation.bun.tsx @@ -0,0 +1,360 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +// Enable React's act() environment so hook state updates are flushed. +( + globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean } +).IS_REACT_ACT_ENVIRONMENT = true; + +/** + * P07: granular cancellation classification integration tests (AC-4). + * + * Split from useSubmitQuery.lifecycle.test.tsx. Proves that + * cancelled_during_api, cancelled_during_tool, cancelled_during_approval are + * selected from real live/terminal phase evidence on AbortSignal + * cancellation, using a real OperationLifecycleRegistry + PerfSink writing to + * temp files. Mocks are limited to external boundaries + * (runStream, prepareQueryForAgent, event handlers) — the lifecycle, sink, + * and retention are real. + */ + +import { describe, it, expect, vi, beforeEach, afterEach } from 'bun:test'; +import { act } from 'react'; +import { renderHook, waitFor } from '../../../../test-utils/render.js'; +import { useSubmitQuery } from '../useSubmitQuery.js'; +import { createDeferred } from './createDeferred.js'; +import { + createLifecycleDeps, + createLifecyclePerfHarness, + buildLifecycleHookDeps, + type LifecycleDeps, +} from './lifecyclePerfFixtures.js'; +import type { + OperationLifecycleRegistry, + ObservableAgentEvent, +} from '../operationLifecycle.js'; + +// ─── Module mocks ─────────────────────────────────────────────────────────── + +void vi.mock('../useStreamEventHandlers.js', () => ({ + useStreamEventHandlers: () => ({ + processStreamEvent: vi.fn(), + displayUserMessage: vi.fn(), + prepareQueryForAgent: vi.fn().mockResolvedValue({ + queryToSend: 'test-query', + shouldProceed: true, + }), + handleLoopDetectedEvent: vi.fn(), + }), +})); + +void vi.mock('../../../contexts/SessionContext.js', () => ({ + useSessionStats: () => ({ + startNewPrompt: vi.fn(), + getPromptCount: () => 0, + }), +})); + +void vi.mock('../turnPreparation.js', () => ({ + prepareTurnForQuery: vi.fn().mockResolvedValue(undefined), +})); + +void vi.mock('../streamUtils.js', () => ({ + handleSubmissionError: vi.fn(), + processSlashCommandResult: vi.fn(), +})); + +// dispatchAgentEvent is called inside processAgentEvent; mock it so terminal +// events release the turn gate without requiring full event-handler wiring. +void vi.mock('../agentEventDispatcher.js', () => ({ + dispatchAgentEvent: vi.fn(() => ({ agentMessageBuffer: '' })), +})); + +// ─── Harness ──────────────────────────────────────────────────────────────── + +const harness = createLifecyclePerfHarness(); +let registry: OperationLifecycleRegistry; + +function renderUseSubmitQuery(deps: LifecycleDeps) { + return renderHook(() => + useSubmitQuery(buildLifecycleHookDeps(deps, harness.registry)), + ); +} + +function drainAndRead() { + return harness.drainAndRead(); +} + +// ─── Tests ────────────────────────────────────────────────────────────────── + +describe('useSubmitQuery — P07 granular cancellation classification (AC-4)', () => { + beforeEach(async () => { + await harness.setup(); + registry = harness.registry; + }); + + afterEach(async () => { + await harness.cleanup(); + }); + + it('classifies as "cancelled_during_api" when the signal aborts during streaming (default phase)', async () => { + const turnDeferred = createDeferred(); + const deps = createLifecycleDeps({ + runStreamRef: { + current: vi.fn().mockReturnValueOnce(turnDeferred.promise), + } as never, + }); + + const { result } = renderUseSubmitQuery(deps); + + // Start turn (blocks on deferred runStream). + let turnPromise!: Promise; + await act(async () => { + turnPromise = result.current.submitQuery( + 'hello', + undefined, + 'sess-1#agentic-loop#uuid-capi', + ); + }); + + await waitFor(() => expect(deps.abortControllerRef.current).not.toBeNull()); + + // Abort the signal during the API/streaming phase (no tool-status event). + deps.abortControllerRef.current!.abort(); + // Reject the deferred to settle runStream. + const abortError = new DOMException('Aborted', 'AbortError'); + await act(async () => { + turnDeferred.reject(abortError); + await turnPromise.catch(() => {}); + }); + + const records = await drainAndRead(); + expect(records).toHaveLength(1); + expect(records[0].status).toBe('cancelled_during_api'); + expect(records[0].operation_id).toBe('sess-1#agentic-loop#uuid-capi'); + }); + + it('classifies as "cancelled_during_tool" when the signal aborts during a tool phase', async () => { + const turnDeferred = createDeferred(); + const deps = createLifecycleDeps({ + runStreamRef: { + current: vi.fn().mockReturnValueOnce(turnDeferred.promise), + } as never, + }); + + const { result } = renderUseSubmitQuery(deps); + + let turnPromise!: Promise; + await act(async () => { + turnPromise = result.current.submitQuery( + 'hello', + undefined, + 'sess-1#agentic-loop#uuid-ctool', + ); + }); + + await waitFor(() => expect(deps.abortControllerRef.current).not.toBeNull()); + const signal = deps.abortControllerRef.current!.signal; + + // Route a tool-status 'executing' event through the registry's real + // event-observation entry point (the path the orchestration wires via + // onAgentEventObserved, OUTSIDE the generic event-handler catch — D8). + act(() => { + registry.observeAgentEvent( + { + type: 'tool-status', + update: { id: 'c1', name: 'shell', status: 'executing' }, + } as never, + signal, + 0, + ); + }); + + // Abort during the tool phase. + deps.abortControllerRef.current!.abort(); + const abortError = new DOMException('Aborted', 'AbortError'); + await act(async () => { + turnDeferred.reject(abortError); + await turnPromise.catch(() => {}); + }); + + const records = await drainAndRead(); + expect(records).toHaveLength(1); + expect(records[0].status).toBe('cancelled_during_tool'); + }); + + it('classifies as "cancelled_during_approval" when the signal aborts during an approval phase', async () => { + const turnDeferred = createDeferred(); + const deps = createLifecycleDeps({ + runStreamRef: { + current: vi.fn().mockReturnValueOnce(turnDeferred.promise), + } as never, + }); + + const { result } = renderUseSubmitQuery(deps); + + let turnPromise!: Promise; + await act(async () => { + turnPromise = result.current.submitQuery( + 'hello', + undefined, + 'sess-1#agentic-loop#uuid-cappr', + ); + }); + + await waitFor(() => expect(deps.abortControllerRef.current).not.toBeNull()); + const signal = deps.abortControllerRef.current!.signal; + + // Route a tool-status 'awaiting-approval' event through the registry's + // real event-observation entry point (outside the generic catch — D8). + act(() => { + registry.observeAgentEvent( + { + type: 'tool-status', + update: { id: 'c1', status: 'awaiting-approval' }, + } satisfies ObservableAgentEvent, + signal, + 0, + ); + }); + + // Abort during the approval phase. + deps.abortControllerRef.current!.abort(); + const abortError = new DOMException('Aborted', 'AbortError'); + await act(async () => { + turnDeferred.reject(abortError); + await turnPromise.catch(() => {}); + }); + + const records = await drainAndRead(); + expect(records).toHaveLength(1); + expect(records[0].status).toBe('cancelled_during_approval'); + }); + + it('deterministic precedence: approval > tool > api when overlapping phases occur', async () => { + const turnDeferred = createDeferred(); + const deps = createLifecycleDeps({ + runStreamRef: { + current: vi.fn().mockReturnValueOnce(turnDeferred.promise), + } as never, + }); + + const { result } = renderUseSubmitQuery(deps); + + let turnPromise!: Promise; + await act(async () => { + turnPromise = result.current.submitQuery( + 'hello', + undefined, + 'sess-1#agentic-loop#uuid-prec', + ); + }); + + await waitFor(() => expect(deps.abortControllerRef.current).not.toBeNull()); + const signal = deps.abortControllerRef.current!.signal; + + // Enter tool phase first, then approval phase (higher precedence). Both + // routed through the registry's real event-observation entry point. + act(() => { + registry.observeAgentEvent( + { + type: 'tool-status', + update: { id: 'c1', status: 'executing' }, + } satisfies ObservableAgentEvent, + signal, + 0, + ); + }); + act(() => { + registry.observeAgentEvent( + { + type: 'tool-status', + update: { id: 'c2', status: 'awaiting-approval' }, + } satisfies ObservableAgentEvent, + signal, + 0, + ); + }); + + // Abort: precedence says approval wins. + deps.abortControllerRef.current!.abort(); + const abortError = new DOMException('Aborted', 'AbortError'); + await act(async () => { + turnDeferred.reject(abortError); + await turnPromise.catch(() => {}); + }); + + const records = await drainAndRead(); + expect(records).toHaveLength(1); + expect(records[0].status).toBe('cancelled_during_approval'); + }); + + it('classifies as "cancelled_during_tool" when a retained tool-status cancelled event holds evidence', async () => { + // This exercises the retainedCancellationEvidence path (set ONLY by a + // tool-status 'cancelled' event), which persists past finalise and wins + // over the live-phase fallback. Even though the tool call is removed from + // activeToolCallIds by the 'cancelled' handler, the retained evidence + // ensures classifyCancellation returns cancelled_during_tool. + const turnDeferred = createDeferred(); + const deps = createLifecycleDeps({ + runStreamRef: { + current: vi.fn().mockReturnValueOnce(turnDeferred.promise), + } as never, + }); + + const { result } = renderUseSubmitQuery(deps); + + let turnPromise!: Promise; + await act(async () => { + turnPromise = result.current.submitQuery( + 'hello', + undefined, + 'sess-1#agentic-loop#uuid-rcancel', + ); + }); + + await waitFor(() => expect(deps.abortControllerRef.current).not.toBeNull()); + const signal = deps.abortControllerRef.current!.signal; + + // A tool enters the executing phase, then is cancelled (terminal 'cancelled' + // status). This retains tool-phase cancellation evidence and closes the + // active tool call — so at abort time activeToolCallIds is empty but the + // retained evidence must still win. + act(() => { + registry.observeAgentEvent( + { + type: 'tool-status', + update: { id: 'c1', status: 'executing' }, + } satisfies ObservableAgentEvent, + signal, + 0, + ); + }); + act(() => { + registry.observeAgentEvent( + { + type: 'tool-status', + update: { id: 'c1', status: 'cancelled' }, + } satisfies ObservableAgentEvent, + signal, + 0, + ); + }); + + // Abort AFTER the tool phase has already closed. Retained evidence must + // still classify as cancelled_during_tool (not the fallback during_api). + deps.abortControllerRef.current!.abort(); + const abortError = new DOMException('Aborted', 'AbortError'); + await act(async () => { + turnDeferred.reject(abortError); + await turnPromise.catch(() => {}); + }); + + const records = await drainAndRead(); + expect(records).toHaveLength(1); + expect(records[0].status).toBe('cancelled_during_tool'); + }); +}); diff --git a/packages/cli/src/ui/hooks/agentStream/__tests__/useSubmitQuery.lifecycle.failfast.bun.tsx b/packages/cli/src/ui/hooks/agentStream/__tests__/useSubmitQuery.lifecycle.failfast.bun.tsx new file mode 100644 index 0000000000..609376634f --- /dev/null +++ b/packages/cli/src/ui/hooks/agentStream/__tests__/useSubmitQuery.lifecycle.failfast.bun.tsx @@ -0,0 +1,637 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +// Enable React's act() environment so hook state updates are flushed. +( + globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean } +).IS_REACT_ACT_ENVIRONMENT = true; + +/** + * D8 fail-fast instrumentation-error integration tests (AC-8) and dual-failure + * AggregateError tests (AC-4), split from useSubmitQuery.lifecycle.test.tsx. + * + * Uses a real OperationLifecycleRegistry + a failing PerfSink (non-errno + * internal append error) writing to temp files. Proves that internal + * instrumentation errors propagate (fail-fast) rather than being silently + * debug-logged, and that when BOTH the provider path and finalisation fail an + * AggregateError preserves both errors in exact order. Mocks are limited to + * external boundaries (runStream, prepareQueryForAgent, event handlers) — the + * lifecycle, sink, and retention are real. + */ + +import { describe, it, expect, vi, beforeEach, afterEach } from 'bun:test'; +import { act } from 'react'; +import { renderHook, waitFor } from '../../../../test-utils/render.js'; +import { useSubmitQuery } from '../useSubmitQuery.js'; +import { + PerfSink, + PerfRetention, +} from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import type { PerfSinkFilesystem } from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import { readPerfRecords } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import { + OperationLifecycleRegistry, + type OperationIdentityProvider, +} from '../operationLifecycle.js'; +import { mkdtemp, rm, readdir, access, mkdir, open } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { createDeferred } from './createDeferred.js'; +import { + createLifecycleDeps, + buildLifecycleHookDeps, + fixtureIdentity, + type LifecycleDeps, +} from './lifecyclePerfFixtures.js'; + +const INTERNAL_ERROR_MESSAGE = 'internal append corruption'; + +// ─── Module mocks ─────────────────────────────────────────────────────────── + +// Controllable prepareQueryForAgent / displayUserMessage so individual tests +// can exercise error/cancellation paths without re-mocking the module. +let shouldProceedValue = true; +let queryToSendValue: string | null = 'test-query'; +// When non-null, prepareQueryForAgent rejects with this error. +let prepareQueryReject: unknown | null = null; +// When non-null, prepareTurnForQuery rejects with this error. +let prepareTurnReject: unknown | null = null; +// When non-null, displayUserMessage throws this error. +let displayUserMessageThrowValue: unknown | null = null; + +void vi.mock('../useStreamEventHandlers.js', () => ({ + useStreamEventHandlers: () => ({ + processStreamEvent: vi.fn(), + displayUserMessage: vi.fn().mockImplementation((_q: string, _t: number) => { + if (displayUserMessageThrowValue !== null) { + throw displayUserMessageThrowValue; + } + }), + prepareQueryForAgent: vi.fn().mockImplementation(() => { + if (prepareQueryReject !== null) { + return Promise.reject(prepareQueryReject); + } + return Promise.resolve({ + queryToSend: queryToSendValue, + shouldProceed: shouldProceedValue, + }); + }), + handleLoopDetectedEvent: vi.fn(), + }), +})); + +void vi.mock('../../../contexts/SessionContext.js', () => ({ + useSessionStats: () => ({ + startNewPrompt: vi.fn(), + getPromptCount: () => 0, + }), +})); + +void vi.mock('../turnPreparation.js', () => ({ + prepareTurnForQuery: vi.fn().mockImplementation(() => { + if (prepareTurnReject !== null) { + return Promise.reject(prepareTurnReject); + } + return Promise.resolve(undefined); + }), +})); + +void vi.mock('../streamUtils.js', () => ({ + handleSubmissionError: vi.fn(), + processSlashCommandResult: vi.fn(), +})); + +// dispatchAgentEvent is called inside processAgentEvent; mock it so terminal +// events release the turn gate without requiring full event-handler wiring. +void vi.mock('../agentEventDispatcher.js', () => ({ + dispatchAgentEvent: vi.fn(() => ({ agentMessageBuffer: '' })), +})); + +// ─── Failing-sink fixtures ────────────────────────────────────────────────── + +/** + * PerfSinkFilesystem whose appendFile throws a non-errno (internal) error. + * PerfSink rethrows non-errno errors (fail-fast), so sink.write rejects. + * ensureDir/openExclusive delegate to real fs so the claim file is created. + */ +class InternalErrorSinkFilesystem implements PerfSinkFilesystem { + async ensureDir(d: string): Promise { + try { + await access(d); + } catch { + await mkdir(d, { recursive: true, mode: 0o700 }); + } + } + async openExclusive(filePath: string, mode: number): Promise { + const handle = await open(filePath, 'wx', mode); + await handle.close(); + } + async appendFile(): Promise { + throw new Error(INTERNAL_ERROR_MESSAGE); + } +} + +interface FailingRegistry { + registry: OperationLifecycleRegistry; + sink: PerfSink; + perfDir: string; +} + +async function createFailingRegistry(): Promise { + const failDir = await mkdtemp(join(tmpdir(), 'perf-d8-')); + const failUuid = crypto.randomUUID(); + let failSink: PerfSink | null = null; + try { + const failRetention = new PerfRetention({ + dir: failDir, + runUuid: failUuid, + }); + failSink = new PerfSink({ + dir: failDir, + runUuid: failUuid, + retention: failRetention, + fs: new InternalErrorSinkFilesystem(), + }); + await failSink.start(); + const provider: OperationIdentityProvider = { + snapshot: () => fixtureIdentity(), + }; + const failRegistry = new OperationLifecycleRegistry({ + identityProvider: provider, + sink: failSink, + retention: failRetention, + }); + return { registry: failRegistry, sink: failSink, perfDir: failDir }; + } catch (err) { + const cleanupErrors: unknown[] = []; + if (failSink !== null) { + try { + await failSink.dispose(); + } catch (e) { + cleanupErrors.push(e); + } + } + try { + await rm(failDir, { recursive: true, force: true }); + } catch (e) { + cleanupErrors.push(e); + } + if (cleanupErrors.length === 1) { + throw new AggregateError( + [err, cleanupErrors[0]], + 'setup cleanup also failed', + ); + } + if (cleanupErrors.length > 1) { + throw new AggregateError( + [err, ...cleanupErrors], + 'setup cleanup also failed', + ); + } + throw err; + } +} + +// ─── Per-test failing-sink cleanup ────────────────────────────────────────── + +/** + * Tracks the failing sink created by each test so afterEach can independently + * attempt disposal and directory removal. The sink's dispose is expected to + * reject (retained internal instrumentation error); in the normal path the test + * body asserts that rejection and sets bodyDisposed so afterEach skips + * redundant disposal. If the body threw early, afterEach disposes — observing + * the expected rejection locally so it does not become unhandled — and surfaces + * only unexpected errors. + */ +interface FailCtx { + sink: PerfSink; + perfDir: string; + bodyDisposed: boolean; +} + +let failCtx: FailCtx | null = null; + +// ─── Helpers ──────────────────────────────────────────────────────────────── + +function renderWithRegistry( + deps: LifecycleDeps, + registry: OperationLifecycleRegistry, +) { + return renderHook(() => + useSubmitQuery(buildLifecycleHookDeps(deps, registry)), + ); +} + +// ─── Tests ────────────────────────────────────────────────────────────────── + +describe('useSubmitQuery — D8 fail-fast + dual-failure AggregateError (AC-4, AC-8)', () => { + beforeEach(() => { + shouldProceedValue = true; + queryToSendValue = 'test-query'; + prepareQueryReject = null; + prepareTurnReject = null; + displayUserMessageThrowValue = null; + failCtx = null; + }); + + afterEach(async () => { + const ctx = failCtx; + failCtx = null; + if (ctx === null) return; + const errors: unknown[] = []; + if (!ctx.bodyDisposed) { + // The test body did not reach its own disposal assertion. Dispose is + // expected to reject with the internal error; observe it locally and + // surface only unexpected errors. + try { + await ctx.sink.dispose(); + errors.push(new Error('failing sink dispose unexpectedly resolved')); + } catch (err) { + if (!(err instanceof Error) || err.message !== INTERNAL_ERROR_MESSAGE) { + errors.push(err); + } + } + } + try { + await rm(ctx.perfDir, { recursive: true, force: true }); + } catch (err) { + errors.push(err); + } + if (errors.length === 1) throw errors[0]; + if (errors.length > 1) { + throw new AggregateError(errors, 'failing-sink cleanup failed'); + } + }); + + // ------------------------------------------------------------------------- + // D8: finalisation is awaited; internal instrumentation errors fail-fast + // rather than being silently debug-logged (AC-8). Only genuinely external + // filesystem errno errors fail-open inside PerfSink/retention. + // ------------------------------------------------------------------------- + + it('rejects (fail-fast) when sink throws a non-errno internal error on the completed path', async () => { + const { + registry: failRegistry, + sink: failSink, + perfDir: failDir, + } = await createFailingRegistry(); + failCtx = { sink: failSink, perfDir: failDir, bodyDisposed: false }; + + const deps = createLifecycleDeps({ + runStreamRef: { + current: vi.fn().mockResolvedValue(undefined), + } as never, + }); + + const { result } = renderWithRegistry(deps, failRegistry); + + // submitQuery must reject — the instrumentation internal error is NOT + // silently debug-logged and swallowed. + await act(async () => { + let caught: unknown = undefined; + try { + await result.current.submitQuery( + 'hello world', + undefined, + 'sess-1#agentic-loop#uuid-d8', + ); + } catch (err) { + caught = err; + } + expect(caught).toBeInstanceOf(Error); + expect((caught as Error).message).toBe(INTERNAL_ERROR_MESSAGE); + }); + + // No records on disk — the write failed (the file may exist from + // openExclusive but contains no appended records). The drain and dispose + // promises each carry the retained internal rejection; observe them with + // local no-op handlers, then await/assert both. + const drainPromise = failRegistry.drain(); + drainPromise.catch(() => {}); + const disposePromise = failSink.dispose(); + disposePromise.catch(() => {}); + await expect(drainPromise).rejects.toThrow(INTERNAL_ERROR_MESSAGE); + await expect(disposePromise).rejects.toThrow(INTERNAL_ERROR_MESSAGE); + failCtx.bodyDisposed = true; + + const names = await readdir(failDir); + let recordCount = 0; + for (const name of names) { + if (!name.endsWith('.jsonl')) continue; + const fileResult = await readPerfRecords(join(failDir, name)); + recordCount += fileResult.records.filter( + (r) => r.record_type === 'operation', + ).length; + } + expect(recordCount).toBe(0); + }); + + it('handles the original provider error AND fails-fast when both stream and finalise fail', async () => { + const { + registry: failRegistry, + sink: failSink, + perfDir: failDir, + } = await createFailingRegistry(); + failCtx = { sink: failSink, perfDir: failDir, bodyDisposed: false }; + + // handleSubmissionError is mocked at the module level; import it to verify + // it was called with the ORIGINAL provider error. + const streamUtils = await import('../streamUtils.js'); + const handleSubmissionErrorMock = + streamUtils.handleSubmissionError as unknown as { + mockClear: () => void; + mock: { calls: unknown[][] }; + }; + handleSubmissionErrorMock.mockClear(); + + const streamError = new Error('provider stream failed'); + + const deps = createLifecycleDeps({ + runStreamRef: { + current: vi.fn().mockRejectedValue(streamError), + } as never, + }); + + const { result } = renderWithRegistry(deps, failRegistry); + + // submitQuery must reject with the INSTRUMENTATION error (fail-fast), + // NOT the provider error — proving the instrumentation error is not + // routed through or replaced by user-facing provider-error handling. + await act(async () => { + await expect( + result.current.submitQuery( + 'hello world', + undefined, + 'sess-1#agentic-loop#uuid-d8-err', + ), + ).rejects.toThrow(INTERNAL_ERROR_MESSAGE); + }); + + // The ORIGINAL provider error was handled for the user + // (handleSubmissionError called with streamError), proving the provider + // error is not lost. + expect(handleSubmissionErrorMock.mock.calls.length).toBeGreaterThanOrEqual( + 1, + ); + const firstCallFirstArg = handleSubmissionErrorMock.mock.calls[0][0]; + expect(firstCallFirstArg).toBe(streamError); + + await expect(failSink.dispose()).rejects.toThrow(INTERNAL_ERROR_MESSAGE); + failCtx.bodyDisposed = true; + }); + + it('propagates the finalisation error (fail-fast) on the no-proceed path when sink fails', async () => { + shouldProceedValue = false; + queryToSendValue = null; + + const { + registry: failRegistry, + sink: failSink, + perfDir: failDir, + } = await createFailingRegistry(); + failCtx = { sink: failSink, perfDir: failDir, bodyDisposed: false }; + + const deps = createLifecycleDeps({ + runStreamRef: { current: vi.fn() } as never, + }); + + const { result } = renderWithRegistry(deps, failRegistry); + + await act(async () => { + await expect( + result.current.submitQuery( + 'hello world', + undefined, + 'sess-1#agentic-loop#uuid-noproceed-failfast', + ), + ).rejects.toThrow(INTERNAL_ERROR_MESSAGE); + }); + + await expect(failSink.dispose()).rejects.toThrow(INTERNAL_ERROR_MESSAGE); + failCtx.bodyDisposed = true; + }); + + // ------------------------------------------------------------------------- + // D8 gap: displayUserMessage throw + cancellation paths where finalisation + // also rejects → AggregateError preserves dual failures (AC-4, AC-8). + // ------------------------------------------------------------------------- + + it('AggregateError [display error, finalisation error] when displayUserMessage throws and finalise also fails', async () => { + displayUserMessageThrowValue = new Error('display failed'); + + const { + registry: failRegistry, + sink: failSink, + perfDir: failDir, + } = await createFailingRegistry(); + failCtx = { sink: failSink, perfDir: failDir, bodyDisposed: false }; + + const deps = createLifecycleDeps({ + runStreamRef: { current: vi.fn() } as never, + }); + + const { result } = renderWithRegistry(deps, failRegistry); + + let caught: unknown = undefined; + await act(async () => { + try { + await result.current.submitQuery( + 'hello world', + undefined, + 'sess-1#agentic-loop#uuid-display-aggr', + ); + } catch (err) { + caught = err; + } + }); + + expect(caught).toBeInstanceOf(AggregateError); + const aggregate = caught as AggregateError; + expect(aggregate.errors).toHaveLength(2); + expect((aggregate.errors[0] as Error).message).toBe('display failed'); + expect((aggregate.errors[1] as Error).message).toBe(INTERNAL_ERROR_MESSAGE); + + await expect(failSink.dispose()).rejects.toThrow(INTERNAL_ERROR_MESSAGE); + failCtx.bodyDisposed = true; + }); + + it('AggregateError [cancellation, finalisation] when query prep is cancelled and finalise also fails', async () => { + const cancelError = new DOMException('Aborted', 'AbortError'); + prepareQueryReject = cancelError; + + const { + registry: failRegistry, + sink: failSink, + perfDir: failDir, + } = await createFailingRegistry(); + failCtx = { sink: failSink, perfDir: failDir, bodyDisposed: false }; + + const deps = createLifecycleDeps({ + runStreamRef: { current: vi.fn() } as never, + }); + + const { result } = renderWithRegistry(deps, failRegistry); + + let caught: unknown = undefined; + await act(async () => { + try { + await result.current.submitQuery( + 'hello world', + undefined, + 'sess-1#agentic-loop#uuid-qprep-cancel-aggr', + ); + } catch (err) { + caught = err; + } + }); + + expect(caught).toBeInstanceOf(AggregateError); + const aggregate = caught as AggregateError; + expect(aggregate.errors).toHaveLength(2); + expect(aggregate.errors[0]).toBe(cancelError); + expect((aggregate.errors[1] as Error).message).toBe(INTERNAL_ERROR_MESSAGE); + + await expect(failSink.dispose()).rejects.toThrow(INTERNAL_ERROR_MESSAGE); + failCtx.bodyDisposed = true; + }); + + it('AggregateError [cancellation, finalisation] when turn prep is cancelled and finalise also fails', async () => { + const cancelError = new DOMException('Aborted', 'AbortError'); + prepareTurnReject = cancelError; + + const { + registry: failRegistry, + sink: failSink, + perfDir: failDir, + } = await createFailingRegistry(); + failCtx = { sink: failSink, perfDir: failDir, bodyDisposed: false }; + + const deps = createLifecycleDeps({ + runStreamRef: { current: vi.fn() } as never, + }); + + const { result } = renderWithRegistry(deps, failRegistry); + + let caught: unknown = undefined; + await act(async () => { + try { + await result.current.submitQuery( + 'hello world', + undefined, + 'sess-1#agentic-loop#uuid-tprep-cancel-aggr', + ); + } catch (err) { + caught = err; + } + }); + + expect(caught).toBeInstanceOf(AggregateError); + const aggregate = caught as AggregateError; + expect(aggregate.errors).toHaveLength(2); + expect(aggregate.errors[0]).toBe(cancelError); + expect((aggregate.errors[1] as Error).message).toBe(INTERNAL_ERROR_MESSAGE); + + await expect(failSink.dispose()).rejects.toThrow(INTERNAL_ERROR_MESSAGE); + failCtx.bodyDisposed = true; + }); + + it('AggregateError [cancellation, finalisation] when stream is cancelled and finalise also fails', async () => { + const turnDeferred = createDeferred(); + + const { + registry: failRegistry, + sink: failSink, + perfDir: failDir, + } = await createFailingRegistry(); + failCtx = { sink: failSink, perfDir: failDir, bodyDisposed: false }; + + const deps = createLifecycleDeps({ + runStreamRef: { + current: vi.fn().mockReturnValueOnce(turnDeferred.promise), + } as never, + }); + + const { result } = renderWithRegistry(deps, failRegistry); + + let turnPromise!: Promise; + await act(async () => { + turnPromise = result.current.submitQuery( + 'hello', + undefined, + 'sess-1#agentic-loop#uuid-stream-cancel-aggr', + ); + }); + + await waitFor(() => expect(deps.abortControllerRef.current).not.toBeNull()); + + deps.abortControllerRef.current!.abort(); + const cancelError = new DOMException('Aborted', 'AbortError'); + + let caught: unknown = undefined; + await act(async () => { + turnDeferred.reject(cancelError); + try { + await turnPromise; + } catch (err) { + caught = err; + } + }); + + expect(caught).toBeInstanceOf(AggregateError); + const aggregate = caught as AggregateError; + expect(aggregate.errors).toHaveLength(2); + expect(aggregate.errors[0]).toBe(cancelError); + expect((aggregate.errors[1] as Error).message).toBe(INTERNAL_ERROR_MESSAGE); + + await expect(failSink.dispose()).rejects.toThrow(INTERNAL_ERROR_MESSAGE); + failCtx.bodyDisposed = true; + }); + + it('AggregateError [setup error, finalisation error] when post-begin setup and finalise both fail', async () => { + const setupError = new Error('committed-segment setup failed'); + const { + registry: failRegistry, + sink: failSink, + perfDir: failDir, + } = await createFailingRegistry(); + failCtx = { sink: failSink, perfDir: failDir, bodyDisposed: false }; + + const runStream = vi.fn(); + const deps = createLifecycleDeps({ + runStreamRef: { current: runStream } as never, + }); + vi.spyOn(deps.pendingResponse, 'beginCommittedSegments').mockImplementation( + () => { + throw setupError; + }, + ); + + const { result } = renderWithRegistry(deps, failRegistry); + let caught: unknown = undefined; + await act(async () => { + try { + await result.current.submitQuery( + 'hello world', + undefined, + 'sess-1#agentic-loop#uuid-setup-aggr', + ); + } catch (error) { + caught = error; + } + }); + + expect(caught).toBeInstanceOf(AggregateError); + const aggregate = caught as AggregateError; + expect(aggregate.errors).toHaveLength(2); + expect(aggregate.errors[0]).toBe(setupError); + expect((aggregate.errors[1] as Error).message).toBe(INTERNAL_ERROR_MESSAGE); + expect(deps.setIsRespondingCalls).toEqual([true, false]); + expect(runStream).not.toHaveBeenCalled(); + + await expect(failSink.dispose()).rejects.toThrow(INTERNAL_ERROR_MESSAGE); + failCtx.bodyDisposed = true; + }); +}); diff --git a/packages/cli/src/ui/hooks/agentStream/__tests__/useSubmitQuery.lifecycle.test.tsx b/packages/cli/src/ui/hooks/agentStream/__tests__/useSubmitQuery.lifecycle.test.tsx new file mode 100644 index 0000000000..f0fb13357c --- /dev/null +++ b/packages/cli/src/ui/hooks/agentStream/__tests__/useSubmitQuery.lifecycle.test.tsx @@ -0,0 +1,490 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +// Enable React's act() environment so hook state updates are flushed. +( + globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean } +).IS_REACT_ACT_ENVIRONMENT = true; + +/** + * Integration tests proving the OperationLifecycleRegistry is wired into the + * real useSubmitQuery turn path (AC-3, AC-4). Uses a real registry + real + * PerfSink/PerfRetention writing to temp files, and reads the records back + * through the real tolerant reader. Mocks are limited to external boundaries + * (runStream, prepareQueryForAgent, event handlers) — the lifecycle, sink, and + * retention are real. + */ + +import { describe, it, expect, vi, beforeEach, afterEach } from 'bun:test'; +import { act } from 'react'; +import { renderHook, waitFor } from '../../../../test-utils/render.js'; +import { useSubmitQuery, type UseSubmitQueryDeps } from '../useSubmitQuery.js'; +import { StreamingState, type HistoryItemWithoutId } from '../../../types.js'; +import { type RecordingIntegration } from '@vybestack/llxprt-code-core'; +import type { Agent } from '@vybestack/llxprt-code-agents'; +import { createStreamRuntimeForTest } from './streamRuntimeTestHelper.js'; +import { PendingResponseBuffer } from '../pendingResponseBuffer.js'; +import { createDeferred } from './createDeferred.js'; +import { + createLifecycleDeps, + createMockAgentClient, + createLifecyclePerfHarness, + buildLifecycleHookDeps, + type LifecycleDeps, +} from './lifecyclePerfFixtures.js'; + +// ─── Module mocks ─────────────────────────────────────────────────────────── + +// Controllable prepareQueryForAgent result so individual tests can exercise +// the pre-send-abort path without re-mocking the module. +let shouldProceedValue = true; +let queryToSendValue: string | null = 'test-query'; +// When non-null, prepareQueryForAgent rejects with this error (P06 prep gap). +let prepareQueryReject: unknown | null = null; +// When non-null, prepareTurnForQuery rejects with this error (P06 prep gap). +let prepareTurnReject: unknown | null = null; +// When non-null, displayUserMessage throws this error (D8 display-error gap). +let displayUserMessageThrowValue: unknown | null = null; + +void vi.mock('../useStreamEventHandlers.js', () => ({ + useStreamEventHandlers: () => ({ + processStreamEvent: vi.fn(), + displayUserMessage: vi.fn().mockImplementation((_q: string, _t: number) => { + if (displayUserMessageThrowValue !== null) { + throw displayUserMessageThrowValue; + } + }), + prepareQueryForAgent: vi.fn().mockImplementation(() => { + if (prepareQueryReject !== null) { + return Promise.reject(prepareQueryReject); + } + return Promise.resolve({ + queryToSend: queryToSendValue, + shouldProceed: shouldProceedValue, + }); + }), + handleLoopDetectedEvent: vi.fn(), + }), +})); + +void vi.mock('../../../contexts/SessionContext.js', () => ({ + useSessionStats: () => ({ + startNewPrompt: vi.fn(), + getPromptCount: () => 0, + }), +})); + +void vi.mock('../turnPreparation.js', () => ({ + prepareTurnForQuery: vi.fn().mockImplementation(() => { + if (prepareTurnReject !== null) { + return Promise.reject(prepareTurnReject); + } + return Promise.resolve(undefined); + }), +})); + +void vi.mock('../streamUtils.js', () => ({ + handleSubmissionError: vi.fn(), + processSlashCommandResult: vi.fn(), +})); + +// dispatchAgentEvent is called inside processAgentEvent; mock it so terminal +// events release the turn gate without requiring full event-handler wiring. +void vi.mock('../agentEventDispatcher.js', () => ({ + dispatchAgentEvent: vi.fn(() => ({ agentMessageBuffer: '' })), +})); + +// ─── Lifecycle perf harness ───────────────────────────────────────────────── + +const harness = createLifecyclePerfHarness(); + +function renderUseSubmitQuery(deps: LifecycleDeps) { + return renderHook(() => + useSubmitQuery(buildLifecycleHookDeps(deps, harness.registry)), + ); +} + +function drainAndRead() { + return harness.drainAndRead(); +} + +// ─── Tests ────────────────────────────────────────────────────────────────── + +describe('useSubmitQuery — operation lifecycle integration (AC-3, AC-4)', () => { + beforeEach(async () => { + await harness.setup(); + // Reset preparation rejection controls between tests. + prepareQueryReject = null; + prepareTurnReject = null; + shouldProceedValue = true; + queryToSendValue = 'test-query'; + displayUserMessageThrowValue = null; + }); + + afterEach(async () => { + await harness.cleanup(); + }); + + it('writes one record with status "completed" when a turn succeeds', async () => { + const deps = createLifecycleDeps({ + runStreamRef: { + current: vi.fn().mockResolvedValue(undefined), + } as never, + }); + + const { result } = renderUseSubmitQuery(deps); + + await act(async () => { + await result.current.submitQuery( + 'hello world', + undefined, + 'sess-1#agentic-loop#uuid-a', + ); + }); + + const records = await drainAndRead(); + expect(records).toHaveLength(1); + expect(records[0].status).toBe('completed'); + expect(records[0].operation_id).toBe('sess-1#agentic-loop#uuid-a'); + expect(records[0].session_operation_index).toBe(0); + }); + + it('writes one record with status "error" when runStream rejects', async () => { + const deps = createLifecycleDeps({ + runStreamRef: { + current: vi.fn().mockRejectedValue(new Error('stream failed')), + } as never, + }); + + const { result } = renderUseSubmitQuery(deps); + + await act(async () => { + await result.current.submitQuery( + 'hello world', + undefined, + 'sess-1#agentic-loop#uuid-err', + ); + }); + + const records = await drainAndRead(); + expect(records).toHaveLength(1); + expect(records[0].status).toBe('error'); + expect(records[0].operation_id).toBe('sess-1#agentic-loop#uuid-err'); + }); + + it('writes one record with status "cancelled_before_send" when the turn does not proceed', async () => { + shouldProceedValue = false; + queryToSendValue = null; + + const deps = createLifecycleDeps({ + runStreamRef: { current: vi.fn() } as never, + }); + + const { result } = renderUseSubmitQuery(deps); + + await act(async () => { + await result.current.submitQuery( + 'hello world', + undefined, + 'sess-1#agentic-loop#uuid-pre', + ); + }); + + const records = await drainAndRead(); + expect(records).toHaveLength(1); + expect(records[0].status).toBe('cancelled_before_send'); + }); + + it('finalises a displaced turn as "superseded" exactly once when a newer turn begins', async () => { + // Turn 1: blocks on a deferred runStream so it stays "active". + const turn1Deferred = createDeferred(); + const deps = createLifecycleDeps({ + runStreamRef: { + current: vi.fn().mockReturnValueOnce(turn1Deferred.promise), + } as never, + }); + + const { result } = renderUseSubmitQuery(deps); + + // Start turn 1. + let turn1Promise!: Promise; + await act(async () => { + turn1Promise = result.current.submitQuery( + 'turn one', + undefined, + 'sess-1#agentic-loop#uuid-1', + ); + }); + + // Wait until turn 1's AbortController is installed and isResponding is set. + await waitFor(() => expect(deps.abortControllerRef.current).not.toBeNull()); + await waitFor(() => + expect(deps.setIsRespondingCalls).toStrictEqual([true]), + ); + const turn1Signal = deps.abortControllerRef.current!.signal; + + // Release the interactive turn gate via a terminal event for turn 1. + // This sets activeTurnRef.current = false while runStream is still pending. + act(() => { + result.current.processAgentEvent( + { type: 'error', message: 'displaced' } as never, + Date.now(), + turn1Signal, + ); + }); + + await waitFor(() => + expect(deps.setIsRespondingCalls).toStrictEqual([true, false]), + ); + + // Turn 2: a new submitQuery call. Because activeTurnRef.current is false + // and streamingState is Idle, it proceeds immediately. Its begin() sweeps + // turn 1 as superseded. + await act(async () => { + await result.current.submitQuery( + 'turn two', + undefined, + 'sess-1#agentic-loop#uuid-2', + ); + }); + + // Settle turn 1's deferred (the stale turn's runStream finally settles, + // but isCurrentTurn is false so its guarded finally does nothing). + await act(async () => { + turn1Deferred.resolve(); + await turn1Promise.catch(() => {}); + }); + + const records = await drainAndRead(); + expect(records).toHaveLength(2); + const statuses = records.map((r) => r.status).sort(); + expect(statuses).toStrictEqual(['completed', 'superseded']); + const superseded = records.find((r) => r.status === 'superseded'); + expect(superseded?.operation_id).toBe('sess-1#agentic-loop#uuid-1'); + const completed = records.find((r) => r.status === 'completed'); + expect(completed?.operation_id).toBe('sess-1#agentic-loop#uuid-2'); + }); + + it('writes exactly one record per turn (exactly-once through real control flow)', async () => { + const deps = createLifecycleDeps({ + runStreamRef: { + current: vi.fn().mockResolvedValue(undefined), + } as never, + }); + + const { result } = renderUseSubmitQuery(deps); + + await act(async () => { + await result.current.submitQuery( + 'once only', + undefined, + 'sess-1#agentic-loop#uuid-once', + ); + }); + + const records = await drainAndRead(); + expect(records).toHaveLength(1); + expect(records[0].status).toBe('completed'); + expect(records[0].operation_id).toBe('sess-1#agentic-loop#uuid-once'); + }); + + it('produces no perf records when operationLifecycle is absent (perf disabled)', async () => { + // Create deps without a registry (simulating perf disabled). + const deps = createLifecycleDeps({ + runStreamRef: { + current: vi.fn().mockResolvedValue(undefined), + } as never, + }); + + const hookDeps: UseSubmitQueryDeps = { + runtime: createStreamRuntimeForTest(), + agent: createMockAgentClient() as unknown as Agent, + addItem: vi.fn().mockReturnValue(1), + settings: {} as never, + onDebugMessage: vi.fn(), + onCancelSubmit: vi.fn(), + setTurnCancelled: vi.fn(), + onAuthError: vi.fn(), + sanitizeContent: (text: string) => ({ text, blocked: false }), + flushPendingHistoryItem: vi.fn(), + pendingResponse: new PendingResponseBuffer(undefined), + pendingHistoryItemRef: { + current: null, + } as React.MutableRefObject, + thinkingBlocksRef: { current: [] }, + turnCancelledRef: { current: false }, + queuedSubmissionsRef: { current: [] }, + drainSuppressedRef: { current: false }, + enqueueSubmission: vi.fn(), + enqueueSubmissionFirst: vi.fn(), + requeueSubmission: vi.fn(), + dequeueSubmission: vi.fn(), + clearSubmissions: vi.fn(), + tryReserveDrain: vi.fn().mockReturnValue(true), + releaseDrain: vi.fn(), + setPendingHistoryItem: vi.fn(), + setIsResponding: vi.fn(), + setInitError: vi.fn(), + setThought: vi.fn(), + setLastAgentActivityTime: vi.fn(), + scheduleToolCalls: vi.fn(), + abortActiveStream: vi.fn(), + handleShellCommand: vi.fn().mockReturnValue(false), + handleSlashCommand: vi.fn().mockResolvedValue(false), + logger: null, + shellModeActive: false, + loopDetectedRef: { current: false }, + lastProfileNameRef: { current: undefined }, + lastModelInfoRef: { current: null }, + lastModelIdentityRef: { current: null }, + abortControllerRef: deps.abortControllerRef, + runStreamRef: deps.runStreamRef, + submitQueryRef: { current: null }, + isResponding: false, + streamingState: StreamingState.Idle, + recordingIntegration: { + flushAtTurnBoundary: vi.fn(), + } as unknown as RecordingIntegration, + // operationLifecycle intentionally omitted — perf disabled. + }; + + const { result } = renderHook(() => useSubmitQuery(hookDeps)); + + await act(async () => { + await result.current.submitQuery( + 'disabled', + undefined, + 'sess-1#agentic-loop#uuid-dis', + ); + }); + + // drainAndRead disposes the harness sink; the disabled turn wrote no + // records because operationLifecycle was omitted. + const records = await drainAndRead(); + expect(records).toHaveLength(0); + }); + + // ------------------------------------------------------------------------- + // P06: preparation rejections must finalise as 'error' exactly once + // (AC-4). The original rejection is preserved to the caller/UI. + // ------------------------------------------------------------------------- + + it('finalises as "error" when prepareQueryForAgent rejects (preserves rejection)', async () => { + const prepError = new Error('query preparation failed'); + prepareQueryReject = prepError; + + const deps = createLifecycleDeps({ + runStreamRef: { current: vi.fn() } as never, + }); + + const { result } = renderUseSubmitQuery(deps); + + // The original rejection must propagate to the caller (not swallowed or + // replaced by an instrumentation error). + await act(async () => { + await expect( + result.current.submitQuery( + 'hello world', + undefined, + 'sess-1#agentic-loop#uuid-prep-rej', + ), + ).rejects.toThrow('query preparation failed'); + }); + + // The op must be finalised exactly once as 'error'. + const records = await drainAndRead(); + expect(records).toHaveLength(1); + expect(records[0].status).toBe('error'); + expect(records[0].operation_id).toBe('sess-1#agentic-loop#uuid-prep-rej'); + }); + + it('finalises as "error" when prepareTurnForQuery rejects (preserves rejection)', async () => { + const turnError = new Error('turn preparation failed'); + prepareTurnReject = turnError; + + const deps = createLifecycleDeps({ + runStreamRef: { current: vi.fn() } as never, + }); + + const { result } = renderUseSubmitQuery(deps); + + await act(async () => { + await expect( + result.current.submitQuery( + 'hello world', + undefined, + 'sess-1#agentic-loop#uuid-turn-rej', + ), + ).rejects.toThrow('turn preparation failed'); + }); + + const records = await drainAndRead(); + expect(records).toHaveLength(1); + expect(records[0].status).toBe('error'); + expect(records[0].operation_id).toBe('sess-1#agentic-loop#uuid-turn-rej'); + }); + + it('finalises as "error" exactly once when displayUserMessage throws (working sink)', async () => { + displayUserMessageThrowValue = new Error('display failed'); + + const deps = createLifecycleDeps({ + runStreamRef: { + current: vi.fn().mockResolvedValue(undefined), + } as never, + }); + + const { result } = renderUseSubmitQuery(deps); + + await act(async () => { + await expect( + result.current.submitQuery( + 'hello world', + undefined, + 'sess-1#agentic-loop#uuid-display-throw', + ), + ).rejects.toThrow('display failed'); + }); + + const records = await drainAndRead(); + expect(records).toHaveLength(1); + expect(records[0].status).toBe('error'); + expect(records[0].operation_id).toBe( + 'sess-1#agentic-loop#uuid-display-throw', + ); + }); + + it('finalises once and clears responding state when post-begin setup throws', async () => { + const setupError = new Error('committed-segment setup failed'); + const runStream = vi.fn().mockResolvedValue(undefined); + const deps = createLifecycleDeps({ + runStreamRef: { current: runStream } as never, + }); + vi.spyOn(deps.pendingResponse, 'beginCommittedSegments').mockImplementation( + () => { + throw setupError; + }, + ); + + const { result } = renderUseSubmitQuery(deps); + + await act(async () => { + await expect( + result.current.submitQuery( + 'hello world', + undefined, + 'sess-1#agentic-loop#uuid-post-begin', + ), + ).rejects.toBe(setupError); + }); + + const records = await drainAndRead(); + expect(records).toHaveLength(1); + expect(records[0].status).toBe('error'); + expect(records[0].operation_id).toBe('sess-1#agentic-loop#uuid-post-begin'); + expect(deps.setIsRespondingCalls).toEqual([true, false]); + expect(runStream).not.toHaveBeenCalled(); + }); +}); diff --git a/packages/cli/src/ui/hooks/agentStream/operationLifecycle.behavior.test.ts b/packages/cli/src/ui/hooks/agentStream/operationLifecycle.behavior.test.ts new file mode 100644 index 0000000000..3b4d149fd4 --- /dev/null +++ b/packages/cli/src/ui/hooks/agentStream/operationLifecycle.behavior.test.ts @@ -0,0 +1,917 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Behavioral tests for the OperationLifecycleRegistry (P06, EVIDENCE-AC3/AC4). + * + * Real registry + real PerfSink/PerfRetention + real temp files + the real + * reader. No mocks. + * + * Covers: seven terminal statuses, duplicate finalise (exactly-once), + * finalise/supersede race, multiple claims including stale claim, exact D1 + * split rule, no child arrays, session index monotonic, schema failure + * fail-fast, filesystem sink fail-open. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { + PerfSink, + PerfRetention, +} from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import type { PerfSinkFilesystem } from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import { readPerfRecords } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import type { PerfOperationRecord } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import { promises as fsp } from 'node:fs'; +import { + OperationLifecycleRegistry, + type OperationIdentityProvider, + type OperationIdentitySnapshot, + type OperationStatus, +} from './operationLifecycle.js'; + +// --------------------------------------------------------------------------- +// Shared setup helper (RULES.md: no copy-pasted boilerplate) +// --------------------------------------------------------------------------- + +let dir: string; + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'op-lifecycle-')); +}); + +afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); +}); + +function fixtureIdentity( + overrides: Partial = {}, +): OperationIdentitySnapshot { + return { + session_id: 'sess-abc', + runtime_id: 'rt-main', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'sha256:project-hash', + llxprt_version: '0.11.0', + git_sha: 'abc1234', + runtime: 'bun-1.3.14', + platform: 'darwin-arm64', + provider: 'openai', + model: 'gpt-4o', + terminal_cols: 120, + terminal_rows: 40, + render_mode: 'incremental', + ...overrides, + }; +} + +function fixtureProvider( + overrides: Partial = {}, +): OperationIdentityProvider { + const snap = fixtureIdentity(overrides); + return { snapshot: () => snap }; +} + +/** + * Filesystem port that fails appendFile with EACCES to test fail-open (D6). + * Delegates directory/exclusive-open to the real fs so the claim is created. + */ +class FailingAppendFilesystem implements PerfSinkFilesystem { + async ensureDir(d: string): Promise { + try { + await fsp.access(d); + } catch { + await fsp.mkdir(d, { recursive: true, mode: 0o700 }); + } + } + + async openExclusive(filePath: string, mode: number): Promise { + const handle = await fsp.open(filePath, 'wx', mode); + await handle.close(); + } + + async appendFile(): Promise { + const err = new Error('EACCES') as NodeJS.ErrnoException; + err.code = 'EACCES'; + throw err; + } +} + +function createRegistry( + overrides: { + identity?: OperationIdentityProvider; + sink?: PerfSink; + retention?: PerfRetention; + wallNow?: () => number; + monotonicNow?: () => number; + } = {}, +): { + registry: OperationLifecycleRegistry; + sink: PerfSink; + retention: PerfRetention; +} { + const runUuid = crypto.randomUUID(); + const retention = + overrides.retention ?? + new PerfRetention({ + dir, + runUuid, + maintenanceIntervalMs: 60_000, + }); + const sink = + overrides.sink ?? + new PerfSink({ + dir, + runUuid, + retention, + }); + const registry = new OperationLifecycleRegistry({ + identityProvider: overrides.identity ?? fixtureProvider(), + sink, + retention, + wallNow: overrides.wallNow, + monotonicNow: overrides.monotonicNow, + }); + return { registry, sink, retention }; +} + +async function startAndCreate( + overrides: { + identity?: OperationIdentityProvider; + runUuid?: string; + sink?: PerfSink; + retention?: PerfRetention; + wallNow?: () => number; + monotonicNow?: () => number; + } = {}, +): Promise<{ + registry: OperationLifecycleRegistry; + sink: PerfSink; + retention: PerfRetention; +}> { + const runUuid = overrides.runUuid ?? crypto.randomUUID(); + const retention = + overrides.retention ?? + new PerfRetention({ + dir, + runUuid, + maintenanceIntervalMs: 60_000, + }); + const sink = + overrides.sink ?? + new PerfSink({ + dir, + runUuid, + retention, + }); + await sink.start(); + const { registry } = createRegistry({ + identity: overrides.identity, + sink, + retention, + wallNow: overrides.wallNow, + monotonicNow: overrides.monotonicNow, + }); + return { registry, sink, retention }; +} + +async function readAllRecords(): Promise { + const files = fs.existsSync(dir) + ? fs.readdirSync(dir).filter((f) => f.endsWith('.jsonl')) + : []; + const records: PerfOperationRecord[] = []; + for (const file of files) { + const result = await readPerfRecords(path.join(dir, file)); + for (const rec of result.records) { + if (rec.record_type === 'operation') { + records.push(rec); + } + } + } + return records; +} + +async function drainSink(sink: PerfSink): Promise { + await sink.dispose(); +} + +// --------------------------------------------------------------------------- +// D1: operation_id derivation through the lifecycle +// --------------------------------------------------------------------------- + +describe('OperationLifecycleRegistry — D1 operation_id (AC-3 split rule)', () => { + it('derives operation_id from an initial prompt id (no marker)', async () => { + const { registry, sink } = await startAndCreate(); + const ac = new AbortController(); + registry.begin(ac.signal, 'sess-1#agentic-loop#uuid-1'); + await registry.finalise(ac.signal, 'completed'); + await drainSink(sink); + + const records = await readAllRecords(); + expect(records).toHaveLength(1); + expect(records[0].operation_id).toBe('sess-1#agentic-loop#uuid-1'); + }); + + it('strips a continuation marker to the prefix', async () => { + const { registry, sink } = await startAndCreate(); + const ac = new AbortController(); + registry.begin(ac.signal, 'sess-1#agentic-loop#uuid-1#continuation#1'); + await registry.finalise(ac.signal, 'completed'); + await drainSink(sink); + + const records = await readAllRecords(); + expect(records).toHaveLength(1); + expect(records[0].operation_id).toBe('sess-1#agentic-loop#uuid-1'); + }); + + it('strips continuation #2 to the same prefix as #1', async () => { + const { registry, sink } = await startAndCreate(); + const ac1 = new AbortController(); + registry.begin(ac1.signal, 'sess-1#agentic-loop#uuid-1#continuation#2'); + await registry.finalise(ac1.signal, 'completed'); + await drainSink(sink); + + const records = await readAllRecords(); + expect(records[0].operation_id).toBe('sess-1#agentic-loop#uuid-1'); + }); + + it('takes the first segment for a non-terminal marker', async () => { + const { registry, sink } = await startAndCreate(); + const ac = new AbortController(); + registry.begin(ac.signal, 'sess-1#continuation#1#more'); + await registry.finalise(ac.signal, 'completed'); + await drainSink(sink); + + const records = await readAllRecords(); + expect(records[0].operation_id).toBe('sess-1'); + }); + + it('preserves a CLI-fallback id without the marker', async () => { + const { registry, sink } = await startAndCreate(); + const ac = new AbortController(); + registry.begin(ac.signal, 'test-session########0'); + await registry.finalise(ac.signal, 'completed'); + await drainSink(sink); + + const records = await readAllRecords(); + expect(records[0].operation_id).toBe('test-session########0'); + }); +}); + +// --------------------------------------------------------------------------- +// D1: no child arrays on the record +// --------------------------------------------------------------------------- + +describe('OperationLifecycleRegistry — D1 no child arrays', () => { + it('produces a record with no prompt_ids/turn_ids fields', async () => { + const { registry, sink } = await startAndCreate(); + const ac = new AbortController(); + registry.begin(ac.signal, 'sess-1#agentic-loop#uuid-1'); + await registry.finalise(ac.signal, 'completed'); + await drainSink(sink); + + const records = await readAllRecords(); + expect(records).toHaveLength(1); + const raw = JSON.parse( + fs.readFileSync( + fs + .readdirSync(dir) + .map((f) => path.join(dir, f)) + .find((f) => f.endsWith('.jsonl'))!, + 'utf8', + ), + ); + expect('prompt_ids' in raw).toBe(false); + expect('turn_ids' in raw).toBe(false); + expect('prompt_ids_total' in raw).toBe(false); + expect('turn_ids_total' in raw).toBe(false); + }); +}); + +// --------------------------------------------------------------------------- +// Seven terminal statuses (AC-4) +// --------------------------------------------------------------------------- + +describe('OperationLifecycleRegistry — seven terminal statuses (AC-4)', () => { + const statuses: OperationStatus[] = [ + 'completed', + 'error', + 'cancelled_before_send', + 'cancelled_during_api', + 'cancelled_during_tool', + 'cancelled_during_approval', + 'superseded', + ]; + + for (const status of statuses) { + it(`writes exactly one record with status "${status}"`, async () => { + const { registry, sink } = await startAndCreate(); + const ac = new AbortController(); + registry.begin(ac.signal, 'sess-1#agentic-loop#uuid-1'); + + if (status === 'superseded') { + // Superseded is written by the sweep, not explicit finalise. + const ac2 = new AbortController(); + registry.begin(ac2.signal, 'sess-1#agentic-loop#uuid-2'); + await registry.finalise(ac2.signal, 'completed'); + } else { + await registry.finalise(ac.signal, status); + } + await drainSink(sink); + + const records = await readAllRecords(); + const matching = records.filter((r) => r.status === status); + expect(matching).toHaveLength(1); + }); + } +}); + +// --------------------------------------------------------------------------- +// Exactly-once: duplicate finalise +// --------------------------------------------------------------------------- + +describe('OperationLifecycleRegistry — exactly-once duplicate finalise', () => { + it('writes exactly one record for a duplicate finalise', async () => { + const { registry, sink } = await startAndCreate(); + const ac = new AbortController(); + registry.begin(ac.signal, 'sess-1#agentic-loop#uuid-1'); + await registry.finalise(ac.signal, 'completed'); + await registry.finalise(ac.signal, 'completed'); + await registry.finalise(ac.signal, 'error'); + await drainSink(sink); + + const records = await readAllRecords(); + expect(records).toHaveLength(1); + expect(records[0].status).toBe('completed'); + }); + + it('finalise on an unknown signal is a no-op', async () => { + const { registry, sink } = await startAndCreate(); + const unknown = new AbortController(); + await registry.finalise(unknown.signal, 'completed'); + await drainSink(sink); + + const records = await readAllRecords(); + expect(records).toHaveLength(0); + }); + + it('late finalise after supersede sweep is a no-op', async () => { + const { registry, sink } = await startAndCreate(); + const ac1 = new AbortController(); + registry.begin(ac1.signal, 'sess-1#agentic-loop#uuid-1'); + // New begin sweeps ac1 as superseded. + const ac2 = new AbortController(); + registry.begin(ac2.signal, 'sess-1#agentic-loop#uuid-2'); + // Late explicit finalise of the swept signal — must not double-write. + await registry.finalise(ac1.signal, 'completed'); + await registry.finalise(ac2.signal, 'completed'); + await drainSink(sink); + + const records = await readAllRecords(); + expect(records).toHaveLength(2); + const statuses = records.map((r) => r.status).sort(); + expect(statuses).toEqual(['completed', 'superseded']); + }); +}); + +// --------------------------------------------------------------------------- +// Superseded sweep (AC-4) +// --------------------------------------------------------------------------- + +describe('OperationLifecycleRegistry — superseded sweep', () => { + it('finalises a displaced op as superseded when a new begin occurs', async () => { + const { registry, sink } = await startAndCreate(); + const ac1 = new AbortController(); + const handle1 = registry.begin(ac1.signal, 'sess-1#agentic-loop#uuid-1'); + expect(handle1.operationId).toBe('sess-1#agentic-loop#uuid-1'); + + const ac2 = new AbortController(); + const handle2 = registry.begin(ac2.signal, 'sess-1#agentic-loop#uuid-2'); + expect(handle2.operationId).toBe('sess-1#agentic-loop#uuid-2'); + + await registry.finalise(ac2.signal, 'completed'); + await drainSink(sink); + + const records = await readAllRecords(); + expect(records).toHaveLength(2); + const superseded = records.filter((r) => r.status === 'superseded'); + const completed = records.filter((r) => r.status === 'completed'); + expect(superseded).toHaveLength(1); + expect(completed).toHaveLength(1); + expect(superseded[0].operation_id).toBe('sess-1#agentic-loop#uuid-1'); + expect(completed[0].operation_id).toBe('sess-1#agentic-loop#uuid-2'); + }); + + it('finalises multiple displaced ops as superseded', async () => { + const { registry, sink } = await startAndCreate(); + const ac1 = new AbortController(); + registry.begin(ac1.signal, 'sess-1#agentic-loop#uuid-1'); + const ac2 = new AbortController(); + registry.begin(ac2.signal, 'sess-1#agentic-loop#uuid-2'); + const ac3 = new AbortController(); + registry.begin(ac3.signal, 'sess-1#agentic-loop#uuid-3'); + await registry.finalise(ac3.signal, 'completed'); + await drainSink(sink); + + const records = await readAllRecords(); + expect(records).toHaveLength(3); + const superseded = records.filter((r) => r.status === 'superseded'); + expect(superseded).toHaveLength(2); + }); + + it('avoids writing two records when explicit finalise races with supersede', async () => { + const { registry, sink } = await startAndCreate(); + const ac1 = new AbortController(); + registry.begin(ac1.signal, 'sess-1#agentic-loop#uuid-1'); + + // Simulate the race: explicit finalise AND supersede sweep. + // The sweep claims synchronously in begin, so the explicit finalise + // finds the signal already finalised and no-ops. + const ac2 = new AbortController(); + registry.begin(ac2.signal, 'sess-1#agentic-loop#uuid-2'); + // This late finalise must no-op (exactly-once). + await registry.finalise(ac1.signal, 'completed'); + await registry.finalise(ac2.signal, 'completed'); + await drainSink(sink); + + const records = await readAllRecords(); + // Exactly 2: one superseded (sweep), one completed (ac2). + expect(records).toHaveLength(2); + const op1Records = records.filter( + (r) => r.operation_id === 'sess-1#agentic-loop#uuid-1', + ); + expect(op1Records).toHaveLength(1); + expect(op1Records[0].status).toBe('superseded'); + }); +}); + +// --------------------------------------------------------------------------- +// concurrent_instances from claims (D3) +// --------------------------------------------------------------------------- + +describe('OperationLifecycleRegistry — concurrent_instances (D3)', () => { + it('includes the own claim in concurrent_instances (minimum 1)', async () => { + const { registry, sink } = await startAndCreate(); + const ac = new AbortController(); + registry.begin(ac.signal, 'sess-1#agentic-loop#uuid-1'); + await registry.finalise(ac.signal, 'completed'); + await drainSink(sink); + + const records = await readAllRecords(); + expect(records[0].concurrent_instances).toBeGreaterThanOrEqual(1); + }); + + it('counts a stale claim from a prior run as non-stale within the lease', async () => { + // Create a prior run's claim file (non-stale). + const priorClaim = path.join(dir, 'prior-run-uuid.claim'); + fs.writeFileSync(priorClaim, '', { mode: 0o600 }); + + const { registry, sink } = await startAndCreate(); + const ac = new AbortController(); + registry.begin(ac.signal, 'sess-1#agentic-loop#uuid-1'); + await registry.finalise(ac.signal, 'completed'); + await drainSink(sink); + + const records = await readAllRecords(); + // At least 2: own claim + prior claim (both fresh/non-stale). + expect(records[0].concurrent_instances).toBeGreaterThanOrEqual(2); + }); + + it('does not count a stale claim beyond the lease window', async () => { + // Create a stale claim (mtime far in the past). + const staleClaim = path.join(dir, 'stale-run-uuid.claim'); + fs.writeFileSync(staleClaim, '', { mode: 0o600 }); + const staleTime = new Date(Date.now() - 600_000); // 10 min ago > 180s lease + fs.utimesSync(staleClaim, staleTime, staleTime); + + const { registry, sink } = await startAndCreate(); + const ac = new AbortController(); + registry.begin(ac.signal, 'sess-1#agentic-loop#uuid-1'); + await registry.finalise(ac.signal, 'completed'); + await drainSink(sink); + + const records = await readAllRecords(); + // Only own claim counts (stale one excluded). + expect(records[0].concurrent_instances).toBe(1); + }); +}); + +// --------------------------------------------------------------------------- +// Session index monotonic +// --------------------------------------------------------------------------- + +describe('OperationLifecycleRegistry — session index monotonic', () => { + it('assigns sequential indices starting at 0', async () => { + const { registry, sink } = await startAndCreate(); + const ac1 = new AbortController(); + const h1 = registry.begin(ac1.signal, 'sess-1#agentic-loop#uuid-1'); + await registry.finalise(ac1.signal, 'completed'); + + const ac2 = new AbortController(); + const h2 = registry.begin(ac2.signal, 'sess-1#agentic-loop#uuid-2'); + await registry.finalise(ac2.signal, 'completed'); + + const ac3 = new AbortController(); + const h3 = registry.begin(ac3.signal, 'sess-1#agentic-loop#uuid-3'); + await registry.finalise(ac3.signal, 'completed'); + + await drainSink(sink); + + expect(h1.sessionOperationIndex).toBe(0); + expect(h2.sessionOperationIndex).toBe(1); + expect(h3.sessionOperationIndex).toBe(2); + + const records = await readAllRecords(); + const indices = records + .map((r) => r.session_operation_index) + .sort((a, b) => a - b); + expect(indices).toEqual([0, 1, 2]); + }); +}); + +// --------------------------------------------------------------------------- +// Measurement handle (P07 seam) +// --------------------------------------------------------------------------- + +describe('OperationLifecycleRegistry — measurement handle', () => { + it('exposes a mutable measurement starting at zero', async () => { + const { registry, sink } = await startAndCreate(); + const ac = new AbortController(); + const handle = registry.begin(ac.signal, 'sess-1#agentic-loop#uuid-1'); + expect(handle.measurement.client_prepare_ms).toBe(0); + expect(handle.measurement.context_tokens).toBe(0); + + // P07 would accumulate here. + handle.measurement.client_prepare_ms = 5; + handle.measurement.context_tokens = 1000; + handle.measurement.output_tokens = 500; + + await registry.finalise(ac.signal, 'completed'); + await drainSink(sink); + + const records = await readAllRecords(); + expect(records[0].client_prepare_ms).toBe(5); + expect(records[0].context_tokens).toBe(1000); + expect(records[0].output_tokens).toBe(500); + }); +}); + +// --------------------------------------------------------------------------- +// Schema-valid record + honest residual +// --------------------------------------------------------------------------- + +describe('OperationLifecycleRegistry — record assembly', () => { + it('produces a schema-valid record with correct identity fields', async () => { + const provider = fixtureProvider({ + session_id: 'my-session', + runtime_id: 'rt-1', + provider: 'anthropic', + model: 'claude-3', + }); + const { registry, sink } = await startAndCreate({ identity: provider }); + const ac = new AbortController(); + registry.begin(ac.signal, 'my-session#agentic-loop#abc'); + await registry.finalise(ac.signal, 'completed'); + await drainSink(sink); + + const records = await readAllRecords(); + expect(records).toHaveLength(1); + const r = records[0]; + expect(r.schema_version).toBe(1); + expect(r.record_type).toBe('operation'); + expect(r.session_id).toBe('my-session'); + expect(r.runtime_id).toBe('rt-1'); + expect(r.provider).toBe('anthropic'); + expect(r.model).toBe('claude-3'); + expect(r.status).toBe('completed'); + }); + + it('reports an honest residual equal to elapsed with zero measurements', async () => { + let mono = 1000; + const { registry, sink } = await startAndCreate({ + monotonicNow: () => mono, + }); + const ac = new AbortController(); + registry.begin(ac.signal, 'sess-1#agentic-loop#uuid-1'); + mono += 500; // 500ms elapsed + await registry.finalise(ac.signal, 'completed'); + await drainSink(sink); + + const records = await readAllRecords(); + expect(records[0].operation_elapsed_ms).toBe(500); + expect(records[0].unclassified_elapsed_ms).toBe(500); + }); + + it('subtracts directly-measured phases from the residual', async () => { + let mono = 1000; + const { registry, sink } = await startAndCreate({ + monotonicNow: () => mono, + }); + const ac = new AbortController(); + const handle = registry.begin(ac.signal, 'sess-1#agentic-loop#uuid-1'); + handle.measurement.client_prepare_ms = 100; + handle.measurement.stream_handler_ms = 50; + mono += 500; + await registry.finalise(ac.signal, 'completed'); + await drainSink(sink); + + const records = await readAllRecords(); + expect(records[0].operation_elapsed_ms).toBe(500); + expect(records[0].unclassified_elapsed_ms).toBe(350); + }); + + it('uses wall-clock ISO for ts and monotonic for uptime', async () => { + const fixedWall = new Date('2026-08-09T12:00:00.000Z').getTime(); + const mono = 30_000; + const { registry, sink } = await startAndCreate({ + wallNow: () => fixedWall, + monotonicNow: () => mono, + }); + const ac = new AbortController(); + registry.begin(ac.signal, 'sess-1#agentic-loop#uuid-1'); + await registry.finalise(ac.signal, 'completed'); + await drainSink(sink); + + const records = await readAllRecords(); + expect(records[0].ts).toBe('2026-08-09T12:00:00.000Z'); + expect(records[0].uptime_ms).toBe(30_000); + }); + + it('omits memory columns in P06', async () => { + const { registry, sink } = await startAndCreate(); + const ac = new AbortController(); + registry.begin(ac.signal, 'sess-1#agentic-loop#uuid-1'); + await registry.finalise(ac.signal, 'completed'); + await drainSink(sink); + + const records = await readAllRecords(); + expect('rss_bytes' in records[0]).toBe(false); + expect('heap_used_bytes' in records[0]).toBe(false); + }); +}); + +// --------------------------------------------------------------------------- +// Error policy: schema failure fail-fast, filesystem fail-open +// --------------------------------------------------------------------------- + +describe('OperationLifecycleRegistry — error policy', () => { + it('rejects when an internal error occurs (identity provider throws)', async () => { + const throwingProvider: OperationIdentityProvider = { + snapshot: (): OperationIdentitySnapshot => { + throw new Error('identity unavailable'); + }, + }; + const { registry, sink } = await startAndCreate({ + identity: throwingProvider, + }); + const ac = new AbortController(); + expect(() => + registry.begin(ac.signal, 'sess-1#agentic-loop#uuid-1'), + ).toThrow('identity unavailable'); + await drainSink(sink); + }); + + it('fail-opens on filesystem sink errors (no throw to the operation path)', async () => { + const failingFs = new FailingAppendFilesystem(); + const runUuid = crypto.randomUUID(); + const retention = new PerfRetention({ + dir, + runUuid, + maintenanceIntervalMs: 60_000, + }); + const sink = new PerfSink({ + dir, + runUuid, + retention, + fs: failingFs, + }); + await sink.start(); + + const { registry } = createRegistry({ sink, retention }); + const ac = new AbortController(); + registry.begin(ac.signal, 'sess-1#agentic-loop#uuid-1'); + // Must NOT throw — filesystem errors fail-open. + await registry.finalise(ac.signal, 'completed'); + await drainSink(sink); + + // No record on disk (append failed), but no throw escaped. + const records = await readAllRecords(); + expect(records).toHaveLength(0); + }); +}); + +// --------------------------------------------------------------------------- +// Queued/requeued submission semantics: each consumed turn has its own operation +// --------------------------------------------------------------------------- + +describe('OperationLifecycleRegistry — each turn has its own operation', () => { + it('assigns distinct operation_ids and sequential indices to successive turns', async () => { + const { registry, sink } = await startAndCreate(); + + const ac1 = new AbortController(); + const h1 = registry.begin(ac1.signal, 'sess-1#agentic-loop#turn-1'); + await registry.finalise(ac1.signal, 'completed'); + + const ac2 = new AbortController(); + const h2 = registry.begin(ac2.signal, 'sess-1#agentic-loop#turn-2'); + await registry.finalise(ac2.signal, 'completed'); + + await drainSink(sink); + + expect(h1.operationId).not.toBe(h2.operationId); + const records = await readAllRecords(); + expect(records).toHaveLength(2); + expect(records.map((r) => r.operation_id).sort()).toEqual([ + 'sess-1#agentic-loop#turn-1', + 'sess-1#agentic-loop#turn-2', + ]); + }); +}); + +// --------------------------------------------------------------------------- +// Drain / queueWrite: internal errors fail-fast, not silently green (P06-D8) +// --------------------------------------------------------------------------- + +/** + * Filesystem port whose appendFile throws a non-errno (internal/programming) + * error. Extends FailingAppendFilesystem to inherit the real ensureDir/ + * openExclusive so start() creates the claim file successfully. + */ +class InternalErrorFilesystem extends FailingAppendFilesystem { + override async appendFile(): Promise { + throw new Error('internal append corruption'); + } +} + +describe('OperationLifecycleRegistry — drain/queueWrite internal error (P06-D8)', () => { + it('drain rejects when a queued write fails with an internal error', async () => { + const failingFs = new InternalErrorFilesystem(); + const runUuid = crypto.randomUUID(); + const retention = new PerfRetention({ + dir, + runUuid, + maintenanceIntervalMs: 60_000, + }); + const sink = new PerfSink({ + dir, + runUuid, + retention, + fs: failingFs, + }); + await sink.start(); + + const { registry } = createRegistry({ sink, retention }); + const ac = new AbortController(); + registry.begin(ac.signal, 'sess-1#agentic-loop#uuid-1'); + // The write fails with an internal error — finalise rejects (fail-fast). + await registry.finalise(ac.signal, 'completed').catch(() => {}); + + // Drain must reject — the internal error is not silently swallowed. + await expect(registry.drain()).rejects.toThrow( + 'internal append corruption', + ); + + // Best-effort cleanup. + await sink.dispose().catch(() => {}); + }); + + it('later records are not silently reported green after an internal failure', async () => { + const failingFs = new InternalErrorFilesystem(); + const runUuid = crypto.randomUUID(); + const retention = new PerfRetention({ + dir, + runUuid, + maintenanceIntervalMs: 60_000, + }); + const sink = new PerfSink({ + dir, + runUuid, + retention, + fs: failingFs, + }); + await sink.start(); + + const { registry } = createRegistry({ sink, retention }); + const ac1 = new AbortController(); + registry.begin(ac1.signal, 'sess-1#agentic-loop#uuid-1'); + // First write fails — finalise rejects (fail-fast). + await registry.finalise(ac1.signal, 'completed').catch(() => {}); + + // Second op — its write chains after the first. With fail-fast semantics, + // the rejected chain means the second finalise also rejects. + const ac2 = new AbortController(); + registry.begin(ac2.signal, 'sess-1#agentic-loop#uuid-2'); + await expect(registry.finalise(ac2.signal, 'completed')).rejects.toThrow( + 'internal append corruption', + ); + + // Drain surfaces the failure — not silently green. + await expect(registry.drain()).rejects.toThrow( + 'internal append corruption', + ); + + await sink.dispose().catch(() => {}); + }); + + it('superseded sweep write failure is surfaced via drain (not hidden)', async () => { + const failingFs = new InternalErrorFilesystem(); + const runUuid = crypto.randomUUID(); + const retention = new PerfRetention({ + dir, + runUuid, + maintenanceIntervalMs: 60_000, + }); + const sink = new PerfSink({ + dir, + runUuid, + retention, + fs: failingFs, + }); + await sink.start(); + + const { registry } = createRegistry({ sink, retention }); + // begin #1 — op1 active + const ac1 = new AbortController(); + registry.begin(ac1.signal, 'sess-1#agentic-loop#uuid-1'); + // begin #2 — sweeps op1 as superseded (queued write, not individually awaited) + const ac2 = new AbortController(); + registry.begin(ac2.signal, 'sess-1#agentic-loop#uuid-2'); + // Finalise op2 — its write chains after the superseded sweep + await expect(registry.finalise(ac2.signal, 'completed')).rejects.toThrow( + 'internal append corruption', + ); + + // Drain must reject — the superseded sweep's internal error is NOT hidden. + await expect(registry.drain()).rejects.toThrow( + 'internal append corruption', + ); + + await sink.dispose().catch(() => {}); + }); +}); + +describe('OperationLifecycleRegistry — read-only active-operation snapshot (P12)', () => { + it('returns null when no operation is active', () => { + const { registry } = createRegistry(); + expect(registry.getActiveOperationSnapshot()).toBe(null); + }); + + it('returns provider, model, and monotonic elapsed for an active operation', () => { + let mono = 1000; + const provider = fixtureProvider({ + provider: 'openai', + model: 'gpt-4o', + }); + const { registry } = createRegistry({ + identity: provider, + monotonicNow: () => mono, + }); + const ac = new AbortController(); + registry.begin(ac.signal, 'sess#agentic-loop#uuid-1'); + + mono = 2500; + const snap = registry.getActiveOperationSnapshot(); + expect(snap).not.toBe(null); + expect(snap!.provider).toBe('openai'); + expect(snap!.model).toBe('gpt-4o'); + // Elapsed = 2500 - 1000 = 1500 ms (monotonic). + expect(snap!.elapsedMs).toBe(1500); + }); + + it('returns null after the operation is finalised', async () => { + const mono = 1000; + const { registry, retention } = createRegistry({ + monotonicNow: () => mono, + }); + await retention.start(); + const ac = new AbortController(); + registry.begin(ac.signal, 'sess#agentic-loop#uuid-1'); + expect(registry.getActiveOperationSnapshot()).not.toBe(null); + + await registry.finalise(ac.signal, 'completed'); + expect(registry.getActiveOperationSnapshot()).toBe(null); + }); + + it('snapshot does not expose mutable operation state', () => { + const { registry } = createRegistry(); + const ac = new AbortController(); + registry.begin(ac.signal, 'sess#agentic-loop#uuid-1'); + const snap = registry.getActiveOperationSnapshot(); + expect(snap).not.toBe(null); + // Only provider, model, elapsedMs (and optional memory) — no measurement, + // no status, no operationId, no signal. + const keys = Object.keys(snap!); + expect(keys).toContain('provider'); + expect(keys).toContain('model'); + expect(keys).toContain('elapsedMs'); + for (const key of keys) { + expect(key).not.toMatch(/measurement|status|operationId|signal/i); + } + }); +}); diff --git a/packages/cli/src/ui/hooks/agentStream/operationLifecycle.p07.behavior.test.ts b/packages/cli/src/ui/hooks/agentStream/operationLifecycle.p07.behavior.test.ts new file mode 100644 index 0000000000..8fafde6526 --- /dev/null +++ b/packages/cli/src/ui/hooks/agentStream/operationLifecycle.p07.behavior.test.ts @@ -0,0 +1,872 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * P07 behavioral tests for OperationLifecycleRegistry: direct client phases, + * provider/tool interval metrics, honest residual, granular cancellation + * classification, and queue behavior (EVIDENCE-AC4/AC5/AC6). + * + * Real registry + real PerfSink/PerfRetention + real IntervalUnion + real + * PerfPhaseObserver seam + real temp files + real reader. No mock theater. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { + PerfSink, + PerfRetention, + setPerfPhaseObserver, + getPerfPhaseObserver, +} from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import { readPerfRecords } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import type { PerfOperationRecord } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import { + setInteractiveRenderObserver, + setInteractiveStdoutObserver, +} from '../../inkRenderOptions.js'; +import { + OperationLifecycleRegistry, + type OperationIdentityProvider, + type OperationIdentitySnapshot, +} from './operationLifecycle.js'; + +// --------------------------------------------------------------------------- +// Shared setup +// --------------------------------------------------------------------------- + +let dir: string; +const startedSinks = new Set(); + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'op-p07-')); + startedSinks.clear(); + setPerfPhaseObserver(null); + setInteractiveRenderObserver(null); + setInteractiveStdoutObserver(null); +}); + +afterEach(async () => { + const errors: unknown[] = []; + for (const sink of startedSinks) { + try { + await sink.dispose(); + } catch (error) { + errors.push(error); + } + } + startedSinks.clear(); + setPerfPhaseObserver(null); + setInteractiveRenderObserver(null); + setInteractiveStdoutObserver(null); + try { + fs.rmSync(dir, { recursive: true, force: true }); + } catch (error) { + errors.push(error); + } + if (errors.length === 1) throw errors[0]; + if (errors.length > 1) { + throw new AggregateError(errors, 'P07 test cleanup failed'); + } +}); + +function fixtureIdentity( + overrides: Partial = {}, +): OperationIdentitySnapshot { + return { + session_id: 'sess-abc', + runtime_id: 'rt-main', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'sha256:project-hash', + llxprt_version: '0.11.0', + git_sha: 'abc1234', + runtime: 'bun-1.3.14', + platform: 'darwin-arm64', + provider: 'openai', + model: 'gpt-4o', + terminal_cols: 120, + terminal_rows: 40, + render_mode: 'incremental', + ...overrides, + }; +} + +function fixtureProvider( + overrides: Partial = {}, +): OperationIdentityProvider { + const snap = fixtureIdentity(overrides); + return { snapshot: () => snap }; +} + +/** + * Creates a registry with a real sink/retention in tmpdir, installs observers, + * and returns everything needed for assertions. + */ +async function createStartedRegistry( + overrides: { + monotonicNow?: () => number; + } = {}, +): Promise<{ + registry: OperationLifecycleRegistry; + sink: PerfSink; + retention: PerfRetention; + readRecords: () => Promise; +}> { + const retention = new PerfRetention({ + dir, + runUuid: crypto.randomUUID(), + maintenanceIntervalMs: 60_000, + }); + const sink = new PerfSink({ + dir, + runUuid: crypto.randomUUID(), + retention, + }); + await sink.start(); + startedSinks.add(sink); + const registry = new OperationLifecycleRegistry({ + identityProvider: fixtureProvider(), + sink, + retention, + monotonicNow: overrides.monotonicNow, + }); + registry.installObservers(); + + const readRecords = async (): Promise => { + await registry.drain(); + await sink.dispose(); + startedSinks.delete(sink); + const files = fs.readdirSync(dir).filter((f) => f.endsWith('.jsonl')); + const records: PerfOperationRecord[] = []; + for (const file of files) { + const result = await readPerfRecords(path.join(dir, file)); + for (const rec of result.records) { + if (rec.record_type === 'operation') { + records.push(rec); + } + } + } + return records; + }; + + return { registry, sink, retention, readRecords }; +} + +// --------------------------------------------------------------------------- +// Direct client phases +// --------------------------------------------------------------------------- + +describe('P07 direct client phases (AC-5)', () => { + it('client_prepare_ms is set via setClientPrepareMs', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.setClientPrepareMs(controller.signal, 12.5); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records).toHaveLength(1); + expect(records[0].client_prepare_ms).toBe(12.5); + }); + + it('stream_handler_ms accumulates sync dispatch time', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.addStreamHandlerMs(controller.signal, 1.5); + registry.addStreamHandlerMs(controller.signal, 2.5); + registry.addStreamHandlerMs(controller.signal, 3.0); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].stream_handler_ms).toBeCloseTo(7.0, 5); + }); + + it('client_finalize_ms is self-measured (synchronous writeRecord bookkeeping boundary)', async () => { + // client_finalize_ms measures ONLY the synchronous record-assembly work + // inside writeRecord (union computation, approval-wait closing, residual). + // It EXCLUDES async claim filesystem checks and sink append time. It is + // directly measured by the registry itself, not set by the caller. + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + // Self-measured: finite and non-negative (per schema). The exact value + // depends on CPU, so we assert schema-validity + non-negativity. + expect(records[0].client_finalize_ms).toBeGreaterThanOrEqual(0); + expect(Number.isFinite(records[0].client_finalize_ms)).toBe(true); + }); + + it('Ink render accumulates renderTime and count independently', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + // Simulate Ink render passes via the observer + registry.onRender(0.5); + registry.onRender(0.3); + registry.onRender(0.2); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].ink_render_ms).toBeCloseTo(1.0, 5); + expect(records[0].ink_render_count).toBe(3); + }); + + it('stdout write accumulates bytes, calls, and sync time', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.onWrite(100, 0.1); + registry.onWrite(200, 0.2); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].stdout_bytes).toBe(300); + expect(records[0].stdout_write_calls).toBe(2); + expect(records[0].stdout_write_sync_ms).toBeCloseTo(0.3, 5); + }); + + it('render count and write count are independent', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + // 3 renders but only 1 write (coalesced frame) + registry.onRender(0.1); + registry.onRender(0.1); + registry.onRender(0.1); + registry.onWrite(500, 0.05); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].ink_render_count).toBe(3); + expect(records[0].stdout_write_calls).toBe(1); + }); + + it('no phase is computed as elapsed − provider − tool', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.setClientPrepareMs(controller.signal, 10); + // Add provider interval + registry.onProviderAttemptStart({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 0, + }); + registry.onProviderAttemptEnd({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 0, + endMs: 100, + status: 'success', + inputTokens: 50, + outputTokens: 20, + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + // client_prepare_ms is independently measured, NOT derived + expect(records[0].client_prepare_ms).toBe(10); + // provider_attempt_sum_ms is independently measured + expect(records[0].provider_attempt_sum_ms).toBe(100); + }); +}); + +// --------------------------------------------------------------------------- +// Provider/tool interval metrics +// --------------------------------------------------------------------------- + +describe('P07 provider/tool interval metrics (AC-5)', () => { + it('provider attempt: count, sum, union from real lifecycle', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.onProviderAttemptStart({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 100, + }); + registry.onProviderAttemptEnd({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 100, + endMs: 200, + status: 'success', + inputTokens: 100, + outputTokens: 50, + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].provider_attempts).toBe(1); + expect(records[0].provider_attempt_sum_ms).toBe(100); + expect(records[0].provider_union_ms).toBe(100); + }); + + it('provider retries: multiple attempts accumulate sum and union', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + // Attempt 1 (error, retried) + registry.onProviderAttemptStart({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 0, + }); + registry.onProviderAttemptEnd({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 0, + endMs: 100, + status: 'error', + inputTokens: 50, + outputTokens: 0, + }); + // Attempt 2 (success) + registry.onProviderAttemptStart({ + attemptId: 'a2', + promptId: 'sess#agentic-loop#uuid', + startMs: 150, + }); + registry.onProviderAttemptEnd({ + attemptId: 'a2', + promptId: 'sess#agentic-loop#uuid', + startMs: 150, + endMs: 300, + status: 'success', + inputTokens: 80, + outputTokens: 40, + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].provider_attempts).toBe(2); + expect(records[0].provider_attempt_sum_ms).toBe(250); // 100 + 150 + // Union has a gap [100,150), so union = 100 + 150 = 250 + expect(records[0].provider_union_ms).toBe(250); + }); + + it('provider overlapping retries: union ≤ sum', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + // Overlapping attempts (concurrent) + registry.onProviderAttemptStart({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 0, + }); + registry.onProviderAttemptEnd({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 0, + endMs: 200, + status: 'error', + inputTokens: 50, + outputTokens: 0, + }); + registry.onProviderAttemptStart({ + attemptId: 'a2', + promptId: 'sess#agentic-loop#uuid', + startMs: 50, + }); + registry.onProviderAttemptEnd({ + attemptId: 'a2', + promptId: 'sess#agentic-loop#uuid', + startMs: 50, + endMs: 150, + status: 'success', + inputTokens: 80, + outputTokens: 40, + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].provider_attempts).toBe(2); + expect(records[0].provider_attempt_sum_ms).toBe(300); // 200 + 100 + // Union is [0,200) (a2 is fully nested) = 200 ≤ 300 + expect(records[0].provider_union_ms).toBe(200); + expect(records[0].provider_union_ms).toBeLessThanOrEqual( + records[0].provider_attempt_sum_ms, + ); + }); + + it('provider consumer-abort end boundary', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.onProviderAttemptStart({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 0, + }); + registry.onProviderAttemptEnd({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 0, + endMs: 50, + status: 'aborted', + inputTokens: 50, + outputTokens: 10, + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].provider_attempts).toBe(1); + expect(records[0].provider_union_ms).toBe(50); + }); + + it('provider dedup by attemptId (exactly once)', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.onProviderAttemptStart({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 0, + }); + registry.onProviderAttemptEnd({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 0, + endMs: 100, + status: 'success', + inputTokens: 50, + outputTokens: 20, + }); + // Duplicate end for the same attemptId — must NOT double-count. + registry.onProviderAttemptEnd({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 0, + endMs: 100, + status: 'success', + inputTokens: 50, + outputTokens: 20, + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].provider_attempts).toBe(1); + expect(records[0].provider_attempt_sum_ms).toBe(100); + }); + + it('provider preserves token counts', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.onProviderAttemptStart({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 0, + }); + registry.onProviderAttemptEnd({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 0, + endMs: 100, + status: 'success', + inputTokens: 500, + outputTokens: 120, + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].context_tokens).toBe(500); + expect(records[0].output_tokens).toBe(120); + }); + + it('tool call: count, sum, union from real logger seam', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.onToolCallCompleted({ + promptId: 'sess#agentic-loop#uuid', + callId: 'tool-1', + startMs: 200, + endMs: 300, + durationMs: 100, + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].tool_calls).toBe(1); + expect(records[0].tool_call_sum_ms).toBe(100); + expect(records[0].tool_union_ms).toBe(100); + }); + + it('tool dedup by real callId (exactly once)', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.onToolCallCompleted({ + promptId: 'sess#agentic-loop#uuid', + callId: 'tool-1', + startMs: 200, + endMs: 300, + durationMs: 100, + }); + // Duplicate — must NOT double-count. + registry.onToolCallCompleted({ + promptId: 'sess#agentic-loop#uuid', + callId: 'tool-1', + startMs: 200, + endMs: 300, + durationMs: 100, + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].tool_calls).toBe(1); + expect(records[0].tool_call_sum_ms).toBe(100); + }); + + it('tool missing callId is counted honestly (no invented ID)', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.onToolCallCompleted({ + promptId: 'sess#agentic-loop#uuid', + callId: undefined, + startMs: 200, + endMs: 250, + durationMs: 50, + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].tool_calls).toBe(1); + expect(records[0].tool_call_sum_ms).toBe(50); + }); + + it('agent_activity_union_ms = union(provider, tool) including overlap', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + // Provider [0, 200) + registry.onProviderAttemptStart({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 0, + }); + registry.onProviderAttemptEnd({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 0, + endMs: 200, + status: 'success', + inputTokens: 0, + outputTokens: 0, + }); + // Tool [150, 250) — overlaps provider + registry.onToolCallCompleted({ + promptId: 'sess#agentic-loop#uuid', + callId: 'tool-1', + startMs: 150, + endMs: 250, + durationMs: 100, + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + // Union [0, 250) = 250 + expect(records[0].agent_activity_union_ms).toBe(250); + }); +}); + +// --------------------------------------------------------------------------- +// Honest residual +// --------------------------------------------------------------------------- + +describe('P07 honest residual (AC-5)', () => { + it('residual = elapsed − direct phases − approval_wait (provider/tool NOT subtracted)', async () => { + let mono = 1000; + const { registry, readRecords } = await createStartedRegistry({ + monotonicNow: () => mono, + }); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.setClientPrepareMs(controller.signal, 100); + registry.addStreamHandlerMs(controller.signal, 50); + registry.onRender(30); + registry.onWrite(0, 20); + // client_finalize_ms is self-measured by the registry; with a static + // monotonic clock it is 0 (finalizeStart === finalizeEnd). + + // Provider interval (NOT subtracted from residual) + registry.onProviderAttemptStart({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 0, + }); + registry.onProviderAttemptEnd({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 0, + endMs: 500, + status: 'success', + inputTokens: 0, + outputTokens: 0, + }); + + mono = 2000; // elapsed = 1000 + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + // residual = 1000 - 100(prepare) - 50(handler) - 30(render) - 20(stdout) + // - 0(finalize, self-measured with static clock) - 0(approval) + // = 800 + // Provider/tool unions NOT subtracted (they overlap client work). + expect(records[0].unclassified_elapsed_ms).toBeCloseTo(800, 0); + expect(records[0].operation_elapsed_ms).toBe(1000); + }); + + it('negative residual remains negative and schema-valid (overlap)', async () => { + let mono = 0; + const { registry, readRecords } = await createStartedRegistry({ + monotonicNow: () => mono, + }); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + // Direct phases that exceed elapsed + registry.setClientPrepareMs(controller.signal, 500); + registry.addStreamHandlerMs(controller.signal, 600); + // elapsed will be 100, but phases sum to 1100 → residual = -1000 + mono = 100; + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].unclassified_elapsed_ms).toBe(-1000); + expect(records[0].operation_elapsed_ms).toBe(100); + // Schema-valid: finite (may be negative — the schema allows it) + expect(Number.isFinite(records[0].unclassified_elapsed_ms)).toBe(true); + }); +}); + +// --------------------------------------------------------------------------- +// Granular cancellation classification (AC-4) +// --------------------------------------------------------------------------- + +describe('P07 granular cancellation classification (AC-4)', () => { + it('classifyCancellation returns cancelled_during_api when phase is api', async () => { + const { registry } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.enterApiPhase(controller.signal); + const status = registry.classifyCancellation(controller.signal); + expect(status).toBe('cancelled_during_api'); + }); + + it('classifyCancellation returns cancelled_during_tool when phase is tool', async () => { + const { registry } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.handleToolStatus(controller.signal, 'executing', 'tool-1'); + const status = registry.classifyCancellation(controller.signal); + expect(status).toBe('cancelled_during_tool'); + }); + + it('classifyCancellation returns cancelled_during_approval when phase is approval', async () => { + const { registry } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.handleToolStatus(controller.signal, 'awaiting-approval', 'app-1'); + const status = registry.classifyCancellation(controller.signal); + expect(status).toBe('cancelled_during_approval'); + }); + + it('deterministic precedence: approval > tool > api on overlap', async () => { + const { registry } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + // Overlapping active states: api + tool + approval all active; the most + // specific (approval) wins at the instant of abort. + registry.enterApiPhase(controller.signal); + registry.handleToolStatus(controller.signal, 'scheduled', 'tool-1'); + registry.handleToolStatus(controller.signal, 'awaiting-approval', 'app-1'); + const status = registry.classifyCancellation(controller.signal); + expect(status).toBe('cancelled_during_approval'); + }); + + it('phase evidence persists after active op is cleared (terminal evidence)', async () => { + const { registry } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + // A tool-status cancelled event retains the cancelled phase (tool) as + // terminal cancellation evidence that persists past finalise. + registry.handleToolStatus(controller.signal, 'cancelled', 'tool-1'); + // Finalise removes the op from active, but classification must still work. + await registry.finalise(controller.signal, 'cancelled_during_tool'); + // After finalise, classifyCancellation should still return the captured phase. + const status = registry.classifyCancellation(controller.signal); + expect(status).toBe('cancelled_during_tool'); + }); + + it('default phase (never entered) classifies as cancelled_during_api', async () => { + const { registry } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + // Never entered any phase (sent but no phase event) + const status = registry.classifyCancellation(controller.signal); + expect(status).toBe('cancelled_during_api'); + }); +}); + +// --------------------------------------------------------------------------- +// D1 continuation correlation +// --------------------------------------------------------------------------- + +describe('P07 D1 continuation correlation (AC-3)', () => { + it('tool call with continuation prompt_id associates to operation', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + // Tool call from continuation #1 + registry.onToolCallCompleted({ + promptId: 'sess#agentic-loop#uuid#continuation#1', + callId: 'tool-cont-1', + startMs: 100, + endMs: 200, + durationMs: 100, + }); + // Tool call from continuation #2 + registry.onToolCallCompleted({ + promptId: 'sess#agentic-loop#uuid#continuation#2', + callId: 'tool-cont-2', + startMs: 300, + endMs: 400, + durationMs: 100, + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].tool_calls).toBe(2); + expect(records[0].tool_call_sum_ms).toBe(200); + }); +}); + +// --------------------------------------------------------------------------- +// Observer fail-fast (D8) +// --------------------------------------------------------------------------- + +describe('P07 observer fail-fast (D8)', () => { + it('registry observer methods are direct (no try/catch swallowing)', async () => { + // The registry implements PerfPhaseObserver directly. Its methods have no + // try/catch — internal/programming errors propagate. This test verifies + // the registry's onProviderAttemptEnd does NOT swallow by confirming it + // processes real events correctly (structural D8 verified by reading the + // source: no catch boundary around the observer method body). + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.onProviderAttemptStart({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 0, + }); + registry.onProviderAttemptEnd({ + attemptId: 'a1', + promptId: 'sess#agentic-loop#uuid', + startMs: 0, + endMs: 100, + status: 'success', + inputTokens: 50, + outputTokens: 20, + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + // If the observer method had swallowed, provider_attempts would be 0. + expect(records[0].provider_attempts).toBe(1); + }); + + it('module-level perf observer propagates errors from installed observer', () => { + // When a throwing observer is installed via setPerfPhaseObserver, calling + // it via getPerfPhaseObserver() propagates the error (D8: no swallow). + setPerfPhaseObserver({ + onProviderAttemptStart: () => undefined, + onProviderAttemptEnd: () => { + throw new Error('perf observer internal error'); + }, + onToolCallCompleted: () => undefined, + }); + const observer = getPerfPhaseObserver(); + expect(observer).not.toBeNull(); + expect(() => + observer!.onProviderAttemptEnd({ + attemptId: 'x', + promptId: 'sess#agentic-loop#uuid', + startMs: 0, + endMs: 10, + status: 'success', + inputTokens: 0, + outputTokens: 0, + }), + ).toThrow('perf observer internal error'); + }); +}); + +// --------------------------------------------------------------------------- +// Queue behavior (unhandled-rejection fix) +// --------------------------------------------------------------------------- + +describe('P07 superseded queue behavior', () => { + it('superseded sweep rejection does not become process-unhandled', async () => { + // A registry with a sink that throws on write (internal error). + const retention = new PerfRetention({ + dir, + runUuid: crypto.randomUUID(), + maintenanceIntervalMs: 60_000, + }); + const throwingSink = { + write(): Promise { + return Promise.reject(new Error('internal write error')); + }, + } as unknown as PerfSink; + const registry = new OperationLifecycleRegistry({ + identityProvider: fixtureProvider(), + sink: throwingSink, + retention, + }); + registry.installObservers(); + + try { + const controller1 = new AbortController(); + const controller2 = new AbortController(); + registry.begin(controller1.signal, 'sess#agentic-loop#uuid1'); + registry.begin(controller2.signal, 'sess#agentic-loop#uuid2'); + + await expect(registry.dispose()).rejects.toThrow('internal write error'); + } finally { + await retention.dispose(); + } + }); +}); + +// --------------------------------------------------------------------------- +// Default-off +// --------------------------------------------------------------------------- + +describe('P07 default-off', () => { + it('absent registry means no perf observer installed', () => { + setPerfPhaseObserver(null); + expect(getPerfPhaseObserver()).toBeNull(); + // No observer, no notification. + }); + + it('installObservers sets perf/render/stdout observers', async () => { + await createStartedRegistry(); + expect(getPerfPhaseObserver()).not.toBeNull(); + }); +}); + +// --------------------------------------------------------------------------- +// Approval wait +// --------------------------------------------------------------------------- + +describe('P07 approval_wait_ms (AC-5)', () => { + it('approval wait accumulates via handleToolStatus approval transitions', async () => { + let mono = 0; + const { registry, readRecords } = await createStartedRegistry({ + monotonicNow: () => mono, + }); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + // Enter approval at t=100 + mono = 100; + registry.handleToolStatus(controller.signal, 'awaiting-approval', 'app-1'); + // Exit approval at t=250 (tool succeeds → closes the approval wait) + mono = 250; + registry.handleToolStatus(controller.signal, 'success', 'app-1'); + mono = 1000; + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].approval_wait_ms).toBeCloseTo(150, 0); + }); +}); diff --git a/packages/cli/src/ui/hooks/agentStream/operationLifecycle.p07.contract.behavior.test.ts b/packages/cli/src/ui/hooks/agentStream/operationLifecycle.p07.contract.behavior.test.ts new file mode 100644 index 0000000000..1ba7738be2 --- /dev/null +++ b/packages/cli/src/ui/hooks/agentStream/operationLifecycle.p07.contract.behavior.test.ts @@ -0,0 +1,768 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * P07 contract-correction behavioral tests (issue #3167). + * + * These tests pin the corrected contracts that the initial P07 pass got wrong: + * + * 1. Provider correlation MUST use prompt/logical-request identity + * (deriveOperationId(info.promptId)), NOT getFirstActiveOp. Unrelated + * prompt IDs and simultaneous attempts must NOT be misattributed to the + * foreground operation. Continuation prompt IDs collapse via the exact D1 + * split. + * 2. Live cancellation phases track REAL active state; a completed/rejected + * approval followed by ordinary API activity then abort classifies + * during_api. Retained terminal cancellation evidence is set ONLY by a + * cancellation terminal signal/event (tool-status cancelled, provider + * attempt aborted end). tool-status transitions are keyed by tool call ID. + * 3. Tool interval honesty: a ToolCallEvent lacking start_ms/end_ms counts/ + * sums its duration but does NOT synthesize an interval. + * + * Real registry + real PerfSink/PerfRetention + real temp files. No mock + * theater. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { + PerfSink, + PerfRetention, + setPerfPhaseObserver, + getPerfPhaseObserver, +} from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import { readPerfRecords } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import type { PerfOperationRecord } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import { + setInteractiveRenderObserver, + setInteractiveStdoutObserver, + getInteractiveRenderObserver, + getInteractiveStdoutObserver, +} from '../../inkRenderOptions.js'; +import { + OperationLifecycleRegistry, + type OperationIdentityProvider, + type OperationIdentitySnapshot, +} from './operationLifecycle.js'; + +let dir: string; +let activeSink: PerfSink | null = null; + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'op-p07-contract-')); + activeSink = null; + setPerfPhaseObserver(null); + setInteractiveRenderObserver(null); + setInteractiveStdoutObserver(null); +}); + +afterEach(async () => { + const errors: unknown[] = []; + if (activeSink !== null) { + try { + await activeSink.dispose(); + } catch (err) { + errors.push(err); + } + activeSink = null; + } + setPerfPhaseObserver(null); + setInteractiveRenderObserver(null); + setInteractiveStdoutObserver(null); + fs.rmSync(dir, { recursive: true, force: true }); + if (errors.length === 1) throw errors[0]; + if (errors.length > 1) { + throw new AggregateError( + errors, + 'p07 contract afterEach sink cleanup failed', + ); + } +}); + +function fixtureIdentity( + overrides: Partial = {}, +): OperationIdentitySnapshot { + return { + session_id: 'sess-abc', + runtime_id: 'rt-main', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'sha256:project-hash', + llxprt_version: '0.11.0', + git_sha: 'abc1234', + runtime: 'bun-1.3.14', + platform: 'darwin-arm64', + provider: 'openai', + model: 'gpt-4o', + terminal_cols: 120, + terminal_rows: 40, + render_mode: 'incremental', + ...overrides, + }; +} + +function fixtureProvider( + overrides: Partial = {}, +): OperationIdentityProvider { + const snap = fixtureIdentity(overrides); + return { snapshot: () => snap }; +} + +async function createStartedRegistry( + overrides: { monotonicNow?: () => number } = {}, +): Promise<{ + registry: OperationLifecycleRegistry; + readRecords: () => Promise; +}> { + const retention = new PerfRetention({ + dir, + runUuid: crypto.randomUUID(), + maintenanceIntervalMs: 60_000, + }); + const sink = new PerfSink({ + dir, + runUuid: crypto.randomUUID(), + retention, + }); + await sink.start(); + activeSink = sink; + const registry = new OperationLifecycleRegistry({ + identityProvider: fixtureProvider(), + sink, + retention, + monotonicNow: overrides.monotonicNow, + }); + registry.installObservers(); + const readRecords = async (): Promise => { + await registry.drain(); + await sink.dispose(); + activeSink = null; + const files = fs.readdirSync(dir).filter((f) => f.endsWith('.jsonl')); + const records: PerfOperationRecord[] = []; + for (const file of files) { + const result = await readPerfRecords(path.join(dir, file)); + for (const rec of result.records) { + if (rec.record_type === 'operation') records.push(rec); + } + } + return records; + }; + return { registry, readRecords }; +} + +// --------------------------------------------------------------------------- +// 1. Provider correlation by prompt/logical-request identity (NOT +// getFirstActiveOp) +// --------------------------------------------------------------------------- + +describe('P07 provider correlation by promptId (AC-3)', () => { + it('attributes a provider attempt whose promptId derives to the active op', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.onProviderAttemptStart({ + attemptId: 'a1', + startMs: 0, + promptId: 'sess#agentic-loop#uuid', + }); + registry.onProviderAttemptEnd({ + attemptId: 'a1', + startMs: 0, + endMs: 100, + status: 'success', + inputTokens: 10, + outputTokens: 5, + promptId: 'sess#agentic-loop#uuid', + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].provider_attempts).toBe(1); + }); + + it('continuation prompt IDs collapse to the same operation via D1 split', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.onProviderAttemptStart({ + attemptId: 'a1', + startMs: 0, + promptId: 'sess#agentic-loop#uuid#continuation#1', + }); + registry.onProviderAttemptEnd({ + attemptId: 'a1', + startMs: 0, + endMs: 100, + status: 'success', + inputTokens: 0, + outputTokens: 0, + promptId: 'sess#agentic-loop#uuid#continuation#1', + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].provider_attempts).toBe(1); + }); + + it('an unrelated prompt ID is NOT misattributed to the foreground op', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#fg'); + // A subagent / unrelated concurrent request with a different prompt ID. + registry.onProviderAttemptStart({ + attemptId: 'sub-a1', + startMs: 0, + promptId: 'sess#agentic-loop#unrelated-subagent', + }); + registry.onProviderAttemptEnd({ + attemptId: 'sub-a1', + startMs: 0, + endMs: 999, + status: 'success', + inputTokens: 777, + outputTokens: 888, + promptId: 'sess#agentic-loop#unrelated-subagent', + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + // The unrelated attempt must NOT have been attributed. + expect(records[0].provider_attempts).toBe(0); + expect(records[0].context_tokens).toBe(0); + expect(records[0].output_tokens).toBe(0); + }); + + it('simultaneous retries attribute correctly while an unrelated attempt does not', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#fg'); + // Retry 1 (error) for the foreground op. + registry.onProviderAttemptStart({ + attemptId: 'fg-a1', + startMs: 0, + promptId: 'sess#agentic-loop#fg', + }); + // An unrelated concurrent attempt interleaved. + registry.onProviderAttemptStart({ + attemptId: 'sub-a1', + startMs: 10, + promptId: 'sess#agentic-loop#sub', + }); + registry.onProviderAttemptEnd({ + attemptId: 'fg-a1', + startMs: 0, + endMs: 50, + status: 'error', + inputTokens: 5, + outputTokens: 0, + promptId: 'sess#agentic-loop#fg', + }); + registry.onProviderAttemptEnd({ + attemptId: 'sub-a1', + startMs: 10, + endMs: 60, + status: 'success', + inputTokens: 999, + outputTokens: 999, + promptId: 'sess#agentic-loop#sub', + }); + // Retry 2 (success) for the foreground op. + registry.onProviderAttemptStart({ + attemptId: 'fg-a2', + startMs: 100, + promptId: 'sess#agentic-loop#fg', + }); + registry.onProviderAttemptEnd({ + attemptId: 'fg-a2', + startMs: 100, + endMs: 200, + status: 'success', + inputTokens: 20, + outputTokens: 8, + promptId: 'sess#agentic-loop#fg', + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + // Only the two foreground retries attribute; the unrelated one does not. + expect(records[0].provider_attempts).toBe(2); + expect(records[0].context_tokens).toBe(25); // 5 + 20 + expect(records[0].output_tokens).toBe(8); // 0 + 8 + }); + + it('a provider attempt whose promptId matches NO active op is ignored', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#fg'); + // End without a matching start promptId — must be ignored entirely. + registry.onProviderAttemptEnd({ + attemptId: 'ghost', + startMs: 0, + endMs: 10, + status: 'success', + inputTokens: 1, + outputTokens: 1, + promptId: 'sess#agentic-loop#no-such-op', + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].provider_attempts).toBe(0); + }); +}); + +// --------------------------------------------------------------------------- +// 2. Live cancellation phases — real active state + retained evidence +// --------------------------------------------------------------------------- + +describe('P07 live cancellation phases (AC-4)', () => { + it('executing tool-status classifies cancelled_during_tool', async () => { + const { registry } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.enterApiPhase(controller.signal); + registry.handleToolStatus(controller.signal, 'executing', 'c1'); + expect(registry.classifyCancellation(controller.signal)).toBe( + 'cancelled_during_tool', + ); + }); + + it('awaiting-approval tool-status classifies cancelled_during_approval', async () => { + const { registry } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.enterApiPhase(controller.signal); + registry.handleToolStatus(controller.signal, 'awaiting-approval', 'c1'); + expect(registry.classifyCancellation(controller.signal)).toBe( + 'cancelled_during_approval', + ); + }); + + it('precedence approval > tool when both active (keyed by call ID)', async () => { + const { registry } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.enterApiPhase(controller.signal); + registry.handleToolStatus(controller.signal, 'executing', 'c1'); + registry.handleToolStatus(controller.signal, 'awaiting-approval', 'c2'); + expect(registry.classifyCancellation(controller.signal)).toBe( + 'cancelled_during_approval', + ); + }); + + it('completed approval then API activity then abort classifies during_api', async () => { + const { registry } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.enterApiPhase(controller.signal); + // Tool awaits approval, then is approved/executed/succeeded → approval + // phase CLOSED. + registry.handleToolStatus(controller.signal, 'awaiting-approval', 'c1'); + registry.handleToolStatus(controller.signal, 'executing', 'c1'); + registry.handleToolStatus(controller.signal, 'success', 'c1'); + // Ordinary API activity resumes; abort must classify during_api, NOT + // during_approval (the approval is no longer active). + expect(registry.classifyCancellation(controller.signal)).toBe( + 'cancelled_during_api', + ); + }); + + it('rejected approval (error) closes the approval phase', async () => { + const { registry } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.enterApiPhase(controller.signal); + registry.handleToolStatus(controller.signal, 'awaiting-approval', 'c1'); + registry.handleToolStatus(controller.signal, 'error', 'c1'); + expect(registry.classifyCancellation(controller.signal)).toBe( + 'cancelled_during_api', + ); + }); + + it('tool-status cancelled retains the cancelled phase past finalise', async () => { + const { registry } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.enterApiPhase(controller.signal); + registry.handleToolStatus(controller.signal, 'executing', 'c1'); + // A cancellation terminal event retains the phase that was cancelled. + registry.handleToolStatus(controller.signal, 'cancelled', 'c1'); + await registry.finalise(controller.signal, 'cancelled_during_tool'); + // Evidence persists after the active op is cleared. + expect(registry.classifyCancellation(controller.signal)).toBe( + 'cancelled_during_tool', + ); + }); + + it('tool-status cancelled during approval retains cancelled_during_approval', async () => { + const { registry } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.enterApiPhase(controller.signal); + registry.handleToolStatus(controller.signal, 'awaiting-approval', 'c1'); + registry.handleToolStatus(controller.signal, 'cancelled', 'c1'); + await registry.finalise(controller.signal, 'cancelled_during_approval'); + expect(registry.classifyCancellation(controller.signal)).toBe( + 'cancelled_during_approval', + ); + }); + + it('provider attempt aborted end retains API cancellation evidence', async () => { + const { registry } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.enterApiPhase(controller.signal); + registry.onProviderAttemptStart({ + attemptId: 'a1', + startMs: 0, + promptId: 'sess#agentic-loop#uuid', + }); + registry.onProviderAttemptEnd({ + attemptId: 'a1', + startMs: 0, + endMs: 50, + status: 'aborted', + inputTokens: 0, + outputTokens: 0, + promptId: 'sess#agentic-loop#uuid', + }); + await registry.finalise(controller.signal, 'cancelled_during_api'); + expect(registry.classifyCancellation(controller.signal)).toBe( + 'cancelled_during_api', + ); + }); + + it('tool-status cancelled is stale once a provider attempt proves continuation (corrected contract)', async () => { + const { registry } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.enterApiPhase(controller.signal); + registry.handleToolStatus(controller.signal, 'executing', 'c1'); + registry.handleToolStatus(controller.signal, 'cancelled', 'c1'); + // BEFORE continuation, the tool evidence is retained. + expect(registry.classifyCancellation(controller.signal)).toBe( + 'cancelled_during_tool', + ); + // A new provider attempt proves the operation continued past the tool + // cancellation, so the tool evidence is stale and cleared. A later abort + // classifies during_api (this is the corrected contract; the prior pass + // incorrectly retained the tool evidence). + registry.onProviderAttemptStart({ + attemptId: 'a1', + startMs: 0, + promptId: 'sess#agentic-loop#uuid', + }); + expect(registry.classifyCancellation(controller.signal)).toBe( + 'cancelled_during_api', + ); + }); + + it('overlapping approval waits union into approval_wait_ms', async () => { + let mono = 0; + const { registry, readRecords } = await createStartedRegistry({ + monotonicNow: () => mono, + }); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + mono = 100; + registry.handleToolStatus(controller.signal, 'awaiting-approval', 'c1'); + mono = 120; + registry.handleToolStatus(controller.signal, 'awaiting-approval', 'c2'); + mono = 200; + // Close c1 (approval interval [100,200) = 100). + registry.handleToolStatus(controller.signal, 'success', 'c1'); + mono = 250; + // Close c2 (approval interval [120,250) = 130; overlaps c1). + registry.handleToolStatus(controller.signal, 'success', 'c2'); + mono = 1000; + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + // Union of [100,200) and [120,250) = [100,250) = 150. + expect(records[0].approval_wait_ms).toBeCloseTo(150, 0); + }); +}); + +// --------------------------------------------------------------------------- +// 3. Tool interval honesty +// --------------------------------------------------------------------------- + +describe('P07 tool interval honesty (AC-5)', () => { + it('missing start_ms/end_ms: counts + sums duration but no synthesized interval', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.onToolCallCompleted({ + promptId: 'sess#agentic-loop#uuid', + callId: 't1', + startMs: undefined, + endMs: undefined, + durationMs: 42, + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].tool_calls).toBe(1); + expect(records[0].tool_call_sum_ms).toBe(42); + // No interval was synthesized from monotonicNow. + expect(records[0].tool_union_ms).toBe(0); + }); + + it('present boundaries still union normally', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.onToolCallCompleted({ + promptId: 'sess#agentic-loop#uuid', + callId: 't1', + startMs: 100, + endMs: 200, + durationMs: 100, + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].tool_union_ms).toBe(100); + }); + + it('each unidentifiable completed event (no callId) counts independently', async () => { + const { registry, readRecords } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + // Two events with no callId — no ID is invented, so each counts. + registry.onToolCallCompleted({ + promptId: 'sess#agentic-loop#uuid', + callId: undefined, + startMs: undefined, + endMs: undefined, + durationMs: 10, + }); + registry.onToolCallCompleted({ + promptId: 'sess#agentic-loop#uuid', + callId: undefined, + startMs: undefined, + endMs: undefined, + durationMs: 20, + }); + await registry.finalise(controller.signal, 'completed'); + const records = await readRecords(); + expect(records[0].tool_calls).toBe(2); + expect(records[0].tool_call_sum_ms).toBe(30); + expect(records[0].tool_union_ms).toBe(0); + }); +}); + +// --------------------------------------------------------------------------- +// 4. Observer ownership — fail-fast on double install, identity-safe dispose +// --------------------------------------------------------------------------- + +describe('P07 observer ownership (D8/AC-2)', () => { + it('installObservers is idempotent for the same registry', async () => { + const { registry } = await createStartedRegistry(); + // Already installed in createStartedRegistry. + expect(() => registry.installObservers()).not.toThrow(); + expect(getPerfPhaseObserver()).toBe(registry); + }); + + it('a second registry installing while another owns throws (fail-fast)', async () => { + const { registry } = await createStartedRegistry(); + const retention = new PerfRetention({ + dir, + runUuid: crypto.randomUUID(), + maintenanceIntervalMs: 60_000, + }); + const sink = new PerfSink({ + dir, + runUuid: crypto.randomUUID(), + retention, + }); + await sink.start(); + const other = new OperationLifecycleRegistry({ + identityProvider: fixtureProvider(), + sink, + retention, + }); + expect(() => other.installObservers()).toThrow('single interactive owner'); + // The first registry still owns the observers. + expect(getPerfPhaseObserver()).toBe(registry); + await sink.dispose(); + }); + + it('dispose clears only its own observers (identity-safe)', async () => { + const retention = new PerfRetention({ + dir, + runUuid: crypto.randomUUID(), + maintenanceIntervalMs: 60_000, + }); + const sink = new PerfSink({ + dir, + runUuid: crypto.randomUUID(), + retention, + }); + await sink.start(); + const owner = new OperationLifecycleRegistry({ + identityProvider: fixtureProvider(), + sink, + retention, + }); + owner.installObservers(); + // A non-owner registry that never installed must not clear the owner's + // observers when it disposes. + const nonOwner = new OperationLifecycleRegistry({ + identityProvider: fixtureProvider(), + sink, + retention, + }); + await nonOwner.dispose(); + expect(getPerfPhaseObserver()).toBe(owner); + expect(getInteractiveRenderObserver()).toBe(owner); + expect(getInteractiveStdoutObserver()).toBe(owner); + await owner.dispose(); + await sink.dispose(); + }); +}); + +// --------------------------------------------------------------------------- +// 5. Stale terminal cancellation evidence — provider-start clearing (AC-4) +// --------------------------------------------------------------------------- + +describe('P07 stale terminal cancellation evidence (AC-4)', () => { + it('a tool-status cancelled terminal is stale once a new provider attempt proves the operation continued', async () => { + const { registry } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.enterApiPhase(controller.signal); + registry.handleToolStatus(controller.signal, 'executing', 'c1'); + registry.handleToolStatus(controller.signal, 'cancelled', 'c1'); + // Retained BEFORE the operation continues. + expect(registry.classifyCancellation(controller.signal)).toBe( + 'cancelled_during_tool', + ); + // A new provider attempt for the same operation proves the tool + // cancellation did NOT terminate the operation → stale evidence cleared, + // so a later independent API abort classifies during_api. + registry.onProviderAttemptStart({ + attemptId: 'a1', + startMs: 0, + promptId: 'sess#agentic-loop#uuid', + }); + expect(registry.classifyCancellation(controller.signal)).toBe( + 'cancelled_during_api', + ); + // A later independent API abort now classifies during_api (not during_tool). + registry.onProviderAttemptEnd({ + attemptId: 'a1', + startMs: 0, + endMs: 10, + status: 'aborted', + inputTokens: 0, + outputTokens: 0, + promptId: 'sess#agentic-loop#uuid', + }); + expect(registry.classifyCancellation(controller.signal)).toBe( + 'cancelled_during_api', + ); + }); + + it('clearing stale evidence does NOT clear current active tool state', async () => { + const { registry } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.enterApiPhase(controller.signal); + // Two tools active; c1 is cancelled (terminal) but c2 stays active. + registry.handleToolStatus(controller.signal, 'executing', 'c1'); + registry.handleToolStatus(controller.signal, 'executing', 'c2'); + registry.handleToolStatus(controller.signal, 'cancelled', 'c1'); + // A new provider attempt clears the stale tool evidence from c1's cancel. + registry.onProviderAttemptStart({ + attemptId: 'a1', + startMs: 0, + promptId: 'sess#agentic-loop#uuid', + }); + // c2 is STILL active → classification remains during_tool. + expect(registry.classifyCancellation(controller.signal)).toBe( + 'cancelled_during_tool', + ); + }); + + it('an approval-status cancelled terminal is also cleared by a provider start', async () => { + const { registry } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.enterApiPhase(controller.signal); + registry.handleToolStatus(controller.signal, 'awaiting-approval', 'c1'); + registry.handleToolStatus(controller.signal, 'cancelled', 'c1'); + expect(registry.classifyCancellation(controller.signal)).toBe( + 'cancelled_during_approval', + ); + registry.onProviderAttemptStart({ + attemptId: 'a1', + startMs: 0, + promptId: 'sess#agentic-loop#uuid', + }); + expect(registry.classifyCancellation(controller.signal)).toBe( + 'cancelled_during_api', + ); + }); + + it('provider-aborted (api) terminal evidence is preserved across a later provider attempt start', async () => { + const { registry } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.enterApiPhase(controller.signal); + registry.onProviderAttemptStart({ + attemptId: 'a1', + startMs: 0, + promptId: 'sess#agentic-loop#uuid', + }); + registry.onProviderAttemptEnd({ + attemptId: 'a1', + startMs: 0, + endMs: 50, + status: 'aborted', + inputTokens: 0, + outputTokens: 0, + promptId: 'sess#agentic-loop#uuid', + }); + expect(registry.classifyCancellation(controller.signal)).toBe( + 'cancelled_during_api', + ); + // A retry attempt must NOT clear the provider-aborted api evidence. + registry.onProviderAttemptStart({ + attemptId: 'a2', + startMs: 100, + promptId: 'sess#agentic-loop#uuid', + }); + expect(registry.classifyCancellation(controller.signal)).toBe( + 'cancelled_during_api', + ); + }); + + it('overlap precedence preserved: a tool cancel upgrades retained api evidence (tool > api)', async () => { + const { registry } = await createStartedRegistry(); + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + registry.enterApiPhase(controller.signal); + // API evidence retained first via a provider abort (no prior tool evidence, + // so the start is a no-op clear). + registry.onProviderAttemptStart({ + attemptId: 'a1', + startMs: 0, + promptId: 'sess#agentic-loop#uuid', + }); + registry.onProviderAttemptEnd({ + attemptId: 'a1', + startMs: 0, + endMs: 50, + status: 'aborted', + inputTokens: 0, + outputTokens: 0, + promptId: 'sess#agentic-loop#uuid', + }); + // A tool-status cancel for a still-active tool UPGRADES the retained + // evidence to tool (overlap precedence: tool > api). No provider start + // intervenes after the tool cancel, so the tool evidence is NOT stale. + registry.handleToolStatus(controller.signal, 'executing', 'c1'); + registry.handleToolStatus(controller.signal, 'cancelled', 'c1'); + expect(registry.classifyCancellation(controller.signal)).toBe( + 'cancelled_during_tool', + ); + }); +}); diff --git a/packages/cli/src/ui/hooks/agentStream/operationLifecycle.p10.memory.behavior.test.ts b/packages/cli/src/ui/hooks/agentStream/operationLifecycle.p10.memory.behavior.test.ts new file mode 100644 index 0000000000..8ac7b779be --- /dev/null +++ b/packages/cli/src/ui/hooks/agentStream/operationLifecycle.p10.memory.behavior.test.ts @@ -0,0 +1,248 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * P10 behavioral tests for OperationLifecycleRegistry memory columns (AC-10). + * + * With a memory sampler present: operation records carry the four memory + * columns (rss_bytes, heap_used_bytes, external_bytes, array_buffers_bytes). + * Without a sampler: all four fields are omitted (absent, never zeros). + * + * Real registry + real PerfSink/PerfRetention + real MemoryTelemetryController + * + real temp files + real reader. No mock theatre. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { + PerfSink, + PerfRetention, +} from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import { readPerfRecords } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import type { PerfOperationRecord } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import { + OperationLifecycleRegistry, + type OperationIdentityProvider, + type OperationIdentitySnapshot, +} from './operationLifecycle.js'; +import { MemoryTelemetryController } from '../memoryTrend/memoryTelemetry.js'; + +let dir: string; + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'op-p10-mem-')); +}); + +afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); +}); + +function fixtureIdentity(): OperationIdentitySnapshot { + return { + session_id: 'sess-abc', + runtime_id: 'rt-main', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'sha256:project-hash', + llxprt_version: '0.11.0', + git_sha: 'abc1234', + runtime: 'bun-1.3.14', + platform: 'darwin-arm64', + provider: 'openai', + model: 'gpt-4o', + terminal_cols: 120, + terminal_rows: 40, + render_mode: 'incremental', + }; +} + +function fixtureProvider(): OperationIdentityProvider { + const snap = fixtureIdentity(); + return { snapshot: () => snap }; +} + +function fixtureMemory(rss: number): NodeJS.MemoryUsage { + return { + rss, + heapUsed: rss + 1000, + external: rss + 2000, + arrayBuffers: rss + 3000, + heapTotal: rss + 5000, + } as NodeJS.MemoryUsage; +} + +async function readOpRecords( + registry: OperationLifecycleRegistry, + sink: PerfSink, +): Promise { + await registry.drain(); + await sink.dispose(); + const files = fs.readdirSync(dir).filter((f) => f.endsWith('.jsonl')); + const records: PerfOperationRecord[] = []; + for (const file of files) { + const result = await readPerfRecords(path.join(dir, file)); + for (const rec of result.records) { + if (rec.record_type === 'operation') { + records.push(rec); + } + } + } + return records; +} + +describe('P10 operation record memory columns (AC-10)', () => { + it('memory ON: operation record includes all four memory columns', async () => { + const runUuid = crypto.randomUUID(); + const retention = new PerfRetention({ + dir, + runUuid, + maintenanceIntervalMs: 60_000, + }); + const sink = new PerfSink({ + dir, + runUuid, + retention, + }); + await sink.start(); + const controller = new MemoryTelemetryController({ + sink, + monotonicNow: () => 5000, + memoryNow: () => fixtureMemory(42_000_000), + }); + const registry = new OperationLifecycleRegistry({ + identityProvider: fixtureProvider(), + sink, + retention, + memorySampler: controller, + }); + + const controller2 = new AbortController(); + registry.begin(controller2.signal, 'sess#agentic-loop#uuid'); + await registry.finalise(controller2.signal, 'completed'); + const records = await readOpRecords(registry, sink); + expect(records).toHaveLength(1); + expect(records[0].rss_bytes).toBe(42_000_000); + expect(records[0].heap_used_bytes).toBe(42_001_000); + expect(records[0].external_bytes).toBe(42_002_000); + expect(records[0].array_buffers_bytes).toBe(42_003_000); + }); + + it('memory OFF (no sampler): operation record omits all four columns', async () => { + const runUuid = crypto.randomUUID(); + const retention = new PerfRetention({ + dir, + runUuid, + maintenanceIntervalMs: 60_000, + }); + const sink = new PerfSink({ + dir, + runUuid, + retention, + }); + await sink.start(); + const registry = new OperationLifecycleRegistry({ + identityProvider: fixtureProvider(), + sink, + retention, + // No memorySampler — memory disabled. + }); + + const controller = new AbortController(); + registry.begin(controller.signal, 'sess#agentic-loop#uuid'); + await registry.finalise(controller.signal, 'completed'); + const records = await readOpRecords(registry, sink); + expect(records).toHaveLength(1); + expect(records[0].rss_bytes).toBeUndefined(); + expect(records[0].heap_used_bytes).toBeUndefined(); + expect(records[0].external_bytes).toBeUndefined(); + expect(records[0].array_buffers_bytes).toBeUndefined(); + }); + + it('memory ON: markOperationEnd is called at finalisation', async () => { + let mono = 10_000; + const runUuid = crypto.randomUUID(); + const retention = new PerfRetention({ + dir, + runUuid, + maintenanceIntervalMs: 60_000, + }); + const sink = new PerfSink({ + dir, + runUuid, + retention, + }); + await sink.start(); + const controller = new MemoryTelemetryController({ + sink, + monotonicNow: () => mono, + memoryNow: () => fixtureMemory(1000), + }); + const registry = new OperationLifecycleRegistry({ + identityProvider: fixtureProvider(), + sink, + retention, + memorySampler: controller, + monotonicNow: () => mono, + }); + + try { + // Operation ends at uptime 10_000 — markOperationEnd sets it. + const c = new AbortController(); + registry.begin(c.signal, 'sess#agentic-loop#uuid'); + await registry.finalise(c.signal, 'completed'); + + // Now a tick sample at uptime 20_000 should have idle = 10_000. + mono = 20_000; + controller.recordTickSample(fixtureMemory(2000)); + const snap = controller.snapshot(); + expect(snap).toHaveLength(1); + expect(snap[0].msSinceLastOperation).toBe(10_000); + } finally { + await controller.drain(); + await sink.dispose(); + } + }); + + it('no slope key in persisted operation record', async () => { + const runUuid = crypto.randomUUID(); + const retention = new PerfRetention({ + dir, + runUuid, + maintenanceIntervalMs: 60_000, + }); + const sink = new PerfSink({ + dir, + runUuid, + retention, + }); + await sink.start(); + const controller = new MemoryTelemetryController({ + sink, + monotonicNow: () => 5000, + memoryNow: () => fixtureMemory(42_000_000), + }); + const registry = new OperationLifecycleRegistry({ + identityProvider: fixtureProvider(), + sink, + retention, + memorySampler: controller, + }); + + const c = new AbortController(); + registry.begin(c.signal, 'sess#agentic-loop#uuid'); + await registry.finalise(c.signal, 'completed'); + await registry.drain(); + await sink.dispose(); + + const files = fs.readdirSync(dir).filter((f) => f.endsWith('.jsonl')); + const raw = JSON.parse(fs.readFileSync(path.join(dir, files[0]), 'utf8')); + for (const key of Object.keys(raw)) { + expect(key).not.toMatch(/slope/i); + } + }); +}); diff --git a/packages/cli/src/ui/hooks/agentStream/operationLifecycle.snapshot.behavior.test.ts b/packages/cli/src/ui/hooks/agentStream/operationLifecycle.snapshot.behavior.test.ts new file mode 100644 index 0000000000..aead3c3e7e --- /dev/null +++ b/packages/cli/src/ui/hooks/agentStream/operationLifecycle.snapshot.behavior.test.ts @@ -0,0 +1,383 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Behavioral tests for the synchronous immutable terminal snapshot (Finding A, + * issue #3167). + * + * Proves that exactly-once finalisation atomically claims the operation and + * synchronously freezes one immutable terminal snapshot before returning/ + * queueing. The queued async work does ONLY external claim counting and sink + * persistence, receiving a frozen copy — never a mutable PendingOp/measurement + * reference. + * + * Real registry + real PerfSink/PerfRetention + real temp files. No mocks. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import * as crypto from 'node:crypto'; +import { + PerfSink, + PerfRetention, +} from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import { readPerfRecords } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import type { PerfOperationRecord } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import { + OperationLifecycleRegistry, + type OperationIdentityProvider, + type OperationIdentitySnapshot, +} from './operationLifecycle.js'; + +let dir: string; + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'op-snapshot-')); +}); + +afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); +}); + +function fixtureIdentity(): OperationIdentitySnapshot { + return { + session_id: 'sess-snap', + runtime_id: 'rt-snap', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'sha256:snap-hash', + llxprt_version: '0.11.0', + git_sha: 'snap1234', + runtime: 'bun-1.3.14', + platform: 'darwin-arm64', + provider: 'openai', + model: 'gpt-4o', + terminal_cols: 120, + terminal_rows: 40, + render_mode: 'incremental', + }; +} + +function fixtureProvider(): OperationIdentityProvider { + return { snapshot: () => fixtureIdentity() }; +} + +async function makeStartedRegistry( + overrides: { + monotonicNow?: () => number; + wallNow?: () => number; + identityProvider?: OperationIdentityProvider; + } = {}, +): Promise<{ + registry: OperationLifecycleRegistry; + sink: PerfSink; + retention: PerfRetention; +}> { + const retention = new PerfRetention({ + dir, + runUuid: crypto.randomUUID(), + maintenanceIntervalMs: 60_000, + }); + const sink = new PerfSink({ + dir, + runUuid: crypto.randomUUID(), + retention, + }); + await sink.start(); + const registry = new OperationLifecycleRegistry({ + identityProvider: overrides.identityProvider ?? fixtureProvider(), + sink, + retention, + monotonicNow: overrides.monotonicNow, + wallNow: overrides.wallNow, + }); + return { registry, sink, retention }; +} + +async function readAllRecords(): Promise { + const files = fs.existsSync(dir) + ? fs.readdirSync(dir).filter((f) => f.endsWith('.jsonl')) + : []; + const records: PerfOperationRecord[] = []; + for (const file of files) { + const result = await readPerfRecords(path.join(dir, file)); + for (const rec of result.records) { + if (rec.record_type === 'operation') { + records.push(rec); + } + } + } + return records; +} + +// --------------------------------------------------------------------------- +// Mutations immediately after finalise cannot change persisted record +// --------------------------------------------------------------------------- + +describe('Finding A — immutable terminal snapshot', () => { + it('mutations to the measurement handle immediately after finalise cannot change the persisted record', async () => { + const { registry, sink } = await makeStartedRegistry(); + const ac = new AbortController(); + const handle = registry.begin(ac.signal, 'sess-1#agentic-loop#uuid-1'); + handle.measurement.context_tokens = 1000; + handle.measurement.output_tokens = 500; + handle.measurement.client_prepare_ms = 10; + + // finalise returns a promise; the snapshot is frozen synchronously BEFORE + // the promise resolves. We capture the promise but do NOT await it yet. + const writePromise = registry.finalise(ac.signal, 'completed'); + + // Mutate the handle's measurement reference immediately after finalise + // returns (before the async write resolves). This mutation must NOT + // affect the frozen snapshot. + handle.measurement.context_tokens = 999_999; + handle.measurement.output_tokens = 888_888; + handle.measurement.client_prepare_ms = 777; + + await writePromise; + await sink.dispose(); + + const records = await readAllRecords(); + expect(records).toHaveLength(1); + expect(records[0].context_tokens).toBe(1000); + expect(records[0].output_tokens).toBe(500); + expect(records[0].client_prepare_ms).toBe(10); + }); + + it('copies begin-time identity before queued persistence', async () => { + const mutableIdentity = { ...fixtureIdentity() }; + const identityProvider: OperationIdentityProvider = { + snapshot: () => mutableIdentity, + }; + const { registry, sink } = await makeStartedRegistry({ identityProvider }); + const ac = new AbortController(); + registry.begin(ac.signal, 'sess-identity#agentic-loop#uuid-1'); + + const writePromise = registry.finalise(ac.signal, 'completed'); + mutableIdentity.provider = 'mutated-provider'; + mutableIdentity.model = 'mutated-model'; + + await writePromise; + await sink.dispose(); + + const records = await readAllRecords(); + expect(records).toHaveLength(1); + expect(records[0].provider).toBe('openai'); + expect(records[0].model).toBe('gpt-4o'); + }); + + it('a delayed countNonStaleClaims cannot alter terminal timestamp/elapsed/status/measurements', async () => { + // Use a retention subclass that delays countNonStaleClaims until released. + // This proves the async claim-counting path cannot re-read mutable state. + const retention = new PerfRetention({ + dir, + runUuid: crypto.randomUUID(), + maintenanceIntervalMs: 60_000, + }); + let releaseClaims: () => void = () => {}; + const claimsGate = new Promise((resolve) => { + releaseClaims = resolve; + }); + const delayedRetention = Object.create(retention) as PerfRetention; + delayedRetention.countNonStaleClaims = async ( + now: number, + ): Promise => { + await claimsGate; + return retention.countNonStaleClaims(now); + }; + + const sink = new PerfSink({ + dir, + runUuid: crypto.randomUUID(), + retention: delayedRetention, + }); + await sink.start(); + + // Controllable clocks for deterministic elapsed. + let mono = 1000; + const monotonicNow = (): number => { + const v = mono; + mono += 100; + return v; + }; + const wallNow = (): number => 2_000_000_000_000; + + const registry = new OperationLifecycleRegistry({ + identityProvider: fixtureProvider(), + sink, + retention: delayedRetention, + monotonicNow, + wallNow, + }); + + const ac = new AbortController(); + const handle = registry.begin(ac.signal, 'sess-1#agentic-loop#uuid-1'); + handle.measurement.context_tokens = 2000; + handle.measurement.client_prepare_ms = 5; + + // finalise freezes the snapshot synchronously. The async write is blocked + // on claimsGate. + const writePromise = registry.finalise(ac.signal, 'completed'); + + // While the async claim counting is pending, mutate the live measurement. + handle.measurement.context_tokens = 424_242; + handle.measurement.client_prepare_ms = 999; + + // Release the gate and drain. + releaseClaims(); + await writePromise; + await sink.dispose(); + + const records = await readAllRecords(); + expect(records).toHaveLength(1); + // The frozen values must be preserved. + expect(records[0].context_tokens).toBe(2000); + expect(records[0].client_prepare_ms).toBe(5); + expect(records[0].status).toBe('completed'); + // Elapsed is frozen from the snapshot, not re-read later. + expect(records[0].operation_elapsed_ms).toBeGreaterThan(0); + }); + + it('client_finalize and elapsed/residual use coherent clocks', async () => { + // With controllable monotonic clock, the finalization boundary is shared. + // The residual = elapsed - client_prepare - ... - client_finalize - ... + // Since elapsed includes the finalize boundary, residual is coherent. + let mono = 500; + const monotonicNow = (): number => { + const v = mono; + mono += 50; // each call advances 50ms + return v; + }; + const wallNow = (): number => 1_700_000_000_000; + + const { registry, sink } = await makeStartedRegistry({ + monotonicNow, + wallNow, + }); + const ac = new AbortController(); + const handle = registry.begin(ac.signal, 'sess-1#agentic-loop#uuid-1'); + // Set a known client_prepare so residual is predictable. + handle.measurement.client_prepare_ms = 0; + + await registry.finalise(ac.signal, 'completed'); + await sink.dispose(); + + const records = await readAllRecords(); + expect(records).toHaveLength(1); + const rec = records[0]; + + // Coherent: elapsed >= client_finalize_ms (the finalize boundary is + // included in elapsed). With the controllable clock, begin takes 50ms + // (mono 500→550 for startedAtMonotonic... actually begin captures startedAt + // at one call). The exact values depend on how many monotonicNow calls + // happen, but the INVARIANT is: residual = elapsed - all measured phases. + const computedResidual = + rec.operation_elapsed_ms - + rec.client_prepare_ms - + rec.stream_handler_ms - + rec.ink_render_ms - + rec.stdout_write_sync_ms - + rec.client_finalize_ms - + rec.approval_wait_ms; + expect(rec.unclassified_elapsed_ms).toBeCloseTo(computedResidual, 5); + + // elapsed MUST include the finalization boundary: elapsed >= + // client_finalize_ms (otherwise finalize work was subtracted without being + // included in elapsed, making the residual artificially low). + expect(rec.operation_elapsed_ms).toBeGreaterThanOrEqual( + rec.client_finalize_ms, + ); + }); + + it('superseded snapshots are frozen (sweep path uses the same immutable snapshot)', async () => { + const { registry, sink } = await makeStartedRegistry(); + const ac1 = new AbortController(); + const handle1 = registry.begin(ac1.signal, 'sess-1#agentic-loop#uuid-1'); + handle1.measurement.context_tokens = 3000; + handle1.measurement.tool_calls = 7; + + // A new begin sweeps ac1 as superseded, freezing its snapshot + // synchronously. + const ac2 = new AbortController(); + registry.begin(ac2.signal, 'sess-1#agentic-loop#uuid-2'); + + // Mutate handle1 after the sweep — the superseded snapshot is already + // frozen. + handle1.measurement.context_tokens = 111_111; + handle1.measurement.tool_calls = 999; + + await registry.finalise(ac2.signal, 'completed'); + await sink.dispose(); + + const records = await readAllRecords(); + expect(records).toHaveLength(2); + const superseded = records.find((r) => r.status === 'superseded'); + expect(superseded).toBeDefined(); + expect(superseded!.context_tokens).toBe(3000); + expect(superseded!.tool_calls).toBe(7); + }); + + it('cancellation evidence works after active-map removal without strong retention', async () => { + // Behavioral proof that retained cancellation evidence survives active-map + // removal. A WeakMap (not a strong Map) holds the evidence keyed by signal; + // it persists as long as the caller holds the AbortController/signal. + const { registry, sink } = await makeStartedRegistry(); + const ac1 = new AbortController(); + registry.begin(ac1.signal, 'sess-1#agentic-loop#uuid-1'); + + // Simulate a tool-status cancelled terminal: retains 'tool' evidence. + registry.handleToolStatus(ac1.signal, 'cancelled', 'call-xyz'); + + // A new begin sweeps ac1 out of the active map (superseded). ac1.signal is + // no longer in the active map. + const ac2 = new AbortController(); + registry.begin(ac2.signal, 'sess-1#agentic-loop#uuid-2'); + + // classifyCancellation must still return the retained phase — evidence + // survived active-map removal. The signal is still held by ac1. + const status = registry.classifyCancellation(ac1.signal); + expect(status).toBe('cancelled_during_tool'); + + await registry.finalise(ac2.signal, 'completed'); + await sink.dispose(); + }); + + it('missing tool boundaries record count/sum without synthesizing intervals (status transitions do not synthesize tool intervals)', async () => { + const { registry, sink } = await makeStartedRegistry(); + const ac = new AbortController(); + registry.begin(ac.signal, 'sess-1#agentic-loop#uuid-1'); + + // A tool call completed with missing boundaries (no start/end). The count + // and sum are recorded, but NO interval is synthesized. + registry.onToolCallCompleted({ + promptId: 'sess-1#agentic-loop#uuid-1', + callId: 'tool-no-bounds', + durationMs: 42, + startMs: undefined, + endMs: undefined, + }); + + // CLI status transitions (e.g. scheduled→executing→success) must NOT + // synthesize tool intervals. + registry.handleToolStatus(ac.signal, 'scheduled', 'tool-no-bounds'); + registry.handleToolStatus(ac.signal, 'executing', 'tool-no-bounds'); + registry.handleToolStatus(ac.signal, 'success', 'tool-no-bounds'); + + await registry.finalise(ac.signal, 'completed'); + await sink.dispose(); + + const records = await readAllRecords(); + expect(records).toHaveLength(1); + const rec = records[0]; + // Count and sum recorded honestly. + expect(rec.tool_calls).toBe(1); + expect(rec.tool_call_sum_ms).toBe(42); + // No interval was synthesized → tool_union_ms is 0. + expect(rec.tool_union_ms).toBe(0); + }); +}); diff --git a/packages/cli/src/ui/hooks/agentStream/operationLifecycle.ts b/packages/cli/src/ui/hooks/agentStream/operationLifecycle.ts new file mode 100644 index 0000000000..2798846f71 --- /dev/null +++ b/packages/cli/src/ui/hooks/agentStream/operationLifecycle.ts @@ -0,0 +1,1061 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Operation lifecycle registry + immutable identity snapshot (P06, issue #3167). + * + * A constructible CLI-owned registry keyed by AbortSignal that owns the + * exactly-once lifecycle of a perf operation record. `begin` derives + * operation_id, snapshots immutable identity/build/dimensions, and initializes + * a mutable per-operation measurement state suitable for P07 phase accumulation. + * `finalise` atomically claims the pending op, derives concurrent_instances + * from non-stale claim files (D3), builds one schema-valid v1 record, and writes + * through PerfSink. A superseded sweep on every new `begin` finalises displaced + * still-active ops as `superseded` exactly once — the stale ownership finally + * block in useSubmitQuery cannot reach them because isCurrentTurn is false. + * + * Disabled mode is achieved by the runtime not constructing this registry + * (AC-2); there are no hidden global side effects. + * + * Decisions D1/D3 applied: operation_id is the sole join key (no child + * prompt_ids/turn_ids arrays); concurrent_instances from claim-file lease + * semantics. + */ + +import { + deriveOperationId, + PERF_RECORD_TYPE_OPERATION, + PERF_SCHEMA_VERSION, +} from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import type { + PerfOperationRecord, + PerfTerminalStatus, +} from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import type { + PerfSink, + PerfRetention, +} from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import type { + OperationMemorySampler, + MemoryColumns, +} from '../memoryTrend/memoryTelemetry.js'; +import { IntervalUnion } from '@vybestack/llxprt-code-telemetry/telemetry/intervalUnion.js'; +import type { + PerfProviderAttemptEndInfo, + PerfToolCallCompletedInfo, +} from '@vybestack/llxprt-code-telemetry/perf/perfPhaseObserver.js'; +import { + setPerfPhaseObserver, + getPerfPhaseObserver, +} from '@vybestack/llxprt-code-telemetry/perf/perfPhaseObserver.js'; +import { + setInteractiveRenderObserver, + setInteractiveStdoutObserver, + getInteractiveRenderObserver, + getInteractiveStdoutObserver, +} from '../../inkRenderOptions.js'; + +// --------------------------------------------------------------------------- +// Immutable identity snapshot contract (P12 constructs the provider) +// --------------------------------------------------------------------------- + +/** + * Narrow immutable snapshot of the identity/build/comparison-dimension fields + * captured at operation `begin` time. All fields are non-empty strings or + * schema-conformant numbers; P12 constructs the provider from actual + * runtime/config/build APIs. No new CLI flags or env vars are invented here. + */ +export interface OperationIdentitySnapshot { + readonly session_id: string; + readonly runtime_id: string; + readonly parent_runtime_id: string | null; + readonly subagent_name: string | null; + readonly project_hash: string; + readonly llxprt_version: string; + readonly git_sha: string; + readonly runtime: string; + readonly platform: string; + readonly provider: string; + readonly model: string; + readonly terminal_cols: number; + readonly terminal_rows: number; + readonly render_mode: string; +} + +/** + * Provider that returns a fresh immutable identity snapshot. P12 wires this + * from the real runtime/config/build identity APIs; tests supply a fixture. + */ +export interface OperationIdentityProvider { + snapshot(): OperationIdentitySnapshot; +} + +// --------------------------------------------------------------------------- +// Mutable per-operation measurement state (P07 accumulates into this) +// --------------------------------------------------------------------------- + +/** + * Mutable per-operation measurement state. All fields begin at zero/default in + * P06; P07 accumulates directly-measured client phases, provider/tool sums and + * unions, and token counts into this object through the typed handle before + * finalization. + */ +export interface OperationMeasurement { + client_prepare_ms: number; + stream_handler_ms: number; + ink_render_ms: number; + ink_render_count: number; + stdout_bytes: number; + stdout_write_calls: number; + stdout_write_sync_ms: number; + client_finalize_ms: number; + provider_attempts: number; + provider_attempt_sum_ms: number; + provider_union_ms: number; + tool_calls: number; + tool_call_sum_ms: number; + tool_union_ms: number; + agent_activity_union_ms: number; + approval_wait_ms: number; + context_tokens: number; + output_tokens: number; +} + +// --------------------------------------------------------------------------- +// Handle + status +// --------------------------------------------------------------------------- + +/** + * Handle returned by {@link OperationLifecycleRegistry.begin}. The + * `measurement` field is the typed mutable state P07 accumulates into; + * `sessionOperationIndex` is the monotonic per-session index assigned at begin. + */ +export interface OperationHandle { + readonly signal: AbortSignal; + readonly operationId: string; + readonly measurement: OperationMeasurement; + readonly sessionOperationIndex: number; +} + +/** + * The seven terminal operation statuses (spec §1.3), including `superseded`. + */ +export type OperationStatus = PerfTerminalStatus; + +// --------------------------------------------------------------------------- +// Registry options +// --------------------------------------------------------------------------- + +export interface OperationLifecycleRegistryOptions { + readonly identityProvider: OperationIdentityProvider; + readonly sink: PerfSink; + readonly retention: PerfRetention; + /** Wall-clock epoch millis for the record `ts`. Defaults to Date.now. */ + readonly wallNow?: () => number; + /** Monotonic millis for elapsed/uptime. Defaults to performance.now. */ + readonly monotonicNow?: () => number; + /** + * Optional memory sampler (P10). Present only when memory telemetry is + * enabled. At exactly-once finalisation, the registry marks operation-end + * and samples process.memoryUsage() once to include the four memory columns. + * Absent/disabled ⇒ all four fields are omitted (never zeros). P12 wires + * this based on real settings. + */ + readonly memorySampler?: OperationMemorySampler; +} + +// --------------------------------------------------------------------------- +// Live phase tracking for granular cancellation classification (P07, AC-4) +// --------------------------------------------------------------------------- + +/** + * The live operation phase used to classify AbortSignal cancellation into the + * granular `cancelled_during_*` statuses. + * + * Deterministic precedence for overlap: `approval` > `tool` > `api`. When a + * cancellation occurs, the most-specific active phase wins. This is documented + * and tested so concurrent phases resolve deterministically. + */ +type LivePhase = 'api' | 'tool' | 'approval'; + +const PHASE_PRECEDENCE: readonly LivePhase[] = ['approval', 'tool', 'api']; + +function phaseToCancelledStatus(phase: LivePhase): OperationStatus { + if (phase === 'approval') return 'cancelled_during_approval'; + if (phase === 'tool') return 'cancelled_during_tool'; + return 'cancelled_during_api'; +} + +/** + * The tool-status lifecycle values projected by the Agent's public event + * stream (ToolUpdate.status). Defined locally so the registry need not depend + * on the agents package's type; the orchestration narrows the AgentEvent and + * forwards the status string. + */ +type ToolStatusValue = + | 'validating' + | 'scheduled' + | 'awaiting-approval' + | 'executing' + | 'success' + | 'error' + | 'cancelled'; + +/** + * Minimal structural view of an AgentEvent for perf observation. The + * orchestration narrows the real AgentEvent to this shape (structurally + * compatible), so the registry need not depend on the agents package's types. + * Only tool-status events carry phase-relevant state. + */ +export interface ObservableAgentEvent { + readonly type: string; + readonly update?: { + readonly id: string; + readonly status: ToolStatusValue; + }; +} + +// --------------------------------------------------------------------------- +// Internal pending-op +// --------------------------------------------------------------------------- + +interface PendingOp { + readonly operationId: string; + readonly identity: OperationIdentitySnapshot; + readonly startedAtMonotonic: number; + readonly index: number; + readonly measurement: OperationMeasurement; + /** Interval union of provider attempt boundaries (P07). */ + readonly providerIntervals: IntervalUnion; + /** Interval union of tool call boundaries (P07). */ + readonly toolIntervals: IntervalUnion; + /** Dedup set for provider attempt IDs (exactly-once). */ + readonly providerSeen: Set; + /** Dedup set for tool call IDs (exactly-once). */ + readonly toolSeen: Set; + /** The AbortSignal owning this op (for evidence persistence after removal). */ + readonly signal: AbortSignal; + /** CallIds currently awaiting approval → approval-wait start monotonic. */ + readonly approvalStarts: Map; + /** CallIds currently in an active tool phase (scheduled/executing). */ + readonly activeToolCallIds: Set; + /** Whether the API/stream phase is active. */ + apiActive: boolean; + /** Interval union of closed approval-wait intervals (overlapping-safe). */ + readonly approvalWaitIntervals: IntervalUnion; + /** Retained cancellation evidence (only set by cancellation events). */ + cancellationEvidence: LivePhase | null; +} + +function zeroMeasurement(): OperationMeasurement { + return { + client_prepare_ms: 0, + stream_handler_ms: 0, + ink_render_ms: 0, + ink_render_count: 0, + stdout_bytes: 0, + stdout_write_calls: 0, + stdout_write_sync_ms: 0, + client_finalize_ms: 0, + provider_attempts: 0, + provider_attempt_sum_ms: 0, + provider_union_ms: 0, + tool_calls: 0, + tool_call_sum_ms: 0, + tool_union_ms: 0, + agent_activity_union_ms: 0, + approval_wait_ms: 0, + context_tokens: 0, + output_tokens: 0, + }; +} + +// --------------------------------------------------------------------------- +// Frozen terminal snapshot (immutable record captured at claim time) +// --------------------------------------------------------------------------- + +/** + * Immutable terminal snapshot captured synchronously at exactly-once claim + * time. Contains every field needed to build the final record EXCEPT + * concurrent_instances (which requires async claim counting). The queued + * async write path receives this frozen object and never touches the mutable + * PendingOp/measurement, so mutations after finalise cannot affect the + * persisted record. + */ +interface FrozenTerminalSnapshot { + readonly operationId: string; + readonly identity: OperationIdentitySnapshot; + readonly index: number; + readonly status: OperationStatus; + readonly wallIso: string; + readonly wallNowMs: number; + readonly elapsedMs: number; + readonly uptimeMs: number; + readonly measurement: Readonly; + readonly residual: number; + readonly memoryColumns: MemoryColumns | null; +} + +// --------------------------------------------------------------------------- +// OperationLifecycleRegistry +// --------------------------------------------------------------------------- + +/** + * Constructible CLI-owned registry keyed by AbortSignal. + * + * Not a global singleton: the runtime constructs and installs it only when perf + * is enabled (AC-2). When perf is disabled, the registry is simply absent — + * no hidden global side effects. + */ +export class OperationLifecycleRegistry { + private readonly active = new Map(); + private readonly finalised = new WeakSet(); + /** + * Retained terminal cancellation evidence keyed by signal. Populated ONLY by + * a cancellation terminal signal/event (tool-status cancelled, provider + * attempt aborted end). Persists after the op is removed from `active` so + * classifyCancellation works even if the active marker cleared before the + * catch (P07 AC-4). WeakMap so evidence is reclaimed once the caller drops + * the AbortSignal/AbortController reference, yet survives active-map removal + * as long as the caller still holds the signal. This is NOT a "highest phase + * ever visited" record. + */ + private readonly retainedCancellationEvidence = new WeakMap< + AbortSignal, + LivePhase + >(); + private readonly identityProvider: OperationIdentityProvider; + private readonly sink: PerfSink; + private readonly retention: PerfRetention; + private readonly wallNow: () => number; + private readonly monotonicNow: () => number; + private readonly memorySampler: OperationMemorySampler | null; + + private sessionIndex = -1; + private lifecycleChain: Promise = Promise.resolve(); + private observersInstalled = false; + + constructor(options: OperationLifecycleRegistryOptions) { + this.identityProvider = options.identityProvider; + this.sink = options.sink; + this.retention = options.retention; + this.wallNow = options.wallNow ?? (() => Date.now()); + this.monotonicNow = options.monotonicNow ?? (() => performance.now()); + this.memorySampler = options.memorySampler ?? null; + } + + /** + * Begins a new operation for the given signal and prompt id. + * + * Derives operation_id via `deriveOperationId`, snapshots immutable identity, + * initializes the monotonic per-session index, and creates a mutable + * measurement state for P07. Before admitting the new op, every prior + * still-active signal it displaces is claimed and finalised as `superseded` + * exactly once (the stale ownership finally cannot reach them). + * + * Returns a typed handle. No prompt_ids/turn_ids are collected (D1). + */ + begin(signal: AbortSignal, promptId: string): OperationHandle { + // Superseded sweep: claim every prior still-active op as superseded. + // Synchronous claim (remove from active + mark finalised + freeze one + // immutable terminal snapshot) ensures exactly-once even if the displaced + // turn's explicit finalise races. + const displaced = Array.from(this.active.entries()).filter( + ([activeSignal]) => + activeSignal !== signal && !this.finalised.has(activeSignal), + ); + for (const [activeSignal, op] of displaced) { + this.finalised.add(activeSignal); + this.active.delete(activeSignal); + const snapshot = this.buildFrozenSnapshot(op, 'superseded'); + void this.queueWrite(snapshot); + } + + this.sessionIndex += 1; + const identity = this.identityProvider.snapshot(); + const op: PendingOp = { + operationId: deriveOperationId(promptId), + identity, + startedAtMonotonic: this.monotonicNow(), + index: this.sessionIndex, + measurement: zeroMeasurement(), + providerIntervals: new IntervalUnion(), + toolIntervals: new IntervalUnion(), + providerSeen: new Set(), + toolSeen: new Set(), + signal, + approvalStarts: new Map(), + activeToolCallIds: new Set(), + apiActive: false, + approvalWaitIntervals: new IntervalUnion(), + cancellationEvidence: null, + }; + this.active.set(signal, op); + + return { + signal, + operationId: op.operationId, + measurement: op.measurement, + sessionOperationIndex: op.index, + }; + } + + /** + * Finalises the operation for the given signal exactly once. + * + * Atomically claims/removes the pending op from the active map (and marks the + * signal finalised) and synchronously freezes one immutable terminal + * snapshot BEFORE returning/queueing. The snapshot captures every terminal + * field (wall timestamp, monotonic elapsed/uptime, identity, status, all + * measurement counters/tokens, interval-union durations, approval-wait + * closure, client_finalize_ms, honest residual, operation-end memory) so the + * queued async work receives a frozen copy — never a mutable PendingOp/ + * measurement reference. Then derives concurrent_instances from non-stale + * claim files (D3), clamped to a schema-valid minimum of 1 if the filesystem + * fail-opened to zero, and writes through PerfSink. + * + * A duplicate or late finalise is a no-op (returns a resolved promise). Sink + * filesystem failures remain fail-open (PerfSink handles them); internal + * schema/programming failures reject the returned promise. + */ + finalise(signal: AbortSignal, status: OperationStatus): Promise { + if (this.finalised.has(signal)) { + return Promise.resolve(); + } + const op = this.active.get(signal); + if (op === undefined) { + return Promise.resolve(); + } + this.finalised.add(signal); + this.active.delete(signal); + const snapshot = this.buildFrozenSnapshot(op, status); + return this.queueWrite(snapshot); + } + + /** + * Awaits all queued record writes. Tests and the runtime (before dispose) + * call this to deterministically drain the serialized lifecycle chain so no + * finalised record is lost when the sink is disposed. + */ + async drain(): Promise { + await this.lifecycleChain; + } + + /** + * Read-only snapshot of the currently active foreground operation. Returns + * only current provider/model and monotonic elapsed time — no mutable + * operation state. Returns null when no operation is active. Used by the + * bare `/perf` live snapshot (P12). + */ + getActiveOperationSnapshot(): { + readonly provider: string; + readonly model: string; + readonly elapsedMs: number; + } | null { + const op = this.getFirstActiveOp(); + if (op === undefined) return null; + return { + provider: op.identity.provider, + model: op.identity.model, + elapsedMs: this.monotonicNow() - op.startedAtMonotonic, + }; + } + + // ----------------------------------------------------------------------- + // P07: Observer installation + direct client phase methods + // ----------------------------------------------------------------------- + + /** + * Installs this registry as the perf phase observer (provider/tool events), + * the Ink render observer, and the stdout write observer. Called once when + * the runtime constructs the registry with perf enabled (P12). Default-off: + * when the registry is absent, none of these observers are installed. + * + * Single interactive owner: if ANY observer is already owned by a DIFFERENT + * registry instance, this throws (fail-fast) rather than silently clobbering + * the other registry's observer. Idempotent for the same registry. + */ + installObservers(): void { + if (this.observersInstalled) return; + const perfOwner = getPerfPhaseObserver(); + const renderOwner = getInteractiveRenderObserver(); + const stdoutOwner = getInteractiveStdoutObserver(); + const perfConflict = perfOwner !== null && perfOwner !== this; + const renderConflict = renderOwner !== null && renderOwner !== this; + const stdoutConflict = stdoutOwner !== null && stdoutOwner !== this; + if (perfConflict || renderConflict || stdoutConflict) { + throw new Error( + 'OperationLifecycleRegistry.installObservers: an observer is already ' + + 'owned by a different registry instance (single interactive owner).', + ); + } + this.observersInstalled = true; + setPerfPhaseObserver(this); + setInteractiveRenderObserver(this); + setInteractiveStdoutObserver(this); + } + + /** + * Clears the observers it owns and drains pending writes. Called by the + * runtime on shutdown / perf disable. Identity-safe: each observer is + * cleared ONLY if it still points at this registry, so a non-owner registry + * disposing cannot clear a different registry's observer. Idempotent. + */ + async dispose(): Promise { + if (this.observersInstalled) { + this.observersInstalled = false; + if (getPerfPhaseObserver() === this) setPerfPhaseObserver(null); + if (getInteractiveRenderObserver() === this) { + setInteractiveRenderObserver(null); + } + if (getInteractiveStdoutObserver() === this) { + setInteractiveStdoutObserver(null); + } + } + await this.drain(); + } + + // --- Direct client phase measurement methods --- + + /** + * Sets client_prepare_ms: the monotonic delta from operation begin/acquire + * to immediately before the first `runStream` send. Directly measured by the + * caller (useSubmitQuery) — NOT computed by subtraction. + */ + setClientPrepareMs(signal: AbortSignal, ms: number): void { + const op = this.active.get(signal); + if (op !== undefined) op.measurement.client_prepare_ms = ms; + } + + /** + * Accumulates synchronous CPU time spent dispatching AgentEvents to CLI + * handlers. Instrumented OUTSIDE the generic catch-and-log boundary so a + * perf observer/programming error propagates (D8). Only the sync invocation + * delta is accumulated — not provider/network/tool await time. + */ + addStreamHandlerMs(signal: AbortSignal, ms: number): void { + const op = this.active.get(signal); + if (op !== undefined) op.measurement.stream_handler_ms += ms; + } + + /** + * Captures client_prepare_ms as the monotonic delta from operation begin + * to NOW. Called by useSubmitQuery immediately before the first `runStream` + * send. Uses the registry's own monotonic clock for consistency with + * operation_elapsed_ms. + */ + captureClientPrepare(signal: AbortSignal): void { + const op = this.active.get(signal); + if (op !== undefined) { + op.measurement.client_prepare_ms = + this.monotonicNow() - op.startedAtMonotonic; + } + } + + /** + * Single entry point for perf event observation, invoked OUTSIDE the generic + * processAgentEvent catch (D8: a throw here rejects the stream). Accumulates + * the synchronous handler duration for the op owning `signal` (so + * measurements never hit the wrong op) and routes tool-status transitions to + * {@link handleToolStatus} for live phase tracking. The orchestration narrows + * the real AgentEvent to the structural {@link ObservableAgentEvent} shape. + */ + observeAgentEvent( + event: ObservableAgentEvent, + signal: AbortSignal, + handlerMs: number, + ): void { + this.addStreamHandlerMs(signal, handlerMs); + if (event.type === 'tool-status' && event.update !== undefined) { + this.handleToolStatus(signal, event.update.status, event.update.id); + } + } + + // --- Cancellation phase tracking (AC-4) --- + + /** + * Marks the API/stream phase as active for the operation. Called by + * useSubmitQuery when the stream begins. This is real active state (not + * "highest phase ever visited"): once tools/approvals close and only the API + * phase remains, an abort classifies during_api. + */ + enterApiPhase(signal: AbortSignal): void { + const op = this.active.get(signal); + if (op !== undefined) op.apiActive = true; + } + + /** + * Routes a tool-status lifecycle transition for granular cancellation + * classification, keyed by tool call ID when available. Maintains REAL active + * tool/approval state and unions overlapping approval waits: + * - `validating`/`scheduled`: a tool becomes active. + * - `awaiting-approval`: an approval wait opens (precedence > tool). + * - `executing`: any open approval wait for this call closes; tool executes. + * - `success`/`error`: terminal — closes any open approval wait + tool. + * - `cancelled`: terminal — retains the phase that was cancelled as terminal + * cancellation evidence, then closes the tool/approval. + * + * `approval` evidence precedence is `approval > tool`: a cancellation while + * awaiting approval retains `approval`; otherwise `tool`. + */ + handleToolStatus( + signal: AbortSignal, + status: ToolStatusValue, + callId: string, + ): void { + const op = this.active.get(signal); + if (op === undefined) return; + switch (status) { + case 'validating': + case 'scheduled': + op.activeToolCallIds.add(callId); + break; + case 'awaiting-approval': + if (!op.approvalStarts.has(callId)) { + op.approvalStarts.set(callId, this.monotonicNow()); + } + op.activeToolCallIds.add(callId); + break; + case 'executing': + this.closeApprovalWaitForCall(op, callId); + op.activeToolCallIds.add(callId); + break; + case 'success': + case 'error': + this.closeApprovalWaitForCall(op, callId); + op.activeToolCallIds.delete(callId); + break; + case 'cancelled': { + const phase: LivePhase = op.approvalStarts.has(callId) + ? 'approval' + : 'tool'; + this.retainCancellationEvidence(op, signal, phase); + this.closeApprovalWaitForCall(op, callId); + op.activeToolCallIds.delete(callId); + break; + } + default: { + // Exhaustiveness guard; unknown statuses do not change phase state. + const _exhaustive: never = status; + void _exhaustive; + } + } + } + + /** + * Classifies the cancellation status for the given signal. + * + * 1. If retained terminal cancellation evidence exists (set ONLY by a + * cancellation terminal signal/event — tool-status cancelled or provider + * attempt aborted end), it wins and persists past finalise. + * 2. Otherwise, the most-specific ACTIVE phase wins with deterministic + * precedence `approval > tool > api` at the instant of abort. A + * completed/rejected approval followed by ordinary API activity yields + * `during_api`. + * 3. Default: `cancelled_during_api`. + */ + classifyCancellation(signal: AbortSignal): OperationStatus { + const retained = this.retainedCancellationEvidence.get(signal); + if (retained !== undefined) { + return phaseToCancelledStatus(retained); + } + const op = this.active.get(signal); + if (op !== undefined) { + if (op.approvalStarts.size > 0) return 'cancelled_during_approval'; + if (op.activeToolCallIds.size > 0) return 'cancelled_during_tool'; + return 'cancelled_during_api'; + } + return 'cancelled_during_api'; + } + + // --- PerfPhaseObserver: provider/tool event accumulation (P07) --- + + /** + * PerfPhaseObserver: notified by AttemptRecorder at attempt start. Associates + * the attempt to an operation via deriveOperationId(info.promptId) (the + * AttemptRecorder's logicalRequestId — D1) so concurrent subagent/unrelated + * requests are NOT misattributed to the foreground operation. Does NOT use + * getFirstActiveOp. + * + * Stale-evidence clearing: a new provider attempt for an operation proves the + * operation continued past a prior tool-status `cancelled` terminal, so any + * retained tool/approval cancellation evidence is now stale and is cleared + * here (a later independent API abort then classifies cancelled_during_api). + * Provider-aborted (api) terminal evidence is PRESERVED (overlap precedence + + * provider-aborted honesty). Current active tool/approval state is NOT + * touched — only the retained evidence map. + */ + onProviderAttemptStart(info: { + readonly attemptId: string; + readonly promptId: string; + readonly startMs: number; + }): void { + const op = this.findOpByPromptId(info.promptId); + if (op !== undefined) { + op.providerSeen.add(info.attemptId); + this.clearStaleToolApprovalCancellationEvidence(op); + } + } + + /** + * PerfPhaseObserver: notified by AttemptRecorder at attempt end. Associates + * via deriveOperationId(info.promptId) (D1); dedup by attemptId; accumulate + * count/sum/union/tokens exactly once. A consumer-abort end retains API + * cancellation evidence. Invoked at an uncaught lifecycle boundary (D8). + */ + onProviderAttemptEnd(info: PerfProviderAttemptEndInfo): void { + const op = this.findOpByPromptId(info.promptId); + if (op === undefined) return; + if (!op.providerSeen.has(info.attemptId)) return; + op.providerSeen.delete(info.attemptId); + op.measurement.provider_attempts += 1; + const duration = Math.max(0, info.endMs - info.startMs); + op.measurement.provider_attempt_sum_ms += duration; + op.providerIntervals.add(info.startMs, info.endMs); + op.measurement.context_tokens += info.inputTokens; + op.measurement.output_tokens += info.outputTokens; + if (info.status === 'aborted') { + this.retainCancellationEvidence(op, op.signal, 'api'); + } + } + + /** + * PerfPhaseObserver: notified by logToolCall at tool completion. Associates + * via deriveOperationId(promptId) (D1). Dedup by real callId exactly-once; + * missing callId counted honestly. Invoked at an uncaught lifecycle boundary. + */ + onToolCallCompleted(info: PerfToolCallCompletedInfo): void { + const operationId = deriveOperationId(info.promptId); + for (const op of this.active.values()) { + if (op.operationId !== operationId) continue; + if (info.callId !== undefined) { + if (op.toolSeen.has(info.callId)) return; + op.toolSeen.add(info.callId); + } + op.measurement.tool_calls += 1; + op.measurement.tool_call_sum_ms += info.durationMs; + // Interval honesty: only union when BOTH real boundaries are present. + // When start_ms or end_ms is absent the count/sum are still recorded, + // but NO interval is synthesized from monotonicNow — we never invent + // timing. A missing call_id cannot be deduplicated exactly, so each + // unidentifiable completed event counts independently (no invented ID). + if (info.startMs !== undefined && info.endMs !== undefined) { + op.toolIntervals.add(info.startMs, info.endMs); + } + return; + } + } + + // --- InteractiveRenderObserver (P07) --- + + /** + * Ink render observer: accumulates renderTime and render count for the + * current active op. Render passes are DISTINCT from stdout writes. + */ + onRender(renderTimeMs: number): void { + const op = this.getFirstActiveOp(); + if (op !== undefined) { + op.measurement.ink_render_ms += renderTimeMs; + op.measurement.ink_render_count += 1; + } + } + + // --- StdoutWriteObserver (P07) --- + + /** + * Stdout write observer: accumulates encoded bytes, write-call count, and + * sync write duration for the current active op. Write calls are DISTINCT + * from render passes. + */ + onWrite(encodedBytes: number, syncDurationMs: number): void { + const op = this.getFirstActiveOp(); + if (op !== undefined) { + op.measurement.stdout_bytes += encodedBytes; + op.measurement.stdout_write_calls += 1; + op.measurement.stdout_write_sync_ms += syncDurationMs; + } + } + + // ----------------------------------------------------------------------- + // Private helpers + // ----------------------------------------------------------------------- + + /** + * Returns the first (and typically only) active op. Used ONLY for + * foreground-only observers (Ink render, stdout write) which can never + * originate from a subagent. Provider/tool correlation uses explicit + * prompt/call identity instead (D1). + */ + private getFirstActiveOp(): PendingOp | undefined { + const entry = this.active.values().next(); + return entry.done === true ? undefined : entry.value; + } + + /** + * Finds the active op whose operation_id matches deriveOperationId(promptId). + * Returns undefined when no active op matches (e.g. a subagent/unrelated + * request), so it is never misattributed to the foreground operation. + */ + private findOpByPromptId(promptId: string): PendingOp | undefined { + const operationId = deriveOperationId(promptId); + for (const op of this.active.values()) { + if (op.operationId === operationId) return op; + } + return undefined; + } + + /** + * Closes the approval-wait interval for a single call ID (if open), + * unioning it into the approval-wait IntervalUnion (overlapping-safe). + */ + private closeApprovalWaitForCall(op: PendingOp, callId: string): void { + const start = op.approvalStarts.get(callId); + if (start === undefined) return; + const end = this.monotonicNow(); + op.approvalWaitIntervals.add(start, end); + op.measurement.approval_wait_ms = op.approvalWaitIntervals.durationMs(); + op.approvalStarts.delete(callId); + } + + /** + * Closes every still-open approval-wait interval (used at record assembly so + * an in-flight approval wait at finalise time is accounted for). + */ + private closeAllApprovalWaits(op: PendingOp): void { + for (const callId of Array.from(op.approvalStarts.keys())) { + this.closeApprovalWaitForCall(op, callId); + } + } + + /** + * Clears retained tool/approval cancellation evidence (set ONLY by a + * tool-status `cancelled` terminal). Called when a new provider attempt + * starts for the operation, which proves the operation continued past the + * tool cancellation. Provider-aborted (api) evidence is NEVER cleared here + * (overlap precedence + provider-aborted honesty). Current active + * tool/approval state is untouched. + */ + private clearStaleToolApprovalCancellationEvidence(op: PendingOp): void { + const existing = op.cancellationEvidence; + if (existing === 'tool' || existing === 'approval') { + op.cancellationEvidence = null; + this.retainedCancellationEvidence.delete(op.signal); + } + } + + /** + * Retains terminal cancellation evidence for a signal. Set ONLY by a + * cancellation terminal signal/event (tool-status cancelled, provider + * attempt aborted end). Precedence `approval > tool > api`: a more-specific + * retained phase is never downgraded by a later less-specific one. Persists + * past finalise so classifyCancellation works after the active op clears. + */ + private retainCancellationEvidence( + op: PendingOp, + signal: AbortSignal, + phase: LivePhase, + ): void { + const existing = op.cancellationEvidence; + if ( + existing === null || + PHASE_PRECEDENCE.indexOf(phase) < PHASE_PRECEDENCE.indexOf(existing) + ) { + op.cancellationEvidence = phase; + this.retainedCancellationEvidence.set(signal, phase); + } + } + + // ----------------------------------------------------------------------- + // Private + // ----------------------------------------------------------------------- + + /** + * Serializes record writes through a single lifecycle chain so writes happen + * in lifecycle order. The chain propagates internal rejections (fail-fast): + * once an internal error occurs, drain() rejects and subsequent writes are + * skipped rather than silently attempted against a poisoned state. External + * filesystem errno errors already resolve inside PerfSink (fail-open), so + * they do not poison the chain. This ensures an internal rejection from a + * write that was not individually awaited (e.g. the superseded sweep) is + * surfaced via drain rather than hidden by a permanently non-rejecting chain. + * + * Receives a frozen {@link FrozenTerminalSnapshot}: the async work does ONLY + * genuinely external claim counting (countNonStaleClaims) and sink + * persistence, never touching the mutable PendingOp/measurement. + */ + private queueWrite(snapshot: FrozenTerminalSnapshot): Promise { + const attempt = (): Promise => this.persistSnapshot(snapshot); + const result = this.lifecycleChain.then(attempt); + // Attach a local rejection handler so a rejected write whose individual + // promise was not awaited (notably the superseded sweep from begin) does + // not become a process-level unhandled rejection. The shared + // lifecycleChain (=== result) remains rejected so drain() still fails + // fast. This does NOT globally catch, log, or convert the chain to green. + void result.catch(() => {}); + this.lifecycleChain = result; + return result; + } + + /** + * Synchronously builds an immutable terminal snapshot of the operation at + * exactly-once claim time. Captures every terminal field coherently: + * + * - Closes any open approval-wait intervals (approval-wait closure). + * - Computes interval-union durations for provider/tool/agent_activity. + * - Captures operation-end memory (marks operation-end then samples once). + * - Captures wall timestamp and the finalization boundary monotonic time. + * + * Coherent clocks: `elapsedMs` and `client_finalize_ms` share the SAME end + * boundary (finalizeEnd) so elapsed includes the full synchronous + * finalization work before subtracting client_finalize_ms. The honest + * residual (unclassified_elapsed_ms) is elapsed minus directly-measured + * client phases and approval wait — never clamped, never zeroed. + * + * The returned snapshot copies identity, measurement, and optional memory + * columns so mutations to provider-owned or live operation objects after this + * point cannot affect the persisted record. + */ + private buildFrozenSnapshot( + op: PendingOp, + status: OperationStatus, + ): FrozenTerminalSnapshot { + const finalizeStart = this.monotonicNow(); + + // Close any open approval-wait intervals before computing the record. + this.closeAllApprovalWaits(op); + + // Compute IntervalUnion durations for provider/tool/agent_activity unions. + const providerUnionMs = op.providerIntervals.durationMs(); + const toolUnionMs = op.toolIntervals.durationMs(); + const agentActivityUnionMs = op.providerIntervals + .union(op.toolIntervals) + .durationMs(); + + // Capture operation-end memory at the synchronous finalization boundary. + const memoryColumns = this.captureOperationEndMemory(); + + // Finalization boundary: elapsed and client_finalize_ms share the SAME + // end so the residual is coherent — elapsed includes the full synchronous + // finalization work (union computation, approval-wait closing, residual + // computation, memory sampling) before subtracting client_finalize_ms. + const finalizeEnd = this.monotonicNow(); + const wallNow = this.wallNow(); + const elapsedMs = finalizeEnd - op.startedAtMonotonic; + const uptimeMs = finalizeEnd; + const clientFinalizeMs = finalizeEnd - finalizeStart; + + const source = op.measurement; + // Frozen copy of all measurement counters/tokens plus the computed union + // durations and client_finalize_ms. Mutations to the live measurement + // after this point cannot affect this snapshot. + const measurement: OperationMeasurement = { + ...source, + provider_union_ms: providerUnionMs, + tool_union_ms: toolUnionMs, + agent_activity_union_ms: agentActivityUnionMs, + client_finalize_ms: clientFinalizeMs, + }; + + const residual = + elapsedMs - + measurement.client_prepare_ms - + measurement.stream_handler_ms - + measurement.ink_render_ms - + measurement.stdout_write_sync_ms - + clientFinalizeMs - + measurement.approval_wait_ms; + + return { + operationId: op.operationId, + identity: { ...op.identity }, + index: op.index, + status, + wallIso: new Date(wallNow).toISOString(), + wallNowMs: wallNow, + elapsedMs, + uptimeMs, + measurement, + residual, + memoryColumns: memoryColumns === null ? null : { ...memoryColumns }, + }; + } + + /** + * Queued async work: does ONLY genuinely external claim counting + * (countNonStaleClaims) and sink persistence. Receives the frozen snapshot, + * derives concurrent_instances (clamped to minimum 1 if the filesystem + * fail-opened), builds the final record, and writes through PerfSink. + */ + private async persistSnapshot( + snapshot: FrozenTerminalSnapshot, + ): Promise { + // --- async work (NOT part of client_finalize_ms) --- + const concurrentInstances = Math.max( + 1, + await this.retention.countNonStaleClaims(snapshot.wallNowMs), + ); + + const m = snapshot.measurement; + const record: PerfOperationRecord = { + schema_version: PERF_SCHEMA_VERSION, + record_type: PERF_RECORD_TYPE_OPERATION, + ts: snapshot.wallIso, + session_id: snapshot.identity.session_id, + operation_id: snapshot.operationId, + runtime_id: snapshot.identity.runtime_id, + parent_runtime_id: snapshot.identity.parent_runtime_id, + subagent_name: snapshot.identity.subagent_name, + project_hash: snapshot.identity.project_hash, + llxprt_version: snapshot.identity.llxprt_version, + git_sha: snapshot.identity.git_sha, + runtime: snapshot.identity.runtime, + platform: snapshot.identity.platform, + provider: snapshot.identity.provider, + model: snapshot.identity.model, + context_tokens: m.context_tokens, + output_tokens: m.output_tokens, + terminal_cols: snapshot.identity.terminal_cols, + terminal_rows: snapshot.identity.terminal_rows, + render_mode: snapshot.identity.render_mode, + concurrent_instances: concurrentInstances, + status: snapshot.status, + client_prepare_ms: m.client_prepare_ms, + stream_handler_ms: m.stream_handler_ms, + ink_render_ms: m.ink_render_ms, + ink_render_count: m.ink_render_count, + stdout_bytes: m.stdout_bytes, + stdout_write_calls: m.stdout_write_calls, + stdout_write_sync_ms: m.stdout_write_sync_ms, + client_finalize_ms: m.client_finalize_ms, + provider_attempts: m.provider_attempts, + provider_attempt_sum_ms: m.provider_attempt_sum_ms, + provider_union_ms: m.provider_union_ms, + tool_calls: m.tool_calls, + tool_call_sum_ms: m.tool_call_sum_ms, + tool_union_ms: m.tool_union_ms, + agent_activity_union_ms: m.agent_activity_union_ms, + operation_elapsed_ms: snapshot.elapsedMs, + approval_wait_ms: m.approval_wait_ms, + unclassified_elapsed_ms: snapshot.residual, + ...(snapshot.memoryColumns ?? {}), + session_operation_index: snapshot.index, + uptime_ms: snapshot.uptimeMs, + }; + + await this.sink.write(record); + } + + /** + * P10: captures operation-end memory columns at exactly-once finalisation. + * Marks operation-end first (so subsequent tick samples compute idle + * relative to this moment), then samples process.memoryUsage() once. + * Returns null when disabled/absent — all four fields omitted, never zeros. + */ + private captureOperationEndMemory(): MemoryColumns | null { + this.memorySampler?.markOperationEnd(); + return this.memorySampler?.sampleOperationEndMemory() ?? null; + } +} diff --git a/packages/cli/src/ui/hooks/agentStream/submitQueryTurnLifecycle.ts b/packages/cli/src/ui/hooks/agentStream/submitQueryTurnLifecycle.ts new file mode 100644 index 0000000000..3758c9568b --- /dev/null +++ b/packages/cli/src/ui/hooks/agentStream/submitQueryTurnLifecycle.ts @@ -0,0 +1,423 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Turn execution + operation-lifecycle helpers extracted from useSubmitQuery + * (issue #3167). Contains the core submit-query execution path that prepares + * the query, runs the stream, and finalises the operation-lifecycle registry + * with the correct terminal status — preserving fail-fast / internal-error + * semantics and granular cancellation classification. + */ + +import type { AgentRequestInput } from '@vybestack/llxprt-code-core'; +import { prepareTurnForQuery } from './turnPreparation.js'; +import { handleSubmissionError } from './streamUtils.js'; +import { + observeTurnFailed, + observeTurnStarted, +} from '../../../observation/jspWiring.js'; +import type { + OperationLifecycleRegistry, + OperationStatus, +} from './operationLifecycle.js'; +import type { UseSubmitQueryDeps } from './useSubmitQuery.js'; + +export interface TurnInit { + userMessageTimestamp: number; + abortSignal: AbortSignal; + promptId: string; + trimmedStr: string; +} + +export interface SubmitQueryCallbackDeps extends UseSubmitQueryDeps { + displayUserMessage: (q: string, t: number) => void; + prepareQueryForAgent: ( + query: AgentRequestInput, + userMessageTimestamp: number, + abortSignal: AbortSignal, + promptId: string, + ) => Promise<{ + queryToSend: AgentRequestInput | null; + shouldProceed: boolean; + }>; + handleLoopDetectedEvent: () => void; + startNewPrompt: () => void; + getPromptCount: () => number; + activeTurnRef: React.MutableRefObject; + scheduleNextQueuedSubmission: () => void; +} + +/** + * Returns the finalise promise so callers can await it. The registry's own + * error policy rejects on internal/schema failures (fail-fast — D8) and + * resolves on filesystem errno errors (fail-open inside PerfSink/retention). + * Callers await this so internal instrumentation errors propagate rather + * than being silently debug-logged and swallowed. + */ +function finaliseOperation( + lifecycle: OperationLifecycleRegistry | undefined, + signal: AbortSignal, + status: OperationStatus, +): Promise | undefined { + return lifecycle?.finalise(signal, status); +} + +/** + * Finalises as 'error' after a preparation rejection, preserving the original + * rejection to the caller. If the instrumentation finalise also has an internal + * failure, an AggregateError carries both (project convention). Always throws. + */ +async function finalisePrepRejection( + lifecycle: OperationLifecycleRegistry | undefined, + signal: AbortSignal, + prepError: unknown, + context: string, +): Promise { + try { + await finaliseOperation(lifecycle, signal, 'error'); + } catch (finaliseError) { + throw new AggregateError( + [prepError, finaliseError], + `${context} (with instrumentation error)`, + ); + } + throw prepError; +} + +/** + * Finalises once as the given cancellation status, preserving the original + * cancellation error to the caller. If the instrumentation finalise has an + * internal failure, an AggregateError carries both the original cancellation + * and the finalisation error (project convention). Always throws. + */ +async function finaliseCancellation( + lifecycle: OperationLifecycleRegistry | undefined, + signal: AbortSignal, + cancelError: unknown, + status: OperationStatus, + context: string, +): Promise { + try { + await finaliseOperation(lifecycle, signal, status); + } catch (finaliseError) { + throw new AggregateError( + [cancelError, finaliseError], + `${context} (with instrumentation error)`, + ); + } + throw cancelError; +} + +/** + * Finalises once as 'error' after a failure that occurs AFTER a successful + * `operationLifecycle.begin` but BEFORE `runSubmitQueryCore` (e.g. a throwing + * `displayUserMessage`). The original failure is preserved; if finalisation + * also has an internal failure, both are surfaced via AggregateError. Always + * throws. Exported so the caller (which owns the `begin` call) can finalise + * exactly once when an intermediate step throws. + */ +export async function finaliseOnceAfterBegin( + lifecycle: OperationLifecycleRegistry | undefined, + signal: AbortSignal, + originalError: unknown, + context: string, +): Promise { + try { + await finaliseOperation(lifecycle, signal, 'error'); + } catch (finaliseError) { + throw new AggregateError( + [originalError, finaliseError], + `${context} (with instrumentation error)`, + ); + } + throw originalError; +} + +async function finaliseTurnStartFailure( + cbd: SubmitQueryCallbackDeps, + turn: TurnInit, + error: unknown, +): Promise { + const errors: unknown[] = [error]; + if (isCurrentTurn(cbd, turn.abortSignal)) { + try { + cbd.setIsResponding(false); + } catch (resetError) { + errors.push(resetError); + } + } + try { + await finaliseOperation(cbd.operationLifecycle, turn.abortSignal, 'error'); + } catch (finaliseError) { + errors.push(finaliseError); + } + if (errors.length > 1) { + throw new AggregateError(errors, 'Turn start failed (with cleanup error)'); + } + throw error; +} + +/** + * Handles a provider stream error for the user (observeTurnFailed + + * handleSubmissionError) only when this turn still owns the controller. + */ +function handleProviderError( + cbd: SubmitQueryCallbackDeps, + turn: TurnInit, + error: unknown, +): void { + if (isCurrentTurn(cbd, turn.abortSignal)) { + observeTurnFailed(); + handleSubmissionError( + error, + cbd.addItem, + cbd.runtime, + cbd.onAuthError, + turn.userMessageTimestamp, + ); + } +} + +/** + * Finalises a provider stream error as 'error'. The original provider error is + * handled for the user via handleSubmissionError. If the finalise has an + * internal failure, the provider error is handled FIRST (so it is not lost), + * then the instrumentation error fails-fast (D8). The instrumentation error is + * NOT routed through user-facing provider-error handling. + */ +async function finaliseStreamError( + cbd: SubmitQueryCallbackDeps, + turn: TurnInit, + streamError: unknown, +): Promise { + try { + await finaliseOperation(cbd.operationLifecycle, turn.abortSignal, 'error'); + } catch (finaliseError) { + handleProviderError(cbd, turn, streamError); + throw finaliseError; + } + handleProviderError(cbd, turn, streamError); +} + +/** + * Prepares the query for the agent, returning the query to send or throwing + * after finalising the operation appropriately. Handles cancellation during + * prep (cancelled_before_send) and genuine errors (error + preserved rejection). + */ +async function prepareAndCheckProceed( + cbd: SubmitQueryCallbackDeps, + query: AgentRequestInput, + turn: TurnInit, +): Promise { + let queryToSend: AgentRequestInput | null = null; + let shouldProceed = false; + try { + const result = await cbd.prepareQueryForAgent( + query, + turn.userMessageTimestamp, + turn.abortSignal, + turn.promptId, + ); + queryToSend = result.queryToSend; + shouldProceed = result.shouldProceed; + } catch (prepError) { + if (isCancellation(prepError, turn.abortSignal)) { + await finaliseCancellation( + cbd.operationLifecycle, + turn.abortSignal, + prepError, + 'cancelled_before_send', + 'Query preparation cancelled', + ); + } + await finalisePrepRejection( + cbd.operationLifecycle, + turn.abortSignal, + prepError, + 'Query preparation failed', + ); + } + + if (!shouldProceed || queryToSend === null) { + // Benign no-proceed: the instrumentation finalise propagates fail-fast + // (D8) — this is intentionally NOT swallowed, consistent with the + // successful-completion path. + await finaliseOperation( + cbd.operationLifecycle, + turn.abortSignal, + 'cancelled_before_send', + ); + return null; + } + + try { + await prepareTurnForQuery( + false, + cbd.runtime, + cbd.startNewPrompt, + cbd.setThought, + cbd.thinkingBlocksRef, + ); + } catch (prepError) { + if (isCancellation(prepError, turn.abortSignal)) { + await finaliseCancellation( + cbd.operationLifecycle, + turn.abortSignal, + prepError, + 'cancelled_before_send', + 'Turn preparation cancelled', + ); + } + await finalisePrepRejection( + cbd.operationLifecycle, + turn.abortSignal, + prepError, + 'Turn preparation failed', + ); + } + + return queryToSend; +} + +async function executeStream( + deps: UseSubmitQueryDeps, + handleLoopDetectedEvent: () => void, + queryToSend: AgentRequestInput, + turn: TurnInit, +): Promise { + const runStream = deps.runStreamRef.current; + if (!runStream) { + throw new Error('Agent event-stream runner is not initialized.'); + } + + // P07: capture client_prepare_ms immediately before the first runStream + // send (the monotonic delta from begin to just-before-send). + deps.operationLifecycle?.captureClientPrepare(turn.abortSignal); + + // The Agent owns the entire multi-turn flow: send → stream → schedule → + // execute → feed-back → repeat. + await runStream(queryToSend, turn.abortSignal, turn.promptId); + + // A newer turn may have started while runStream was settling (e.g. the user + // cancelled this turn and submitted a new prompt). If the current + // AbortController no longer belongs to this turn, skip post-stream cleanup + // so it does not clobber the newer turn's state (issue #2259). + if (!isCurrentTurn(deps, turn.abortSignal)) { + return; + } + + if (deps.pendingHistoryItemRef.current) { + deps.flushPendingHistoryItem(turn.userMessageTimestamp); + deps.setPendingHistoryItem(null); + } + if (deps.loopDetectedRef.current) { + deps.loopDetectedRef.current = false; + handleLoopDetectedEvent(); + } +} + +/** + * Runs the stream and finalises with the correct terminal status. On + * cancellation, classifies granularly from live/terminal phase evidence. + */ +async function streamAndFinalise( + cbd: SubmitQueryCallbackDeps, + queryToSend: AgentRequestInput, + turn: TurnInit, +): Promise { + cbd.operationLifecycle?.enterApiPhase(turn.abortSignal); + + try { + let streamError: unknown = null; + try { + await executeStream(cbd, cbd.handleLoopDetectedEvent, queryToSend, turn); + } catch (error) { + streamError = error; + } + + if (streamError === null) { + // Successful completion: the instrumentation finalise propagates + // fail-fast (D8) — this is intentionally NOT swallowed. + await finaliseOperation( + cbd.operationLifecycle, + turn.abortSignal, + 'completed', + ); + } else if (isCancellation(streamError, turn.abortSignal)) { + const status = + cbd.operationLifecycle?.classifyCancellation(turn.abortSignal) ?? + 'cancelled_during_api'; + try { + await finaliseOperation( + cbd.operationLifecycle, + turn.abortSignal, + status, + ); + } catch (finaliseError) { + throw new AggregateError( + [streamError, finaliseError], + 'Query stream cancelled (with instrumentation error)', + ); + } + } else { + await finaliseStreamError(cbd, turn, streamError); + } + } finally { + if (isCurrentTurn(cbd, turn.abortSignal)) { + cbd.setIsResponding(false); + } + if (isCurrentTurn(cbd, turn.abortSignal)) { + try { + await cbd.recordingIntegration?.flushAtTurnBoundary(); + } catch { + /* non-fatal */ + } + } + } +} + +export async function runSubmitQueryCore( + cbd: SubmitQueryCallbackDeps, + query: AgentRequestInput, + turn: TurnInit, +): Promise { + const queryToSend = await prepareAndCheckProceed(cbd, query, turn); + if (queryToSend === null) return; + + try { + cbd.setIsResponding(true); + cbd.setInitError(null); + observeTurnStarted(); + cbd.pendingResponse.beginCommittedSegments(); + } catch (error) { + await finaliseTurnStartFailure(cbd, turn, error); + } + + await streamAndFinalise(cbd, queryToSend, turn); +} + +/** + * Returns true when the given signal still belongs to the active turn. When a + * newer turn starts (via initTurn) it replaces abortControllerRef.current with + * a fresh AbortController; comparing signals proves the caller still owns the + * current AbortController (issue #2259, #2954). + */ +export function isCurrentTurn( + deps: UseSubmitQueryDeps, + signal: AbortSignal, +): boolean { + return deps.abortControllerRef.current?.signal === signal; +} + +/** + * Returns true when an error is an AbortError or the signal is aborted. + * Used to classify cancellation vs genuine stream errors. + */ +function isCancellation(error: unknown, signal: AbortSignal): boolean { + if (signal.aborted) return true; + if (error instanceof Error && error.name === 'AbortError') return true; + return false; +} diff --git a/packages/cli/src/ui/hooks/agentStream/useAgentEventStream.ts b/packages/cli/src/ui/hooks/agentStream/useAgentEventStream.ts index 286772138a..74cb78b51b 100644 --- a/packages/cli/src/ui/hooks/agentStream/useAgentEventStream.ts +++ b/packages/cli/src/ui/hooks/agentStream/useAgentEventStream.ts @@ -45,6 +45,24 @@ import { const logger = DebugLogger.getLogger('llxprt:cli:agent-event-stream'); +/** + * Package-private monotonic clock seam for default-off testing (P07). Defaults + * to `performance.now`; tests inject a counting clock to prove the + * absent-observer path performs NO timing work. NOT exported via the package + * barrel (index.ts) — tests deep-import it directly. + */ +let monotonicClock: () => number = () => performance.now(); + +/** + * @internal Package-private test seam for the event-dispatch monotonic clock. + * NOT part of the package barrel/API; pass null to restore the default clock. + */ +export function __setMonotonicClockForTesting( + clock: (() => number) | null, +): void { + monotonicClock = clock ?? (() => performance.now()); +} + /** Routes a single public AgentEvent into React state. */ export type AgentEventRouter = ( event: AgentEvent, @@ -80,6 +98,24 @@ export interface UseAgentEventStreamArgs { getPreferredEditor?: () => EditorType | undefined; onEditorOpen?: () => void; onEditorClose?: () => void; + /** + * Optional perf callback invoked DIRECTLY OUTSIDE the ordinary + * processAgentEvent try/catch for EACH observed AgentEvent, carrying the + * turn's AbortSignal so measurements route to the correct operation (not the + * "current active op" by position). This performs live phase tracking + * (tool-status → tool/approval states, provider-abort → API evidence) and + * accumulates stream_handler_ms. + * + * D8: because this is invoked OUTSIDE the generic catch-and-log boundary, a + * perf observer/programming error propagates and REJECTS the stream. + * Ordinary event-handler errors remain caught/logged so the stream + * continues. When undefined (perf disabled), this is a no-op. + */ + onAgentEventObserved?: ( + event: AgentEvent, + signal: AbortSignal, + syncHandlerMs: number, + ) => void; } export interface UseAgentEventStreamReturn { @@ -289,8 +325,11 @@ function iterateAgentStream( return (async () => { const input = toAgentInput(message); const iterator = agent.stream(input, { signal, promptId }); - for await (const event of iterator) { - if (signal.aborted) break; + const observer = args.onAgentEventObserved; + // Shared dispatch-with-catch: one bad event must not abort the entire + // stream. Extracted so both the observer-absent and observer-present + // branches use identical dispatch behavior. + const dispatchEvent = (event: AgentEvent): void => { try { args.processAgentEventRef.current?.( event, @@ -301,6 +340,24 @@ function iterateAgentStream( // One bad event must not abort the entire stream. logger.error('Error processing agent event:', error); } + }; + for await (const event of iterator) { + if (signal.aborted) break; + if (observer === undefined) { + // Default-off: no perf observer means NO timing calls and NO sample + // allocation. The ordinary handler dispatch/catch behavior is preserved + // so one bad event never aborts the stream. + dispatchEvent(event); + } else { + // Measure the synchronous dispatch, then invoke the observer OUTSIDE + // the generic catch-and-log boundary (D8: a perf-callback throw rejects + // the stream / fail-fast). Ordinary handler throws were already + // caught/logged inside dispatchEvent so the stream continues. + const handlerStart = monotonicClock(); + dispatchEvent(event); + const handlerMs = monotonicClock() - handlerStart; + observer(event, signal, handlerMs); + } } })(); } diff --git a/packages/cli/src/ui/hooks/agentStream/useAgentStream.ts b/packages/cli/src/ui/hooks/agentStream/useAgentStream.ts index a7c788a21a..468fe72c55 100644 --- a/packages/cli/src/ui/hooks/agentStream/useAgentStream.ts +++ b/packages/cli/src/ui/hooks/agentStream/useAgentStream.ts @@ -36,6 +36,7 @@ import type { useStreamState } from './useStreamState.js'; import type { AgentStreamOrchestrationDeps } from './useAgentStreamOrchestration.js'; import { useAgentStreamOrchestration } from './useAgentStreamOrchestration.js'; import type { StreamRuntime, UiSubagentManager } from '../../cliUiRuntime.js'; +import type { OperationLifecycleRegistry } from './operationLifecycle.js'; export const useAgentStream = ( agent: Agent, @@ -61,6 +62,7 @@ export const useAgentStream = ( runtimeMessageBus?: MessageBus, subagentManager?: UiSubagentManager, removeItems?: RemoveHistoryItems, + operationLifecycle?: OperationLifecycleRegistry, ) => { const orchestration = useAgentStreamOrchestration({ agent, @@ -83,6 +85,7 @@ export const useAgentStream = ( recordingIntegration, runtimeMessageBus, subagentManager, + operationLifecycle, } satisfies AgentStreamOrchestrationDeps); return useAgentStreamReturn( diff --git a/packages/cli/src/ui/hooks/agentStream/useAgentStreamOrchestration.ts b/packages/cli/src/ui/hooks/agentStream/useAgentStreamOrchestration.ts index 38036cb131..1393b4a38f 100644 --- a/packages/cli/src/ui/hooks/agentStream/useAgentStreamOrchestration.ts +++ b/packages/cli/src/ui/hooks/agentStream/useAgentStreamOrchestration.ts @@ -39,6 +39,7 @@ import { } from './useAgentStreamLifecycle.js'; import type { QueuedSubmission } from './types.js'; import type { StreamRuntime, UiSubagentManager } from '../../cliUiRuntime.js'; +import type { OperationLifecycleRegistry } from './operationLifecycle.js'; export interface AgentStreamOrchestrationDeps { agent: Agent; @@ -63,6 +64,12 @@ export interface AgentStreamOrchestrationDeps { recordingIntegration?: RecordingIntegration; runtimeMessageBus?: MessageBus; subagentManager?: UiSubagentManager; + /** + * Optional perf operation lifecycle registry (P06/P07). When undefined + * (perf disabled), no lifecycle instrumentation occurs. Supplied by P12 + * integration wiring. + */ + operationLifecycle?: OperationLifecycleRegistry; } export interface AgentStreamOrchestrationResult { @@ -274,6 +281,7 @@ function useEventStreamForAgent( scheduler: ToolSchedulerState, processAgentEventRef: React.MutableRefObject, ) { + const lifecycle = args.operationLifecycle; return useAgentEventStream({ agent: args.agent, addItem: args.addItem, @@ -291,6 +299,15 @@ function useEventStreamForAgent( getPreferredEditor: args.getPreferredEditor, onEditorOpen: args.onEditorOpen, onEditorClose: args.onEditorClose, + // P07: perf event observation routed OUTSIDE the generic event-handler + // catch (D8: a perf callback throw rejects the stream). The turn's + // AbortSignal is passed through so measurements route to the correct op, + // never to "the current active op" by position. Wired only when a registry + // exists (perf enabled). + onAgentEventObserved: lifecycle + ? (event, signal, handlerMs) => + lifecycle.observeAgentEvent(event, signal, handlerMs) + : undefined, }); } @@ -419,5 +436,6 @@ function buildSubmitQueryDeps({ streamingState, runStreamRef, subagentManager: args.subagentManager, + operationLifecycle: args.operationLifecycle, }; } diff --git a/packages/cli/src/ui/hooks/agentStream/useSubmitQuery.ts b/packages/cli/src/ui/hooks/agentStream/useSubmitQuery.ts index a6d5a97c07..688fb14f5d 100644 --- a/packages/cli/src/ui/hooks/agentStream/useSubmitQuery.ts +++ b/packages/cli/src/ui/hooks/agentStream/useSubmitQuery.ts @@ -30,8 +30,6 @@ import { } from '../../types.js'; import { isSlashCommand } from '../../utils/commandUtils.js'; import { useSessionStats } from '../../contexts/SessionContext.js'; -import { handleSubmissionError } from './streamUtils.js'; -import { prepareTurnForQuery } from './turnPreparation.js'; import { useStreamEventHandlers } from './useStreamEventHandlers.js'; import { dispatchAgentEvent } from './agentEventDispatcher.js'; import type { AgentEventRouter } from './useAgentEventStream.js'; @@ -41,13 +39,17 @@ import { } from '../../utils/modelIdentity.js'; import type { QueuedSubmission } from './types.js'; import type { StreamRuntime, UiSubagentManager } from '../../cliUiRuntime.js'; -import { - observeAgentEvent, - observeTurnFailed, - observeTurnStarted, -} from '../../../observation/jspWiring.js'; +import { observeAgentEvent } from '../../../observation/jspWiring.js'; import type { PendingResponseBuffer } from './pendingResponseBuffer.js'; +import type { OperationLifecycleRegistry } from './operationLifecycle.js'; +import { + runSubmitQueryCore, + finaliseOnceAfterBegin, + isCurrentTurn, + type TurnInit, + type SubmitQueryCallbackDeps, +} from './submitQueryTurnLifecycle.js'; export type SubmissionDisposition = | 'consumed' | 'requeue' @@ -155,6 +157,12 @@ export interface UseSubmitQueryDeps { submitQueryRef: React.MutableRefObject; isResponding: boolean; streamingState: StreamingState; + /** + * Optional operation lifecycle registry for perf telemetry (P06). When + * undefined (perf disabled), no lifecycle instrumentation occurs. Supplied + * later by P12 integration wiring. + */ + operationLifecycle?: OperationLifecycleRegistry; } export interface UseSubmitQueryReturn { @@ -281,6 +289,12 @@ function useProcessAgentEvent( if (!isCurrentTurn(latestDeps.current, signal)) { return; } + // P07: live phase tracking for granular cancellation classification is + // routed OUTSIDE this handler's call site (via the + // onAgentEventObserved callback in useAgentEventStream, invoked directly + // outside the generic catch — D8). It must NOT live inside + // processAgentEvent, where it could be swallowed by this handler's + // callers. See useAgentStreamOrchestration.useEventStreamForAgent. // Release the interactive active-turn gate and responding state BEFORE // dispatching fallible event rendering for terminal public Agent // error/idle-timeout events. If rendering throws, the gate must already @@ -575,24 +589,6 @@ function useScheduleNext( return schedule; } -interface SubmitQueryCallbackDeps extends UseSubmitQueryDeps { - displayUserMessage: (q: string, t: number) => void; - prepareQueryForAgent: ( - query: AgentRequestInput, - userMessageTimestamp: number, - abortSignal: AbortSignal, - promptId: string, - ) => Promise<{ - queryToSend: AgentRequestInput | null; - shouldProceed: boolean; - }>; - handleLoopDetectedEvent: () => void; - startNewPrompt: () => void; - getPromptCount: () => number; - activeTurnRef: React.MutableRefObject; - scheduleNextQueuedSubmission: () => void; -} - function useSubmitQueryCallback(cbd: SubmitQueryCallbackDeps) { // Keep the callback identity stable while every invocation reads current deps. const latestCbdRef = useRef(cbd); @@ -652,11 +648,25 @@ function useSubmitQueryCallback(cbd: SubmitQueryCallbackDeps) { current.getPromptCount, turnSignal, ); + current.operationLifecycle?.begin(turnSignal, turn.promptId); + // Once `begin` has succeeded, any later setup failure must finalise the + // operation exactly once as 'error'. This caller owns display setup; + // runSubmitQueryCore owns its subsequent turn-start setup. Both preserve + // the original failure and aggregate a finalisation failure if needed. if (shouldDisplayUserMessage(turn.trimmedStr)) { - current.displayUserMessage( - turn.trimmedStr, - turn.userMessageTimestamp, - ); + try { + current.displayUserMessage( + turn.trimmedStr, + turn.userMessageTimestamp, + ); + } catch (displayError) { + await finaliseOnceAfterBegin( + current.operationLifecycle, + turnSignal, + displayError, + 'Display user message failed', + ); + } } await runSubmitQueryCore(current, query, turn); @@ -677,71 +687,6 @@ function useSubmitQueryCallback(cbd: SubmitQueryCallbackDeps) { ); } -async function runSubmitQueryCore( - cbd: SubmitQueryCallbackDeps, - query: AgentRequestInput, - turn: TurnInit, -): Promise { - const { queryToSend, shouldProceed } = await cbd.prepareQueryForAgent( - query, - turn.userMessageTimestamp, - turn.abortSignal, - turn.promptId, - ); - if (!shouldProceed || queryToSend === null) { - return; - } - - await prepareTurnForQuery( - false, - cbd.runtime, - cbd.startNewPrompt, - cbd.setThought, - cbd.thinkingBlocksRef, - ); - cbd.setIsResponding(true); - cbd.setInitError(null); - observeTurnStarted(); - // Establish a fresh committed-segment boundary for this top-level turn - // before any event can arrive, so ids left over from a previous completed - // message cannot be retracted by this turn's retry (issue #3048 review). - cbd.pendingResponse.beginCommittedSegments(); - - try { - await executeStream(cbd, cbd.handleLoopDetectedEvent, queryToSend, turn); - } catch (error: unknown) { - // Only surface errors for the active turn. A superseded turn's stale - // errors (e.g. AbortError or auth failures from a cancelled request) - // must not leak into the newer turn (issue #2259). - if (isCurrentTurn(cbd, turn.abortSignal)) { - observeTurnFailed(); - handleSubmissionError( - error, - cbd.addItem, - cbd.runtime, - cbd.onAuthError, - turn.userMessageTimestamp, - ); - } - } finally { - // A superseded turn no longer owns responding state; stale cleanup here - // would clear the newer turn's active indicator. Note: a terminal - // error/idle-timeout event may have already set isResponding(false); - // re-setting it is harmless here because this branch only runs when this - // turn still owns the controller (issue #2954). - if (isCurrentTurn(cbd, turn.abortSignal)) { - cbd.setIsResponding(false); - } - if (isCurrentTurn(cbd, turn.abortSignal)) { - try { - await cbd.recordingIntegration?.flushAtTurnBoundary(); - } catch { - /* non-fatal */ - } - } - } -} - function isQueueable(streamingState: StreamingState): boolean { return ( streamingState === StreamingState.Responding || @@ -753,13 +698,6 @@ function shouldDisplayUserMessage(trimmedStr: string): boolean { return !!trimmedStr && !isSlashCommand(trimmedStr); } -interface TurnInit { - userMessageTimestamp: number; - abortSignal: AbortSignal; - promptId: string; - trimmedStr: string; -} - function initTurn( deps: UseSubmitQueryDeps, query: AgentRequestInput, @@ -785,49 +723,6 @@ function initTurn( }; } -async function executeStream( - deps: UseSubmitQueryDeps, - handleLoopDetectedEvent: () => void, - queryToSend: AgentRequestInput, - turn: TurnInit, -): Promise { - const runStream = deps.runStreamRef.current; - if (!runStream) { - throw new Error('Agent event-stream runner is not initialized.'); - } - - // The Agent owns the entire multi-turn flow: send → stream → schedule → - // execute → feed-back → repeat. - await runStream(queryToSend, turn.abortSignal, turn.promptId); - - // A newer turn may have started while runStream was settling (e.g. the user - // cancelled this turn and submitted a new prompt). If the current - // AbortController no longer belongs to this turn, skip post-stream cleanup - // so it does not clobber the newer turn's state (issue #2259). - if (!isCurrentTurn(deps, turn.abortSignal)) { - return; - } - - if (deps.pendingHistoryItemRef.current) { - deps.flushPendingHistoryItem(turn.userMessageTimestamp); - deps.setPendingHistoryItem(null); - } - if (deps.loopDetectedRef.current) { - deps.loopDetectedRef.current = false; - handleLoopDetectedEvent(); - } -} - -/** - * Returns true when the given signal still belongs to the active turn. When a - * newer turn starts (via initTurn) it replaces abortControllerRef.current with - * a fresh AbortController; comparing signals proves the caller still owns the - * current AbortController (issue #2259, #2954). - */ -function isCurrentTurn(deps: UseSubmitQueryDeps, signal: AbortSignal): boolean { - return deps.abortControllerRef.current?.signal === signal; -} - /** * Issue #3169: detects a fresh submission that arrives while an acknowledged * cancellation still suppresses queue draining. diff --git a/packages/cli/src/ui/hooks/memoryTrend/index.ts b/packages/cli/src/ui/hooks/memoryTrend/index.ts new file mode 100644 index 0000000000..5c389eaaf4 --- /dev/null +++ b/packages/cli/src/ui/hooks/memoryTrend/index.ts @@ -0,0 +1,29 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Memory trend module barrel (P10, issue #3167). + * + * Exports for downstream P11 (reader/consumer + `/perf` live view) and P12 + * (integration wiring). Uses existing package conventions. + */ + +export { MemoryRing, MEMORY_RING_CAPACITY } from './memoryRing.js'; +export type { MemoryRingSample } from './memoryRing.js'; +export { MemoryTelemetryController } from './memoryTelemetry.js'; +export type { + MemoryColumns, + OperationMemorySampler, + MemoryTelemetryControllerOptions, +} from './memoryTelemetry.js'; +export { + derivePerOperationMemorySlope, + derivePerMinuteMemorySlope, +} from './memorySlope.js'; +export type { + PerOperationMemorySlope, + PerMinuteMemorySlope, +} from './memorySlope.js'; diff --git a/packages/cli/src/ui/hooks/memoryTrend/memoryRing.behavior.test.ts b/packages/cli/src/ui/hooks/memoryTrend/memoryRing.behavior.test.ts new file mode 100644 index 0000000000..9f224d4488 --- /dev/null +++ b/packages/cli/src/ui/hooks/memoryTrend/memoryRing.behavior.test.ts @@ -0,0 +1,164 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * P10 behavioral tests for MemoryRing (EVIDENCE-AC11). + * + * Fixed-capacity overwrite ring for the live /perf memory view. At the 60 s + * sampling cadence the default capacity holds two hours of history. Pushing + * beyond capacity overwrites the oldest entry. Snapshots are defensive copies + * ordered oldest→newest with no internal aliasing. + */ + +import { describe, it, expect } from 'bun:test'; +import { MemoryRing, MEMORY_RING_CAPACITY } from './memoryRing.js'; +import type { MemoryRingSample } from './memoryRing.js'; + +function makeSample(rss: number): MemoryRingSample { + return { + rss, + heapUsed: rss + 1, + external: rss + 2, + arrayBuffers: rss + 3, + uptimeMs: rss * 100, + msSinceLastOperation: rss * 10, + timestampMs: rss * 1000, + }; +} + +describe('MemoryRing (AC-11)', () => { + it('snapshot of an empty ring is empty', () => { + const ring = new MemoryRing(4); + expect(ring.snapshot()).toEqual([]); + expect(ring.size).toBe(0); + }); + + it('pushes preserve oldest→newest order below capacity', () => { + const ring = new MemoryRing(4); + ring.push(makeSample(10)); + ring.push(makeSample(20)); + ring.push(makeSample(30)); + const snap = ring.snapshot(); + expect(snap).toHaveLength(3); + expect(snap[0].rss).toBe(10); + expect(snap[1].rss).toBe(20); + expect(snap[2].rss).toBe(30); + expect(ring.size).toBe(3); + }); + + it('overwrites oldest when capacity is exceeded', () => { + const ring = new MemoryRing(3); + ring.push(makeSample(10)); + ring.push(makeSample(20)); + ring.push(makeSample(30)); + ring.push(makeSample(40)); // overwrites 10 + const snap = ring.snapshot(); + expect(snap).toHaveLength(3); + expect(snap[0].rss).toBe(20); + expect(snap[1].rss).toBe(30); + expect(snap[2].rss).toBe(40); + expect(ring.size).toBe(3); + }); + + it('wraps around multiple times keeping only the newest capacity entries', () => { + const ring = new MemoryRing(3); + for (let i = 1; i <= 10; i++) { + ring.push(makeSample(i * 10)); + } + const snap = ring.snapshot(); + expect(snap).toHaveLength(3); + expect(snap[0].rss).toBe(80); + expect(snap[1].rss).toBe(90); + expect(snap[2].rss).toBe(100); + }); + + it('snapshot is a defensive copy (mutating it does not affect the ring)', () => { + const ring = new MemoryRing(4); + ring.push(makeSample(10)); + ring.push(makeSample(20)); + const snap1 = ring.snapshot(); + // Attempt to mutate the returned array and its objects. + (snap1 as Array>).push({ rss: 999 }); + (snap1[0] as { rss: number }).rss = 999; + // The ring's internal state is unaffected. + const snap2 = ring.snapshot(); + expect(snap2).toHaveLength(2); + expect(snap2[0].rss).toBe(10); + expect(snap2[1].rss).toBe(20); + }); + + it('two snapshot calls return independent arrays', () => { + const ring = new MemoryRing(4); + ring.push(makeSample(10)); + const snap1 = ring.snapshot(); + const snap2 = ring.snapshot(); + expect(snap1).not.toBe(snap2); + expect(snap1).toEqual(snap2); + }); + + it('exposes the documented default capacity', () => { + // 120 samples × 60 s = 2 hours of history at the monitor cadence. + expect(MEMORY_RING_CAPACITY).toBe(120); + }); + + it('default capacity ring holds exactly capacity entries after overflow', () => { + const ring = new MemoryRing(); + for (let i = 0; i < MEMORY_RING_CAPACITY + 50; i++) { + ring.push(makeSample(i)); + } + expect(ring.size).toBe(MEMORY_RING_CAPACITY); + const snap = ring.snapshot(); + expect(snap).toHaveLength(MEMORY_RING_CAPACITY); + // The oldest surviving entry is index 50 (first 50 were overwritten). + expect(snap[0].rss).toBe(50); + expect(snap[snap.length - 1].rss).toBe(MEMORY_RING_CAPACITY + 49); + }); +}); + +describe('MemoryRing capacity validation (AC-11)', () => { + it('rejects capacity 0', () => { + expect(() => new MemoryRing(0)).toThrow(RangeError); + }); + + it('rejects a negative capacity', () => { + expect(() => new MemoryRing(-1)).toThrow(RangeError); + expect(() => new MemoryRing(-100)).toThrow(RangeError); + }); + + it('rejects a fractional capacity', () => { + expect(() => new MemoryRing(2.5)).toThrow(RangeError); + expect(() => new MemoryRing(0.5)).toThrow(RangeError); + }); + + it('rejects NaN capacity', () => { + expect(() => new MemoryRing(NaN)).toThrow(RangeError); + }); + + it('rejects Infinity capacity', () => { + expect(() => new MemoryRing(Infinity)).toThrow(RangeError); + expect(() => new MemoryRing(-Infinity)).toThrow(RangeError); + }); + + it('accepts capacity 1 and overwrites on every subsequent push', () => { + const ring = new MemoryRing(1); + expect(ring.size).toBe(0); + expect(ring.snapshot()).toEqual([]); + + ring.push(makeSample(10)); + expect(ring.size).toBe(1); + expect(ring.snapshot()[0].rss).toBe(10); + + // Overwrite: the sole slot is replaced. + ring.push(makeSample(20)); + expect(ring.size).toBe(1); + expect(ring.snapshot()).toHaveLength(1); + expect(ring.snapshot()[0].rss).toBe(20); + + ring.push(makeSample(30)); + expect(ring.size).toBe(1); + expect(ring.snapshot()[0].rss).toBe(30); + }); +}); diff --git a/packages/cli/src/ui/hooks/memoryTrend/memoryRing.ts b/packages/cli/src/ui/hooks/memoryTrend/memoryRing.ts new file mode 100644 index 0000000000..c4daa94795 --- /dev/null +++ b/packages/cli/src/ui/hooks/memoryTrend/memoryRing.ts @@ -0,0 +1,79 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Fixed-capacity overwrite ring for in-session memory samples (P10, AC-11). + * + * At the 60 s `MEMORY_CHECK_INTERVAL_MS` cadence, the default capacity of 120 + * samples holds two hours of history for the live `/perf` view. The ring never + * grows beyond its capacity — pushing beyond it overwrites the oldest entry. + * The leak detector must not leak. + * + * Snapshot returns a defensive copy ordered oldest→newest; the caller cannot + * obtain a mutable alias to the internal buffer. + */ + +/** + * A single memory sample held by the ring. Carries the four memory values plus + * the timing context needed for the live view and idle interpretation. + */ +export interface MemoryRingSample { + readonly rss: number; + readonly heapUsed: number; + readonly external: number; + readonly arrayBuffers: number; + readonly uptimeMs: number; + readonly msSinceLastOperation: number; + readonly timestampMs: number; +} + +/** + * Default capacity: 120 samples × 60 s = 2 hours at the monitor cadence. + * Bounded — never grows beyond this regardless of session length. + */ +export const MEMORY_RING_CAPACITY = 120; + +export class MemoryRing { + private readonly buffer: Array; + private readonly capacity: number; + private head = 0; + private len = 0; + + constructor(capacity: number = MEMORY_RING_CAPACITY) { + if (!Number.isInteger(capacity) || capacity < 1) { + throw new RangeError( + `MemoryRing capacity must be a positive integer (got ${capacity})`, + ); + } + this.capacity = capacity; + this.buffer = new Array(capacity); + } + + push(sample: MemoryRingSample): void { + this.buffer[this.head] = sample; + this.head = (this.head + 1) % this.capacity; + if (this.len < this.capacity) { + this.len += 1; + } + } + + get size(): number { + return this.len; + } + + snapshot(): readonly MemoryRingSample[] { + const result: MemoryRingSample[] = []; + const start = this.len < this.capacity ? 0 : this.head; + for (let i = 0; i < this.len; i++) { + const idx = (start + i) % this.capacity; + const entry = this.buffer[idx]; + if (entry !== undefined) { + result.push({ ...entry }); + } + } + return result; + } +} diff --git a/packages/cli/src/ui/hooks/memoryTrend/memorySlope.behavior.test.ts b/packages/cli/src/ui/hooks/memoryTrend/memorySlope.behavior.test.ts new file mode 100644 index 0000000000..8931a45fd5 --- /dev/null +++ b/packages/cli/src/ui/hooks/memoryTrend/memorySlope.behavior.test.ts @@ -0,0 +1,326 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * P10 behavioral tests for read-time memory slope derivation (EVIDENCE-AC10). + * + * Slopes are DERIVED at read time, never persisted. Two axes: + * - per-operation: least-squares of each memory column on session_operation_index + * - per-minute: least-squares of each memory column on uptime_ms (→ bytes/min) + * + * Requires ≥2 usable points and nonzero x variance; otherwise null (never + * NaN/Infinity). Negative slopes are preserved (not clamped). Functions operate + * on a single record series (one run/file) — P11 invokes per file. + */ + +import { describe, it, expect } from 'bun:test'; +import { + derivePerOperationMemorySlope, + derivePerMinuteMemorySlope, +} from './memorySlope.js'; +import type { + PerfOperationRecord, + PerfMemorySampleRecord, +} from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; + +// --------------------------------------------------------------------------- +// Helpers — build valid records with known linear memory data +// --------------------------------------------------------------------------- + +function makeOpRecord( + index: number, + mem: { rss: number; heap: number; ext: number; arr: number }, +): PerfOperationRecord { + return { + schema_version: 1, + record_type: 'operation', + ts: new Date(1_700_000_000_000 + index * 60_000).toISOString(), + session_id: 'sess-1', + operation_id: `op-${index}`, + runtime_id: 'rt-1', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'hash', + llxprt_version: '0.11.0', + git_sha: 'abc1234', + runtime: 'bun-1.3.14', + platform: 'darwin-arm64', + provider: 'openai', + model: 'gpt-4o', + context_tokens: 100, + output_tokens: 50, + terminal_cols: 120, + terminal_rows: 40, + render_mode: 'incremental', + concurrent_instances: 1, + status: 'completed', + client_prepare_ms: 10, + stream_handler_ms: 5, + ink_render_ms: 3, + ink_render_count: 1, + stdout_bytes: 100, + stdout_write_calls: 1, + stdout_write_sync_ms: 1, + client_finalize_ms: 1, + provider_attempts: 1, + provider_attempt_sum_ms: 100, + provider_union_ms: 100, + tool_calls: 0, + tool_call_sum_ms: 0, + tool_union_ms: 0, + agent_activity_union_ms: 100, + operation_elapsed_ms: 500, + approval_wait_ms: 0, + unclassified_elapsed_ms: 400, + session_operation_index: index, + uptime_ms: index * 60_000, + rss_bytes: mem.rss, + heap_used_bytes: mem.heap, + external_bytes: mem.ext, + array_buffers_bytes: mem.arr, + }; +} + +function makeOpRecordNoMemory(index: number): PerfOperationRecord { + const rec = makeOpRecord(index, { rss: 0, heap: 0, ext: 0, arr: 0 }); + const { + rss_bytes: _r, + heap_used_bytes: _h, + external_bytes: _e, + array_buffers_bytes: _a, + ...rest + } = rec; + void _r; + void _h; + void _e; + void _a; + return rest as PerfOperationRecord; +} + +function makeSampleRecord( + uptimeMs: number, + mem: { rss: number; heap: number; ext: number; arr: number }, +): PerfMemorySampleRecord { + return { + schema_version: 1, + record_type: 'memory_sample', + ts: new Date(1_700_000_000_000 + uptimeMs).toISOString(), + rss_bytes: mem.rss, + heap_used_bytes: mem.heap, + external_bytes: mem.ext, + array_buffers_bytes: mem.arr, + uptime_ms: uptimeMs, + ms_since_last_operation: 30_000, + }; +} + +// --------------------------------------------------------------------------- +// Per-operation slope +// --------------------------------------------------------------------------- + +describe('derivePerOperationMemorySlope (AC-10)', () => { + it('derives correct positive slope across all four metrics', () => { + // rss grows 1000 bytes/operation; heap 2000; ext 3000; arr 4000. + const ops: PerfOperationRecord[] = [ + makeOpRecord(0, { rss: 10_000, heap: 20_000, ext: 30_000, arr: 40_000 }), + makeOpRecord(1, { rss: 11_000, heap: 22_000, ext: 33_000, arr: 44_000 }), + makeOpRecord(2, { rss: 12_000, heap: 24_000, ext: 36_000, arr: 48_000 }), + makeOpRecord(3, { rss: 13_000, heap: 26_000, ext: 39_000, arr: 52_000 }), + makeOpRecord(4, { rss: 14_000, heap: 28_000, ext: 42_000, arr: 56_000 }), + ]; + const slope = derivePerOperationMemorySlope(ops); + expect(slope.rss_bytes_per_operation).toBeCloseTo(1000, 5); + expect(slope.heap_used_bytes_per_operation).toBeCloseTo(2000, 5); + expect(slope.external_bytes_per_operation).toBeCloseTo(3000, 5); + expect(slope.array_buffers_bytes_per_operation).toBeCloseTo(4000, 5); + }); + + it('preserves negative slope (memory decreasing)', () => { + const ops: PerfOperationRecord[] = [ + makeOpRecord(0, { rss: 50_000, heap: 50_000, ext: 50_000, arr: 50_000 }), + makeOpRecord(1, { rss: 45_000, heap: 45_000, ext: 45_000, arr: 45_000 }), + makeOpRecord(2, { rss: 40_000, heap: 40_000, ext: 40_000, arr: 40_000 }), + ]; + const slope = derivePerOperationMemorySlope(ops); + expect(slope.rss_bytes_per_operation).toBeCloseTo(-5000, 5); + expect(slope.heap_used_bytes_per_operation).toBeCloseTo(-5000, 5); + }); + + it('returns null for fewer than 2 usable points', () => { + const slope = derivePerOperationMemorySlope([ + makeOpRecord(0, { rss: 10_000, heap: 20_000, ext: 30_000, arr: 40_000 }), + ]); + expect(slope.rss_bytes_per_operation).toBeNull(); + expect(slope.heap_used_bytes_per_operation).toBeNull(); + expect(slope.external_bytes_per_operation).toBeNull(); + expect(slope.array_buffers_bytes_per_operation).toBeNull(); + }); + + it('returns null for empty input', () => { + const slope = derivePerOperationMemorySlope([]); + expect(slope.rss_bytes_per_operation).toBeNull(); + }); + + it('returns null when x variance is zero (all same index)', () => { + const ops: PerfOperationRecord[] = [ + makeOpRecord(5, { rss: 10_000, heap: 20_000, ext: 30_000, arr: 40_000 }), + makeOpRecord(5, { rss: 11_000, heap: 21_000, ext: 31_000, arr: 41_000 }), + ]; + const slope = derivePerOperationMemorySlope(ops); + expect(slope.rss_bytes_per_operation).toBeNull(); + }); + + it('ignores records without memory columns', () => { + const ops: PerfOperationRecord[] = [ + makeOpRecordNoMemory(0), + makeOpRecord(1, { rss: 11_000, heap: 22_000, ext: 33_000, arr: 44_000 }), + makeOpRecord(2, { rss: 12_000, heap: 24_000, ext: 36_000, arr: 48_000 }), + ]; + const slope = derivePerOperationMemorySlope(ops); + // Only 2 usable points (indices 1,2); slope = 1000 bytes/operation. + expect(slope.rss_bytes_per_operation).toBeCloseTo(1000, 5); + }); + + it('exactly 2 points with nonzero variance yields a slope', () => { + const ops: PerfOperationRecord[] = [ + makeOpRecord(0, { rss: 10_000, heap: 20_000, ext: 30_000, arr: 40_000 }), + makeOpRecord(1, { rss: 15_000, heap: 25_000, ext: 35_000, arr: 45_000 }), + ]; + const slope = derivePerOperationMemorySlope(ops); + expect(slope.rss_bytes_per_operation).toBeCloseTo(5000, 5); + }); +}); + +// --------------------------------------------------------------------------- +// Per-minute slope +// --------------------------------------------------------------------------- + +describe('derivePerMinuteMemorySlope (AC-10)', () => { + it('derives correct positive slope across all four metrics', () => { + // rss grows 1 byte/ms = 60000 bytes/min; heap 2; ext 3; arr 4. + const samples: PerfMemorySampleRecord[] = [ + makeSampleRecord(0, { + rss: 10_000, + heap: 20_000, + ext: 30_000, + arr: 40_000, + }), + makeSampleRecord(60_000, { + rss: 70_000, + heap: 140_000, + ext: 210_000, + arr: 280_000, + }), + makeSampleRecord(120_000, { + rss: 130_000, + heap: 260_000, + ext: 390_000, + arr: 520_000, + }), + makeSampleRecord(180_000, { + rss: 190_000, + heap: 380_000, + ext: 570_000, + arr: 760_000, + }), + makeSampleRecord(240_000, { + rss: 250_000, + heap: 500_000, + ext: 750_000, + arr: 1_000_000, + }), + ]; + const slope = derivePerMinuteMemorySlope(samples); + expect(slope.rss_bytes_per_minute).toBeCloseTo(60_000, 0); + expect(slope.heap_used_bytes_per_minute).toBeCloseTo(120_000, 0); + expect(slope.external_bytes_per_minute).toBeCloseTo(180_000, 0); + expect(slope.array_buffers_bytes_per_minute).toBeCloseTo(240_000, 0); + }); + + it('preserves negative slope (memory decreasing over time)', () => { + const samples: PerfMemorySampleRecord[] = [ + makeSampleRecord(0, { + rss: 1_000_000, + heap: 1_000_000, + ext: 1_000_000, + arr: 1_000_000, + }), + makeSampleRecord(60_000, { + rss: 940_000, + heap: 940_000, + ext: 940_000, + arr: 940_000, + }), + makeSampleRecord(120_000, { + rss: 880_000, + heap: 880_000, + ext: 880_000, + arr: 880_000, + }), + ]; + const slope = derivePerMinuteMemorySlope(samples); + // -1 byte/ms × 60000 = -60000 bytes/min. + expect(slope.rss_bytes_per_minute).toBeCloseTo(-60_000, 0); + expect(slope.heap_used_bytes_per_minute).toBeCloseTo(-60_000, 0); + }); + + it('returns null for fewer than 2 usable points', () => { + const slope = derivePerMinuteMemorySlope([ + makeSampleRecord(0, { + rss: 10_000, + heap: 20_000, + ext: 30_000, + arr: 40_000, + }), + ]); + expect(slope.rss_bytes_per_minute).toBeNull(); + expect(slope.heap_used_bytes_per_minute).toBeNull(); + }); + + it('returns null for empty input', () => { + const slope = derivePerMinuteMemorySlope([]); + expect(slope.rss_bytes_per_minute).toBeNull(); + }); + + it('returns null when x variance is zero (all same uptime)', () => { + const samples: PerfMemorySampleRecord[] = [ + makeSampleRecord(60_000, { + rss: 10_000, + heap: 20_000, + ext: 30_000, + arr: 40_000, + }), + makeSampleRecord(60_000, { + rss: 20_000, + heap: 30_000, + ext: 40_000, + arr: 50_000, + }), + ]; + const slope = derivePerMinuteMemorySlope(samples); + expect(slope.rss_bytes_per_minute).toBeNull(); + }); + + it('exactly 2 points with nonzero variance yields a slope', () => { + const samples: PerfMemorySampleRecord[] = [ + makeSampleRecord(0, { + rss: 10_000, + heap: 20_000, + ext: 30_000, + arr: 40_000, + }), + makeSampleRecord(60_000, { + rss: 70_000, + heap: 80_000, + ext: 90_000, + arr: 100_000, + }), + ]; + const slope = derivePerMinuteMemorySlope(samples); + // 1 byte/ms × 60000 = 60000 bytes/min. + expect(slope.rss_bytes_per_minute).toBeCloseTo(60_000, 0); + }); +}); diff --git a/packages/cli/src/ui/hooks/memoryTrend/memorySlope.ts b/packages/cli/src/ui/hooks/memoryTrend/memorySlope.ts new file mode 100644 index 0000000000..7e8cd9804b --- /dev/null +++ b/packages/cli/src/ui/hooks/memoryTrend/memorySlope.ts @@ -0,0 +1,27 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Read-time memory slope derivation (P10, AC-10) — thin re-export layer. + * + * The canonical generic memory slope algorithm and its types are OWNED by the + * telemetry package (`perfSlopeBridge.ts`), below the CLI layer, so the + * longitudinal report and this live view share one implementation rather than + * maintaining parallel copies. This module preserves the historical CLI import + * path (`memoryTrend/memorySlope.js`) for P10 callers while delegating every + * computation to telemetry. + * + * Slopes are DERIVED at read time, never persisted. + */ + +export { + derivePerOperationMemorySlope, + derivePerMinuteMemorySlope, +} from '@vybestack/llxprt-code-telemetry/perf/perfSlopeBridge.js'; +export type { + PerOperationMemorySlope, + PerMinuteMemorySlope, +} from '@vybestack/llxprt-code-telemetry/perf/perfSlopeBridge.js'; diff --git a/packages/cli/src/ui/hooks/memoryTrend/memoryTelemetry.behavior.test.ts b/packages/cli/src/ui/hooks/memoryTrend/memoryTelemetry.behavior.test.ts new file mode 100644 index 0000000000..8203731b1c --- /dev/null +++ b/packages/cli/src/ui/hooks/memoryTrend/memoryTelemetry.behavior.test.ts @@ -0,0 +1,437 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * P10 behavioral tests for MemoryTelemetryController (EVIDENCE-AC10). + * + * Real PerfSink + real temp files + real tolerant reader. The controller is + * constructible (no singleton), shares the existing PerfSink, maintains the + * bounded ring, writes schema-valid memory_sample records, and exposes + * snapshots for P11. No mock theatre. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { PerfSink } from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import { readPerfRecords } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import type { PerfMemorySampleRecord } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import { MemoryTelemetryController } from './memoryTelemetry.js'; +import type { OperationMemorySampler } from './memoryTelemetry.js'; + +let dir: string; +const activeSinks: PerfSink[] = []; + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'memtelemetry-')); + activeSinks.length = 0; +}); + +afterEach(async () => { + const errors: unknown[] = []; + for (const sink of activeSinks) { + try { + await sink.dispose(); + } catch (err) { + errors.push(err); + } + } + activeSinks.length = 0; + fs.rmSync(dir, { recursive: true, force: true }); + if (errors.length === 1) throw errors[0]; + if (errors.length > 1) { + throw new AggregateError( + errors, + 'memoryTelemetry afterEach sink cleanup failed', + ); + } +}); + +function makeSink(): PerfSink { + const sink = new PerfSink({ dir, runUuid: crypto.randomUUID() }); + activeSinks.push(sink); + return sink; +} + +function fixtureMemory(rss: number): NodeJS.MemoryUsage { + return { + rss, + heapUsed: rss + 1000, + external: rss + 2000, + arrayBuffers: rss + 3000, + } as NodeJS.MemoryUsage; +} + +async function readSampleRecords( + controller: MemoryTelemetryController, + sink: PerfSink, +): Promise { + await controller.drain(); + await sink.dispose(); + const files = fs.readdirSync(dir).filter((f) => f.endsWith('.jsonl')); + const records: PerfMemorySampleRecord[] = []; + for (const file of files) { + const result = await readPerfRecords(path.join(dir, file)); + for (const rec of result.records) { + if (rec.record_type === 'memory_sample') { + records.push(rec); + } + } + } + return records; +} + +describe('MemoryTelemetryController — serialized write chain + drain (P12)', () => { + it('drain() resolves when there are no internal errors', async () => { + const sink = makeSink(); + const controller = new MemoryTelemetryController({ + sink, + monotonicNow: () => 1000, + memoryNow: () => fixtureMemory(5000), + }); + controller.recordTickSample(fixtureMemory(5000)); + controller.recordTickSample(fixtureMemory(6000)); + await controller.drain(); + // Both writes completed before drain resolved. + const records = await readSampleRecords(controller, sink); + expect(records).toHaveLength(2); + }); + + it('drain() rejects on internal sink/programming failure', async () => { + // A sink that throws synchronously on write (a programming/schema error). + const failingSink = { + write(): Promise { + throw new Error('internal programming error'); + }, + start(): Promise { + return Promise.resolve(); + }, + dispose(): Promise { + return Promise.resolve(); + }, + get byteCount(): number { + return 0; + }, + get lastWriteErrorCode(): string | null { + return null; + }, + } as unknown as PerfSink; + + const controller = new MemoryTelemetryController({ + sink: failingSink, + monotonicNow: () => 1000, + memoryNow: () => fixtureMemory(5000), + }); + controller.recordTickSample(fixtureMemory(5000)); + await expect(controller.drain()).rejects.toThrow( + 'internal programming error', + ); + }); + + it('writes are serialized in order through the chain', async () => { + const writeOrder: number[] = []; + const resolvers: Array<() => void> = []; + const orderedSink = { + write(_record: unknown): Promise { + const idx = writeOrder.length; + writeOrder.push(idx); + return new Promise((resolve) => { + resolvers.push(resolve); + }); + }, + start(): Promise { + return Promise.resolve(); + }, + dispose(): Promise { + return Promise.resolve(); + }, + get byteCount(): number { + return 0; + }, + get lastWriteErrorCode(): string | null { + return null; + }, + } as unknown as PerfSink; + + const controller = new MemoryTelemetryController({ + sink: orderedSink, + monotonicNow: () => 1000, + memoryNow: () => fixtureMemory(5000), + }); + controller.recordTickSample(fixtureMemory(1000)); + controller.recordTickSample(fixtureMemory(2000)); + controller.recordTickSample(fixtureMemory(3000)); + + // Allow the first chained write's microtask to run. Only write 0 starts; + // writes 1 and 2 are queued behind write 0's unresolved promise. + await new Promise((r) => setTimeout(r, 10)); + expect(writeOrder).toEqual([0]); + + // Resolve the first write; the second starts. + resolvers[0](); + await new Promise((r) => setTimeout(r, 10)); + expect(writeOrder).toEqual([0, 1]); + + // Resolve the rest and drain. + resolvers[1](); + await new Promise((r) => setTimeout(r, 10)); + resolvers[2](); + await controller.drain(); + expect(writeOrder).toEqual([0, 1, 2]); + }); + + it('does not emit a process-level unhandled rejection on internal error', async () => { + // If the no-op local rejection observer is missing, a rejected write + // chain that was not individually awaited becomes an unhandled rejection. + // We assert drain() surfaces the error AND no unhandledRejection event + // fires during the window. + let unhandledFired = false; + const handler = (): void => { + unhandledFired = true; + }; + process.on('unhandledRejection', handler); + + try { + const failingSink = { + write(): Promise { + return Promise.reject(new Error('internal programming error')); + }, + start(): Promise { + return Promise.resolve(); + }, + dispose(): Promise { + return Promise.resolve(); + }, + get byteCount(): number { + return 0; + }, + get lastWriteErrorCode(): string | null { + return null; + }, + } as unknown as PerfSink; + + const controller = new MemoryTelemetryController({ + sink: failingSink, + monotonicNow: () => 1000, + memoryNow: () => fixtureMemory(5000), + }); + controller.recordTickSample(fixtureMemory(5000)); + // Give the microtask queue a chance to emit unhandledRejection. + await new Promise((r) => setTimeout(r, 50)); + await expect(controller.drain()).rejects.toThrow( + 'internal programming error', + ); + // Allow any pending microtasks to settle. + await new Promise((r) => setTimeout(r, 50)); + expect(unhandledFired).toBe(false); + } finally { + process.off('unhandledRejection', handler); + } + }); +}); + +describe('MemoryTelemetryController — recordTickSample (AC-10)', () => { + it('writes a schema-valid memory_sample record with correct values', async () => { + const mono = 5000; + const wall = 1_700_000_000_000; + const sink = makeSink(); + const controller = new MemoryTelemetryController({ + sink, + wallNow: () => wall, + monotonicNow: () => mono, + memoryNow: () => fixtureMemory(50_000_000), + }); + + controller.recordTickSample(fixtureMemory(50_000_000)); + const records = await readSampleRecords(controller, sink); + expect(records).toHaveLength(1); + const rec = records[0]; + expect(rec.record_type).toBe('memory_sample'); + expect(rec.schema_version).toBe(1); + expect(rec.rss_bytes).toBe(50_000_000); + expect(rec.heap_used_bytes).toBe(50_001_000); + expect(rec.external_bytes).toBe(50_002_000); + expect(rec.array_buffers_bytes).toBe(50_003_000); + expect(rec.uptime_ms).toBe(5000); + // Pre-first-operation: ms_since_last_operation = uptime (no fabricated + // previous operation). + expect(rec.ms_since_last_operation).toBe(5000); + // ISO 8601 timestamp from the wall clock. + expect(rec.ts).toBe(new Date(wall).toISOString()); + }); + + it('pre-first-operation ms_since_last_operation equals uptime (honest)', async () => { + const mono = 42_000; + const sink = makeSink(); + const controller = new MemoryTelemetryController({ + sink, + monotonicNow: () => mono, + memoryNow: () => fixtureMemory(1000), + }); + + controller.recordTickSample(fixtureMemory(1000)); + const records = await readSampleRecords(controller, sink); + // No operation has ended, so idle = uptime since process start. + expect(records[0].ms_since_last_operation).toBe(42_000); + expect(records[0].uptime_ms).toBe(42_000); + }); + + it('after markOperationEnd, ms_since_last_operation is uptime minus last op end', async () => { + let mono = 10_000; + const sink = makeSink(); + const controller = new MemoryTelemetryController({ + sink, + monotonicNow: () => mono, + memoryNow: () => fixtureMemory(1000), + }); + + // First tick at uptime 10_000 — pre-first-operation idle. + controller.recordTickSample(fixtureMemory(1000)); + + // Operation ends at uptime 15_000. + mono = 15_000; + controller.markOperationEnd(); + + // Second tick at uptime 25_000 — idle = 25_000 - 15_000 = 10_000. + mono = 25_000; + controller.recordTickSample(fixtureMemory(2000)); + + const records = await readSampleRecords(controller, sink); + expect(records).toHaveLength(2); + expect(records[0].ms_since_last_operation).toBe(10_000); + expect(records[1].ms_since_last_operation).toBe(10_000); + expect(records[1].uptime_ms).toBe(25_000); + }); +}); + +describe('MemoryTelemetryController — ring snapshot (AC-10/AC-11)', () => { + it('snapshot exposes ring contents oldest→newest', async () => { + let mono = 0; + const sink = makeSink(); + const controller = new MemoryTelemetryController({ + sink, + monotonicNow: () => mono, + memoryNow: () => fixtureMemory(1000), + }); + + mono = 1000; + controller.recordTickSample(fixtureMemory(10_000)); + mono = 2000; + controller.recordTickSample(fixtureMemory(20_000)); + mono = 3000; + controller.recordTickSample(fixtureMemory(30_000)); + + const snap = controller.snapshot(); + expect(snap).toHaveLength(3); + expect(snap[0].rss).toBe(10_000); + expect(snap[1].rss).toBe(20_000); + expect(snap[2].rss).toBe(30_000); + await controller.drain(); + }); +}); + +describe('MemoryTelemetryController — operation-end memory (AC-10)', () => { + it('sampleOperationEndMemory returns the four columns', () => { + const sink = makeSink(); + const controller = new MemoryTelemetryController({ + sink, + memoryNow: () => fixtureMemory(77_777_777), + }); + + const columns = controller.sampleOperationEndMemory(); + expect(columns.rss_bytes).toBe(77_777_777); + expect(columns.heap_used_bytes).toBe(77_778_777); + expect(columns.external_bytes).toBe(77_779_777); + expect(columns.array_buffers_bytes).toBe(77_780_777); + }); + + it('implements OperationMemorySampler interface', () => { + const sink = makeSink(); + const controller = new MemoryTelemetryController({ + sink, + memoryNow: () => fixtureMemory(1000), + }); + const sampler: OperationMemorySampler = controller; + expect(typeof sampler.markOperationEnd).toBe('function'); + expect(typeof sampler.sampleOperationEndMemory).toBe('function'); + }); + + it('markOperationEnd + sampleOperationEndMemory are idempotent for the columns', () => { + const mono = 5000; + const sink = makeSink(); + const controller = new MemoryTelemetryController({ + sink, + monotonicNow: () => mono, + memoryNow: () => fixtureMemory(999), + }); + + controller.markOperationEnd(); + const cols1 = controller.sampleOperationEndMemory(); + const cols2 = controller.sampleOperationEndMemory(); + expect(cols1).toEqual(cols2); + }); +}); + +describe('MemoryTelemetryController — no slope key in persisted records', () => { + it('memory_sample records have no slope-related fields', async () => { + const sink = makeSink(); + const controller = new MemoryTelemetryController({ + sink, + monotonicNow: () => 1000, + memoryNow: () => fixtureMemory(5000), + }); + controller.recordTickSample(fixtureMemory(5000)); + const records = await readSampleRecords(controller, sink); + expect(records).toHaveLength(1); + const raw = JSON.parse( + fs.readFileSync( + path.join( + dir, + fs.readdirSync(dir).filter((f) => f.endsWith('.jsonl'))[0], + ), + 'utf8', + ), + ); + // No slope field anywhere in the persisted record. + for (const key of Object.keys(raw)) { + expect(key).not.toMatch(/slope/i); + } + }); +}); + +describe('MemoryTelemetryController — wallNow captured once (P11 quality fix)', () => { + it('ring timestamp and record ts derive from the same wallNow call', async () => { + // A wallNow that increments on every call so two calls are distinguishable. + let wallCallCount = 0; + const baseWall = 1_700_000_000_000; + const wallNow = (): number => { + wallCallCount++; + return baseWall + (wallCallCount - 1) * 1000; + }; + + const sink = makeSink(); + const controller = new MemoryTelemetryController({ + sink, + wallNow, + monotonicNow: () => 5000, + memoryNow: () => fixtureMemory(50_000_000), + }); + + controller.recordTickSample(fixtureMemory(50_000_000)); + + const snap = controller.snapshot(); + expect(snap).toHaveLength(1); + + const records = await readSampleRecords(controller, sink); + expect(records).toHaveLength(1); + + // Both the ring timestamp and the record ts must derive from the same + // wallNow() call. If wallNow were called twice (the pre-fix bug), the + // record ts would be 1000ms later than the ring timestamp. + expect(new Date(records[0].ts).getTime()).toBe(snap[0].timestampMs); + }); +}); diff --git a/packages/cli/src/ui/hooks/memoryTrend/memoryTelemetry.ts b/packages/cli/src/ui/hooks/memoryTrend/memoryTelemetry.ts new file mode 100644 index 0000000000..125095e4f3 --- /dev/null +++ b/packages/cli/src/ui/hooks/memoryTrend/memoryTelemetry.ts @@ -0,0 +1,181 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Constructible memory telemetry controller (P10, AC-10). + * + * Not a singleton: P12 constructs one only when perf AND memory telemetry are + * enabled. It shares the existing PerfSink, maintains the bounded MemoryRing, + * writes schema-valid `memory_sample` records, and exposes snapshots for the + * P11 live `/perf` view. + * + * Two call sites: + * - useMemoryMonitor's 60 s tick → `recordTickSample` (memory pre-sampled once + * by the hook, passed in). + * - OperationLifecycleRegistry finalise → `markOperationEnd` + + * `sampleOperationEndMemory` (the controller samples memory itself once). + * + * Pre-first-operation `ms_since_last_operation` is the full uptime since + * process start — honest, not a fabricated previous operation. + */ + +import { + PERF_SCHEMA_VERSION, + PERF_RECORD_TYPE_MEMORY_SAMPLE, +} from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import type { + PerfSink, + PerfMemorySampleRecord, +} from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import { MemoryRing } from './memoryRing.js'; +import type { MemoryRingSample } from './memoryRing.js'; + +/** + * The four operation-end memory columns. Present on the operation record IFF + * memory telemetry is enabled; omitted (never zeroed) when disabled. + */ +export interface MemoryColumns { + readonly rss_bytes: number; + readonly heap_used_bytes: number; + readonly external_bytes: number; + readonly array_buffers_bytes: number; +} + +/** + * Narrow interface the OperationLifecycleRegistry depends on. Decouples the + * registry from the full controller so P12 can wire it with the real + * controller or a test double. + */ +export interface OperationMemorySampler { + /** Records the monotonic time of operation end for idle computation. */ + markOperationEnd(): void; + /** Samples process.memoryUsage() once and returns the four memory columns. */ + sampleOperationEndMemory(): MemoryColumns; +} + +export interface MemoryTelemetryControllerOptions { + readonly sink: PerfSink; + /** Wall-clock epoch millis for the record `ts`. Defaults to Date.now. */ + readonly wallNow?: () => number; + /** Monotonic millis for uptime/idle. Defaults to performance.now. */ + readonly monotonicNow?: () => number; + /** Memory sampler for operation-end. Defaults to process.memoryUsage. */ + readonly memoryNow?: () => NodeJS.MemoryUsage; +} + +export class MemoryTelemetryController implements OperationMemorySampler { + private readonly ring = new MemoryRing(); + private readonly sink: PerfSink; + private readonly wallNow: () => number; + private readonly monotonicNow: () => number; + private readonly memoryNow: () => NodeJS.MemoryUsage; + private lastOperationEndMs: number | null = null; + /** + * Controller-owned serialized write chain. Replaces the former + * `void sink.write(...)` fire-and-forget so writes happen in order and + * internal errors surface deterministically via {@link drain}. A local + * no-op rejection handler is attached so a rejected write whose individual + * promise was not awaited does not become a process-level unhandled + * rejection; the chain itself remains rejected so drain() fails fast. + */ + private writeChain: Promise = Promise.resolve(); + + constructor(options: MemoryTelemetryControllerOptions) { + this.sink = options.sink; + this.wallNow = options.wallNow ?? (() => Date.now()); + this.monotonicNow = options.monotonicNow ?? (() => performance.now()); + this.memoryNow = options.memoryNow ?? (() => process.memoryUsage()); + } + + /** + * Records a periodic memory sample from the 60 s tick. The caller + * (useMemoryMonitor) has already called process.memoryUsage() once for the + * warning check; the same full sample is passed here so there is exactly one + * memoryUsage() call per tick. + * + * Pushes to the bounded ring and writes a schema-valid memory_sample record + * through the shared PerfSink. + */ + recordTickSample(sample: NodeJS.MemoryUsage): void { + // Capture wallNow exactly once so the ring timestamp and the record ts + // describe the same sample (P11 quality fix). + const wallMs = this.wallNow(); + const uptime = this.monotonicNow(); + const msSinceLastOperation = + this.lastOperationEndMs === null + ? uptime + : Math.max(0, uptime - this.lastOperationEndMs); + + const ringSample: MemoryRingSample = { + rss: sample.rss, + heapUsed: sample.heapUsed, + external: sample.external, + arrayBuffers: sample.arrayBuffers, + uptimeMs: uptime, + msSinceLastOperation, + timestampMs: wallMs, + }; + this.ring.push(ringSample); + + const record: PerfMemorySampleRecord = { + schema_version: PERF_SCHEMA_VERSION, + record_type: PERF_RECORD_TYPE_MEMORY_SAMPLE, + ts: new Date(wallMs).toISOString(), + rss_bytes: sample.rss, + heap_used_bytes: sample.heapUsed, + external_bytes: sample.external, + array_buffers_bytes: sample.arrayBuffers, + uptime_ms: uptime, + ms_since_last_operation: msSinceLastOperation, + }; + void this.writeSerialized(record); + } + + /** + * Queues a record write through the controller-owned serialized chain. + * Attaches a local no-op rejection handler so a rejected write that was not + * individually awaited does not become a process-level unhandled rejection. + * The chain itself remains rejected so {@link drain} surfaces the error. + * External errno filesystem writes resolve fail-open inside PerfSink. + */ + private writeSerialized(record: PerfMemorySampleRecord): Promise { + this.writeChain = this.writeChain.then(() => this.sink.write(record)); + void this.writeChain.catch(() => {}); + return this.writeChain; + } + + /** + * Awaits all queued memory_sample writes through the serialized chain. + * The runtime (before disposal) and tests call this to deterministically + * flush pending writes. Rejects on internal async sink/programming failures; + * external errno filesystem writes already resolve fail-open in PerfSink. + */ + async drain(): Promise { + await this.writeChain; + } + + markOperationEnd(): void { + this.lastOperationEndMs = this.monotonicNow(); + } + + sampleOperationEndMemory(): MemoryColumns { + const s = this.memoryNow(); + return { + rss_bytes: s.rss, + heap_used_bytes: s.heapUsed, + external_bytes: s.external, + array_buffers_bytes: s.arrayBuffers, + }; + } + + /** + * Returns a defensive copy of the ring contents (oldest→newest) for the P11 + * live `/perf` view. + */ + snapshot(): readonly MemoryRingSample[] { + return this.ring.snapshot(); + } +} diff --git a/packages/cli/src/ui/hooks/memoryTrend/useMemoryMonitor.behavior.test.ts b/packages/cli/src/ui/hooks/memoryTrend/useMemoryMonitor.behavior.test.ts new file mode 100644 index 0000000000..3bc64d8ec0 --- /dev/null +++ b/packages/cli/src/ui/hooks/memoryTrend/useMemoryMonitor.behavior.test.ts @@ -0,0 +1,394 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +// Enable React's act() environment so hook state updates are flushed. +( + globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean } +).IS_REACT_ACT_ENVIRONMENT = true; + +/** + * P10 behavioral tests for useMemoryMonitor (EVIDENCE-AC11). + * + * Tests the extended 60 s interval: warn-once latch is separated from the + * sampling loop (DEFECT 1 fix — the interval no longer clearInterval's itself + * after warning). No new timer is created. When a memory controller is present, + * each tick records a sample; when absent, warn-only behavior is retained. + * + * Uses package-private timer/memory ports for deterministic behavior. No mock + * theatre — real hook lifecycle via renderHook, real MemoryTelemetryController + * with a real PerfSink. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import { renderHook } from '../../../test-utils/render.js'; +import type { HistoryItemWithoutId } from '../../types.js'; +import { + useMemoryMonitor, + __setMemoryMonitorPortsForTesting, + __getRealMemoryMonitorPortsForTesting, + type MemoryMonitorPorts, +} from '../useMemoryMonitor.js'; +import { MemoryTelemetryController } from './memoryTelemetry.js'; +import { PerfSink } from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; + +let dir: string; +const activeControllers: MemoryTelemetryController[] = []; +const activeSinks: PerfSink[] = []; + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'usememmon-')); + activeControllers.length = 0; + activeSinks.length = 0; +}); + +afterEach(async () => { + __setMemoryMonitorPortsForTesting(null); + const errors: unknown[] = []; + for (const controller of activeControllers) { + try { + await controller.drain(); + } catch (error) { + errors.push(error); + } + } + for (const sink of activeSinks) { + try { + await sink.dispose(); + } catch (error) { + errors.push(error); + } + } + activeControllers.length = 0; + activeSinks.length = 0; + try { + fs.rmSync(dir, { recursive: true, force: true }); + } catch (error) { + errors.push(error); + } + if (errors.length === 1) throw errors[0]; + if (errors.length > 1) { + throw new AggregateError(errors, 'memory monitor test cleanup failed'); + } +}); + +function fixtureMemory(rss: number): NodeJS.MemoryUsage { + return { + rss, + heapUsed: rss + 1000, + external: rss + 2000, + arrayBuffers: rss + 3000, + heapTotal: rss + 5000, + } as NodeJS.MemoryUsage; +} + +/** + * Controllable interval: stores the handler so tests can fire it + * deterministically, and counts setInterval/clearInterval calls. + */ +function makeControllablePorts( + memoryUsage: () => NodeJS.MemoryUsage, +): MemoryMonitorPorts & { + fireTick: () => void; + intervalCount: () => number; + clearCount: () => number; + isCleared: () => boolean; +} { + let handler: (() => void) | null = null; + let intervals = 0; + let clears = 0; + return { + setInterval: (h: () => void, _ms: number) => { + intervals += 1; + handler = h; + return intervals; // unique id + }, + clearInterval: () => { + clears += 1; + handler = null; + }, + memoryUsage, + rssBytes: () => memoryUsage().rss, + fireTick: () => { + if (handler !== null) handler(); + }, + intervalCount: () => intervals, + clearCount: () => clears, + isCleared: () => handler === null, + }; +} + +function makeAddItem(): { + addItem: (item: HistoryItemWithoutId, ts: number) => void; + calls: () => number; +} { + let calls = 0; + return { + addItem: () => { + calls += 1; + }, + calls: () => calls, + }; +} + +describe('useMemoryMonitor — one interval, no new timers (AC-11)', () => { + it('creates exactly one interval on mount', () => { + const ports = makeControllablePorts(() => fixtureMemory(1000)); + __setMemoryMonitorPortsForTesting(ports); + const { addItem } = makeAddItem(); + + const { unmount } = renderHook(() => useMemoryMonitor({ addItem })); + expect(ports.intervalCount()).toBe(1); + unmount(); + }); + + it('clears the interval on unmount', () => { + const ports = makeControllablePorts(() => fixtureMemory(1000)); + __setMemoryMonitorPortsForTesting(ports); + const { addItem } = makeAddItem(); + + const { unmount } = renderHook(() => useMemoryMonitor({ addItem })); + expect(ports.clearCount()).toBe(0); + unmount(); + expect(ports.clearCount()).toBe(1); + }); +}); + +describe('useMemoryMonitor — warn-once latch separated from sampling (AC-11)', () => { + it('fires warning once but interval continues (DEFECT 1 fix)', () => { + const ports = makeControllablePorts(() => + fixtureMemory(8 * 1024 * 1024 * 1024), + ); + __setMemoryMonitorPortsForTesting(ports); + const { addItem, calls: warningCalls } = makeAddItem(); + + const { unmount } = renderHook(() => useMemoryMonitor({ addItem })); + + // Tick 1: fires warning (rss > 7GB threshold) + ports.fireTick(); + expect(warningCalls()).toBe(1); + // Interval is STILL active — not cleared. + expect(ports.isCleared()).toBe(false); + + // Tick 2: warning NOT repeated (latch), but interval still running. + ports.fireTick(); + expect(warningCalls()).toBe(1); + expect(ports.isCleared()).toBe(false); + + // Tick 3: still running. + ports.fireTick(); + expect(warningCalls()).toBe(1); + expect(ports.isCleared()).toBe(false); + + unmount(); + }); + + it('does not fire warning when rss is below threshold', () => { + const ports = makeControllablePorts(() => fixtureMemory(1_000_000)); + __setMemoryMonitorPortsForTesting(ports); + const { addItem, calls: warningCalls } = makeAddItem(); + + const { unmount } = renderHook(() => useMemoryMonitor({ addItem })); + ports.fireTick(); + ports.fireTick(); + ports.fireTick(); + expect(warningCalls()).toBe(0); + expect(ports.isCleared()).toBe(false); + unmount(); + }); +}); + +describe('useMemoryMonitor — memory off (no controller) retains warn-only (AC-10)', () => { + it('without controller, warn-only behavior is retained', () => { + const ports = makeControllablePorts(() => fixtureMemory(1000)); + __setMemoryMonitorPortsForTesting(ports); + const { addItem, calls: warningCalls } = makeAddItem(); + + const { unmount } = renderHook(() => useMemoryMonitor({ addItem })); + ports.fireTick(); + ports.fireTick(); + expect(warningCalls()).toBe(0); + expect(ports.isCleared()).toBe(false); + expect(ports.intervalCount()).toBe(1); + unmount(); + }); +}); + +describe('useMemoryMonitor — memory on records tick samples (AC-10)', () => { + it('with controller, each tick records a sample to the ring', () => { + let rss = 10_000_000; + const ports = makeControllablePorts(() => fixtureMemory(rss)); + __setMemoryMonitorPortsForTesting(ports); + const { addItem } = makeAddItem(); + + const sink = new PerfSink({ dir, runUuid: crypto.randomUUID() }); + const controller = new MemoryTelemetryController({ + sink, + monotonicNow: () => 0, + memoryNow: () => fixtureMemory(rss), + }); + activeSinks.push(sink); + activeControllers.push(controller); + + const { unmount } = renderHook(() => + useMemoryMonitor({ addItem, memoryController: controller }), + ); + + rss = 20_000_000; + ports.fireTick(); + rss = 30_000_000; + ports.fireTick(); + + const snap = controller.snapshot(); + expect(snap).toHaveLength(2); + expect(snap[0].rss).toBe(20_000_000); + expect(snap[1].rss).toBe(30_000_000); + + unmount(); + }); +}); + +describe('useMemoryMonitor — disabled path uses rss() not full memoryUsage (P12)', () => { + it('without controller, tick calls rssBytes() not full memoryUsage()', () => { + let fullCalls = 0; + let rssCalls = 0; + const handler: { current: (() => void) | null } = { current: null }; + const ports: MemoryMonitorPorts = { + setInterval: (h: () => void, _ms: number) => { + handler.current = h; + return 1; + }, + clearInterval: () => { + handler.current = null; + }, + memoryUsage: () => { + fullCalls++; + return fixtureMemory(1000); + }, + rssBytes: () => { + rssCalls++; + return 1000; + }, + }; + __setMemoryMonitorPortsForTesting(ports); + const { addItem } = makeAddItem(); + + const { unmount } = renderHook(() => useMemoryMonitor({ addItem })); + // The hook mounted and setInterval captured the handler. + handler.current?.(); + + expect(rssCalls).toBe(1); + expect(fullCalls).toBe(0); + + unmount(); + }); + + it('with controller, tick calls full memoryUsage() exactly once per tick', () => { + let fullCalls = 0; + let rssCalls = 0; + const handler: { current: (() => void) | null } = { current: null }; + const ports: MemoryMonitorPorts = { + setInterval: (h: () => void, _ms: number) => { + handler.current = h; + return 1; + }, + clearInterval: () => { + handler.current = null; + }, + memoryUsage: () => { + fullCalls++; + return fixtureMemory(10_000_000); + }, + rssBytes: () => { + rssCalls++; + return 10_000_000; + }, + }; + __setMemoryMonitorPortsForTesting(ports); + const { addItem } = makeAddItem(); + const sink = new PerfSink({ dir, runUuid: crypto.randomUUID() }); + const controller = new MemoryTelemetryController({ + sink, + monotonicNow: () => 0, + memoryNow: () => fixtureMemory(10_000_000), + }); + activeSinks.push(sink); + activeControllers.push(controller); + + const { unmount } = renderHook(() => + useMemoryMonitor({ addItem, memoryController: controller }), + ); + + handler.current?.(); + // Enabled tick: exactly one full memoryUsage call, zero rssBytes calls. + expect(fullCalls).toBe(1); + expect(rssCalls).toBe(0); + + unmount(); + }); + + describe('useMemoryMonitor — real default disabled port (P12)', () => { + it('the real production default rssBytes calls process.memoryUsage.rss(), not process.memoryUsage().rss', () => { + // Prove the real default disabled path uses the cheap rss accessor + // (process.memoryUsage.rss()) rather than allocating a full MemoryUsage + // object (process.memoryUsage().rss). Spies on both the .rss accessor + // and the full memoryUsage call on a wrapper that preserves .rss. + const origMu = process.memoryUsage; + try { + let rssCalls = 0; + let fullCalls = 0; + const wrapper = function memoryUsage(): NodeJS.MemoryUsage { + fullCalls += 1; + return origMu.call(process); + } as typeof process.memoryUsage; + (wrapper as unknown as { rss: () => number }).rss = + function rss(): number { + rssCalls += 1; + return 99_999; + }; + ( + process as unknown as { memoryUsage: typeof process.memoryUsage } + ).memoryUsage = wrapper; + + const real = __getRealMemoryMonitorPortsForTesting(); + const result = real.rssBytes(); + + expect(rssCalls).toBe(1); + expect(fullCalls).toBe(0); + expect(result).toBe(99_999); + } finally { + ( + process as unknown as { memoryUsage: typeof process.memoryUsage } + ).memoryUsage = origMu; + } + }); + + it('the real production default memoryUsage calls process.memoryUsage() (controller-enabled tick)', () => { + const origMu = process.memoryUsage; + try { + let fullCalls = 0; + const wrapper = function memoryUsage(): NodeJS.MemoryUsage { + fullCalls += 1; + return origMu.call(process); + } as typeof process.memoryUsage; + ( + process as unknown as { memoryUsage: typeof process.memoryUsage } + ).memoryUsage = wrapper; + + const real = __getRealMemoryMonitorPortsForTesting(); + real.memoryUsage(); + + expect(fullCalls).toBe(1); + } finally { + ( + process as unknown as { memoryUsage: typeof process.memoryUsage } + ).memoryUsage = origMu; + } + }); + }); +}); diff --git a/packages/cli/src/ui/hooks/perf/dynamicIdentity.behavior.test.ts b/packages/cli/src/ui/hooks/perf/dynamicIdentity.behavior.test.ts new file mode 100644 index 0000000000..9c6620632e --- /dev/null +++ b/packages/cli/src/ui/hooks/perf/dynamicIdentity.behavior.test.ts @@ -0,0 +1,235 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * P12 dynamic identity persistence behavior test (Item 6). + * + * Mutates provider/model and terminal geometry between two operations through + * getter-based identity. Asserts records contain the corresponding snapshots + * for each operation. Also asserts platform contains + * `${process.platform}-${process.arch}`. + * + * The identity provider uses getters so each registry.begin() snapshots + * CURRENT values rather than freezing startup values. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import { promises as fsp } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { + PerfSink, + PerfRetention, +} from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import { readPerfRecords } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import type { PerfOperationRecord } from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; +import { OperationLifecycleRegistry } from '../agentStream/operationLifecycle.js'; +import { + createIdentityProviderFromGetters, + resolvePlatformArch, +} from './interactivePerfRuntime.js'; + +let dir: string; +let sink: PerfSink; +let retention: PerfRetention; + +beforeEach(async () => { + dir = await fsp.mkdtemp(join(tmpdir(), 'perf-dyn-id-')); + const runUuid = crypto.randomUUID(); + retention = new PerfRetention({ dir, runUuid }); + sink = new PerfSink({ dir, runUuid, retention }); + await sink.start(); +}); + +afterEach(async () => { + try { + await sink.dispose(); + } catch { + // ignore + } + try { + await fsp.rm(dir, { recursive: true, force: true }); + } catch { + // ignore + } +}); + +async function drainAndRead(): Promise { + const names = await fsp.readdir(dir); + const records: PerfOperationRecord[] = []; + for (const name of names) { + if (!name.endsWith('.jsonl')) continue; + const result = await readPerfRecords(join(dir, name)); + for (const r of result.records) { + if (r.record_type === 'operation') { + records.push(r); + } + } + } + return records; +} + +describe('dynamic identity persistence — mutable getters snapshot per operation (Item 6)', () => { + it('mutating provider/model/geometry between operations yields correct snapshots', async () => { + // Mutable state read by getters at each begin(). + let currentProvider = 'openai'; + let currentModel = 'gpt-4o'; + let currentCols = 80; + let currentRows = 24; + let currentRenderMode = 'incremental'; + + const identityProvider = createIdentityProviderFromGetters( + { + sessionId: 'sess-dyn', + runtimeId: 'rt-dyn', + projectHash: 'hash-dyn', + cliVersion: '0.11.0', + gitSha: 'abc1234', + runtime: 'bun-1.3.14', + platform: resolvePlatformArch(), + }, + { + provider: () => currentProvider, + model: () => currentModel, + terminalCols: () => currentCols, + terminalRows: () => currentRows, + renderMode: () => currentRenderMode, + }, + ); + + const registry = new OperationLifecycleRegistry({ + identityProvider, + sink, + retention, + }); + + // Operation 1: openai / gpt-4o / 80x24 / incremental. + const ac1 = new AbortController(); + registry.begin(ac1.signal, 'sess#agentic-loop#op-1'); + await registry.finalise(ac1.signal, 'completed'); + + // Mutate between operations. + currentProvider = 'anthropic'; + currentModel = 'claude-sonnet-4-20250514'; + currentCols = 120; + currentRows = 40; + currentRenderMode = 'plain'; + + // Operation 2: anthropic / claude / 120x40 / plain. + const ac2 = new AbortController(); + registry.begin(ac2.signal, 'sess#agentic-loop#op-2'); + await registry.finalise(ac2.signal, 'completed'); + + await registry.drain(); + const records = await drainAndRead(); + expect(records).toHaveLength(2); + + // Operation 1 snapshot. + expect(records[0]?.provider).toBe('openai'); + expect(records[0]?.model).toBe('gpt-4o'); + expect(records[0]?.terminal_cols).toBe(80); + expect(records[0]?.terminal_rows).toBe(24); + expect(records[0]?.render_mode).toBe('incremental'); + + // Operation 2 snapshot — reflects mutated values. + expect(records[1]?.provider).toBe('anthropic'); + expect(records[1]?.model).toBe('claude-sonnet-4-20250514'); + expect(records[1]?.terminal_cols).toBe(120); + expect(records[1]?.terminal_rows).toBe(40); + expect(records[1]?.render_mode).toBe('plain'); + }); + + it('platform field in records contains process.platform-process.arch', async () => { + const identityProvider = createIdentityProviderFromGetters( + { + sessionId: 'sess-plt', + runtimeId: 'rt-plt', + projectHash: 'hash-plt', + cliVersion: '0.11.0', + gitSha: 'abc1234', + runtime: 'bun-1.3.14', + platform: resolvePlatformArch(), + }, + { + provider: () => 'test', + model: () => 'test', + terminalCols: () => 80, + terminalRows: () => 24, + renderMode: () => 'ink', + }, + ); + + const registry = new OperationLifecycleRegistry({ + identityProvider, + sink, + retention, + }); + + const ac = new AbortController(); + registry.begin(ac.signal, 'sess#agentic-loop#op-plt'); + await registry.finalise(ac.signal, 'completed'); + + await registry.drain(); + const records = await drainAndRead(); + expect(records).toHaveLength(1); + expect(records[0]?.platform).toBe(`${process.platform}-${process.arch}`); + }); + + it('immutable fields are identical across operations even when mutable fields change', async () => { + let currentProvider = 'a'; + const identityProvider = createIdentityProviderFromGetters( + { + sessionId: 'sess-imm', + runtimeId: 'rt-imm', + projectHash: 'hash-imm', + cliVersion: '0.11.0', + gitSha: 'deadbeef', + runtime: 'bun', + platform: resolvePlatformArch(), + }, + { + provider: () => currentProvider, + model: () => 'fixed-model', + terminalCols: () => 80, + terminalRows: () => 24, + renderMode: () => 'ink', + }, + ); + + const registry = new OperationLifecycleRegistry({ + identityProvider, + sink, + retention, + }); + + const ac1 = new AbortController(); + registry.begin(ac1.signal, 'sess#agentic-loop#imm-1'); + await registry.finalise(ac1.signal, 'completed'); + + currentProvider = 'b'; + + const ac2 = new AbortController(); + registry.begin(ac2.signal, 'sess#agentic-loop#imm-2'); + await registry.finalise(ac2.signal, 'completed'); + + await registry.drain(); + const records = await drainAndRead(); + expect(records).toHaveLength(2); + + // Immutable fields are the same across both operations. + expect(records[0]?.session_id).toBe(records[1]?.session_id); + expect(records[0]?.runtime_id).toBe(records[1]?.runtime_id); + expect(records[0]?.project_hash).toBe(records[1]?.project_hash); + expect(records[0]?.llxprt_version).toBe(records[1]?.llxprt_version); + expect(records[0]?.git_sha).toBe(records[1]?.git_sha); + expect(records[0]?.runtime).toBe(records[1]?.runtime); + expect(records[0]?.platform).toBe(records[1]?.platform); + + // Mutable provider changed. + expect(records[0]?.provider).toBe('a'); + expect(records[1]?.provider).toBe('b'); + }); +}); diff --git a/packages/cli/src/ui/hooks/perf/interactiveLifecycle.behavior.test.ts b/packages/cli/src/ui/hooks/perf/interactiveLifecycle.behavior.test.ts new file mode 100644 index 0000000000..4eb0b2023c --- /dev/null +++ b/packages/cli/src/ui/hooks/perf/interactiveLifecycle.behavior.test.ts @@ -0,0 +1,560 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * P12 behavioral tests for interactive instance/owner lifecycle composition. + * + * Every assertion exercises ACTUAL production helpers — never mirrored or + * copied cleanup/render-catch logic: + * - {@link replacePreviousInstanceAndOwner} (exported from interactiveUI.tsx) + * driven through the tracked-state test seam + * {@link __setTrackedInstanceAndOwnerForTesting}. + * - {@link cleanupInstanceAndOwner} and {@link rollbackInteractiveFailure} + * (exported from session/interactiveUiLifecycle.ts), the same routines the + * production composition calls for pre-start replacement, registered global + * cleanup, render-failure rollback, and post-render setup-failure teardown. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import { promises as fsp } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { + createInteractivePerfRuntime, + type InteractivePerfRuntimeOptions, +} from './interactivePerfRuntime.js'; +import type { OperationIdentitySnapshot } from '../agentStream/operationLifecycle.js'; +import { + setInteractiveStdoutObserver, + setInteractiveRenderObserver, + getInteractiveStdoutObserver, + getInteractiveRenderObserver, +} from '../../inkRenderOptions.js'; +import { + getPerfPhaseObserver, + setPerfPhaseObserver, +} from '@vybestack/llxprt-code-telemetry/perf/perfPhaseObserver.js'; +import type { PerfScheduler } from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import { + replacePreviousInstanceAndOwner, + __setTrackedInstanceAndOwnerForTesting, + __resetInteractiveUIStateForTesting, +} from '../../../session/interactiveUI.js'; +import { + cleanupInstanceAndOwner, + rollbackInteractiveFailure, +} from '../../../session/interactiveUiLifecycle.js'; + +let dir: string; + +function fixtureIdentity(): OperationIdentitySnapshot { + return { + session_id: 'sess-lifecycle', + runtime_id: 'rt-lifecycle', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'hash-lifecycle', + llxprt_version: '0.11.0', + git_sha: 'abc1234', + runtime: 'bun-1.3.14', + platform: `${process.platform}-${process.arch}`, + provider: 'test-provider', + model: 'test-model', + terminal_cols: 80, + terminal_rows: 24, + render_mode: 'incremental', + }; +} + +function makeOptions( + overrides: Partial = {}, +): InteractivePerfRuntimeOptions & { perfDir: string } { + return { + enabled: true, + memoryEnabled: false, + perfDir: dir, + identityProvider: { snapshot: () => fixtureIdentity() }, + ...overrides, + }; +} + +/** + * Scheduler that counts timer clear() calls so a test can prove disposal + * actually cancels the retention maintenance interval. + */ +class CountingScheduler implements PerfScheduler { + clearCount = 0; + setInterval(_callback: () => Promise, _ms: number) { + return { + unref() {}, + // Arrow captures the lexical `this` (the instance) without aliasing it. + clear: () => { + this.clearCount += 1; + }, + }; + } +} + +beforeEach(() => { + dir = fs.mkdtempSync(join(tmpdir(), 'perf-lifecycle-')); + setInteractiveStdoutObserver(null); + setInteractiveRenderObserver(null); + setPerfPhaseObserver(null); +}); + +afterEach(async () => { + // Clear any tracked instance/owner left behind by a test without invoking + // production cleanup (tests dispose real owners explicitly). This is the + // test-only reset path, not a production code path. + __setTrackedInstanceAndOwnerForTesting(undefined, null); + __resetInteractiveUIStateForTesting(); + setInteractiveStdoutObserver(null); + setInteractiveRenderObserver(null); + setPerfPhaseObserver(null); + try { + await fsp.rm(dir, { recursive: true, force: true }); + } catch { + // ignore + } +}); + +// --------------------------------------------------------------------------- +// Item 1: Pre-start replacement via the ACTUAL replacePreviousInstanceAndOwner +// --------------------------------------------------------------------------- + +describe('replacePreviousInstanceAndOwner — owner A then owner B (Item 1)', () => { + it('A observers/claim/timer are gone before B installs (no observer conflict)', async () => { + const schedulerA = new CountingScheduler(); + const ownerA = createInteractivePerfRuntime( + makeOptions({ + runUuid: '00000000-0000-4000-8000-000000000000', + __schedulerForTesting: schedulerA, + }), + ); + await ownerA!.start(); + + // A installed observers, created a claim, and started the maintenance timer. + expect(getInteractiveStdoutObserver()).toBe(ownerA!.registry); + expect(getInteractiveRenderObserver()).toBe(ownerA!.registry); + expect(getPerfPhaseObserver()).toBe(ownerA!.registry); + const filesAfterA = fs.readdirSync(dir); + expect( + filesAfterA.some((f) => + f.includes('00000000-0000-4000-8000-000000000000.claim'), + ), + ).toBe(true); + + // Track A as the latest owner, then invoke the ACTUAL pre-start + // replacement (the same routine startInteractiveUI calls before building + // a new owner). + __setTrackedInstanceAndOwnerForTesting(undefined, ownerA); + await replacePreviousInstanceAndOwner(); + + // A's observers, claim, and timer are gone. + expect(getInteractiveStdoutObserver()).toBe(null); + expect(getInteractiveRenderObserver()).toBe(null); + expect(getPerfPhaseObserver()).toBe(null); + expect(schedulerA.clearCount).toBeGreaterThanOrEqual(1); + const filesAfterADispose = fs.readdirSync(dir); + expect( + filesAfterADispose.some((f) => + f.includes('00000000-0000-4000-8000-000000000000.claim'), + ), + ).toBe(false); + + // B starts without observer conflict. + const ownerB = createInteractivePerfRuntime( + makeOptions({ runUuid: '00000000-0000-4000-8000-000000000001' }), + ); + await ownerB!.start(); + try { + expect(getInteractiveStdoutObserver()).toBe(ownerB!.registry); + expect(getInteractiveRenderObserver()).toBe(ownerB!.registry); + expect(getPerfPhaseObserver()).toBe(ownerB!.registry); + + // B's claim exists, A's claim does not. + const filesAfterB = fs.readdirSync(dir); + expect( + filesAfterB.some((f) => + f.includes('00000000-0000-4000-8000-000000000001.claim'), + ), + ).toBe(true); + expect( + filesAfterB.some((f) => + f.includes('00000000-0000-4000-8000-000000000000.claim'), + ), + ).toBe(false); + } finally { + await ownerB!.dispose(); + } + }); + + it('A → B sequential: replacement clears A, then B owns and disposes cleanly', async () => { + const ownerA = createInteractivePerfRuntime( + makeOptions({ runUuid: '00000000-0000-4000-8000-000000000002' }), + ); + await ownerA!.start(); + expect(getInteractiveStdoutObserver()).toBe(ownerA!.registry); + + // ACTUAL replacement of A. + __setTrackedInstanceAndOwnerForTesting(undefined, ownerA); + await replacePreviousInstanceAndOwner(); + expect(getInteractiveStdoutObserver()).toBe(null); + + // Start B. + const ownerB = createInteractivePerfRuntime( + makeOptions({ runUuid: '00000000-0000-4000-8000-000000000003' }), + ); + await ownerB!.start(); + try { + expect(getInteractiveStdoutObserver()).toBe(ownerB!.registry); + + // Dispose B directly — B's observers cleared, no leak. + await ownerB!.dispose(); + expect(getInteractiveStdoutObserver()).toBe(null); + expect(getInteractiveRenderObserver()).toBe(null); + expect(getPerfPhaseObserver()).toBe(null); + + // No claims left. + const claims = fs.readdirSync(dir).filter((f) => f.endsWith('.claim')); + expect(claims).toHaveLength(0); + } finally { + await ownerB!.dispose(); + } + }); +}); + +// --------------------------------------------------------------------------- +// Item 2: A throwing clear does not skip actual unmount or real owner dispose +// --------------------------------------------------------------------------- + +describe('actual cleanup helper — clear failure does not skip unmount/dispose (Item 2)', () => { + it('a throwing clear does not skip actual unmount or real owner dispose', async () => { + const owner = createInteractivePerfRuntime( + makeOptions({ runUuid: '00000000-0000-4000-8000-000000000004' }), + ); + await owner!.start(); + + let unmountRan = false; + const throwingClearInstance = { + clear() { + throw new Error('clear failed'); + }, + unmount() { + unmountRan = true; + }, + }; + + // Track both, then invoke the ACTUAL replacement: clear/unmount/dispose run + // through the shared cleanupInstanceAndOwner. + __setTrackedInstanceAndOwnerForTesting(throwingClearInstance, owner); + await expect(replacePreviousInstanceAndOwner()).rejects.toThrow( + 'clear failed', + ); + + // clear threw, but unmount ran and the real owner disposed (claim gone, + // observers cleared). + expect(unmountRan).toBe(true); + const files = fs.readdirSync(dir); + expect(files.some((f) => f.endsWith('.claim'))).toBe(false); + expect(getInteractiveStdoutObserver()).toBe(null); + }); + + it('cleanupInstanceAndOwner runs clear, unmount, dispose in order and aggregates errors', async () => { + const order: string[] = []; + const instance = { + clear() { + order.push('clear'); + throw new Error('clear-err'); + }, + unmount() { + order.push('unmount'); + throw new Error('unmount-err'); + }, + }; + const owner = { + async dispose() { + order.push('dispose'); + throw new Error('dispose-err'); + }, + }; + + let caught: unknown; + try { + await cleanupInstanceAndOwner(instance, owner); + } catch (err) { + caught = err; + } + + // All three steps ran in order despite each throwing. + expect(order).toEqual(['clear', 'unmount', 'dispose']); + expect(caught).toBeInstanceOf(AggregateError); + const messages = (caught as AggregateError).errors.map( + (e) => (e as Error).message, + ); + expect(messages).toEqual(['clear-err', 'unmount-err', 'dispose-err']); + }); +}); + +// --------------------------------------------------------------------------- +// Item 3: ACTUAL rollbackInteractiveFailure — aggregates errors, every +// callback runs; real owner disposal during rollback. +// --------------------------------------------------------------------------- + +describe('rollbackInteractiveFailure — aggregate errors, every callback runs (Item 3)', () => { + it('aggregates render + owner + mouse + restore errors while every callback runs', async () => { + const calls: string[] = []; + const renderErr = new Error('render failed'); + + let caught: unknown; + try { + await rollbackInteractiveFailure(renderErr, { + instance: undefined, + owner: { + async dispose() { + calls.push('owner'); + throw new Error('owner failed'); + }, + }, + mouse: { + disable() { + calls.push('mouse-disable'); + throw new Error('mouse failed'); + }, + removeListener() { + calls.push('mouse-remove'); + }, + }, + restore: { + restore() { + calls.push('restore'); + throw new Error('restore failed'); + }, + removeListener() { + calls.push('restore-remove'); + }, + }, + }); + } catch (err) { + caught = err; + } + + // The primary render error is preserved first. + expect(caught).toBeInstanceOf(AggregateError); + const agg = caught as AggregateError; + expect(agg.errors[0]).toBe(renderErr); + + // Every cleanup error is present. + const messages = agg.errors.map((e) => (e as Error).message); + expect(messages).toContain('owner failed'); + expect(messages).toContain('mouse failed'); + expect(messages).toContain('restore failed'); + + // Every callback ran (including the non-throwing listener removals). + expect(calls).toEqual([ + 'owner', + 'mouse-disable', + 'mouse-remove', + 'restore', + 'restore-remove', + ]); + }); + + it('rethrows the primary error unchanged when no cleanup step fails', async () => { + const renderErr = new Error('render failed'); + const calls: string[] = []; + let caught: unknown; + try { + await rollbackInteractiveFailure(renderErr, { + instance: undefined, + owner: null, + mouse: null, + restore: { + restore() { + calls.push('restore'); + }, + removeListener() { + calls.push('restore-remove'); + }, + }, + }); + } catch (err) { + caught = err; + } + expect(caught).toBe(renderErr); + expect(calls).toEqual(['restore', 'restore-remove']); + }); + + it('real owner disposal runs during rollback (claim removed)', async () => { + const owner = createInteractivePerfRuntime( + makeOptions({ runUuid: '00000000-0000-4000-8000-000000000005' }), + ); + await owner!.start(); + + try { + // Claim exists before rollback. + const filesBefore = fs.readdirSync(dir); + expect(filesBefore.some((f) => f.includes('.claim'))).toBe(true); + + const renderErr = new Error('render boom'); + let caught: unknown; + try { + await rollbackInteractiveFailure(renderErr, { + instance: undefined, + owner, + mouse: null, + restore: { + restore() {}, + removeListener() {}, + }, + }); + } catch (err) { + caught = err; + } + + // Primary render error rethrown (no cleanup step failed). + expect(caught).toBe(renderErr); + + // Real owner disposed during rollback: claim removed, observers cleared. + const filesAfter = fs.readdirSync(dir); + expect(filesAfter.some((f) => f.includes('.claim'))).toBe(false); + expect(getInteractiveStdoutObserver()).toBe(null); + } finally { + // Idempotent: no-op if rollback already disposed the owner. + await owner!.dispose(); + } + }); + + it('setup-failure path: clear/unmount run on the rendered instance', async () => { + // When render succeeds but setup fails, the transactional catch passes the + // rendered instance to rollbackInteractiveFailure, so clear/unmount run. + const calls: string[] = []; + const setupErr = new Error('setup failed'); + const instance = { + clear() { + calls.push('clear'); + }, + unmount() { + calls.push('unmount'); + }, + }; + let caught: unknown; + try { + await rollbackInteractiveFailure(setupErr, { + instance, + owner: null, + mouse: null, + restore: { + restore() { + calls.push('restore'); + }, + removeListener() { + calls.push('restore-remove'); + }, + }, + }); + } catch (err) { + caught = err; + } + expect(caught).toBe(setupErr); + // The rendered instance was torn down (clear/unmount) plus restore ran. + expect(calls).toEqual(['clear', 'unmount', 'restore', 'restore-remove']); + }); +}); + +// --------------------------------------------------------------------------- +// Item 4: Exactly-once disposal — capture+clear before dispose guarantees +// registered cleanup + replacement never double-dispose. +// --------------------------------------------------------------------------- + +describe('exactly-once disposal — capture+clear before dispose (Item 4)', () => { + it('replacement then a second replacement does not dispose twice', async () => { + let clearCount = 0; + let unmountCount = 0; + let disposeCount = 0; + const instance = { + clear() { + clearCount++; + }, + unmount() { + unmountCount++; + }, + }; + const owner = { + async dispose() { + disposeCount++; + }, + }; + + __setTrackedInstanceAndOwnerForTesting(instance, owner); + await replacePreviousInstanceAndOwner(); + + // Simulate the registered global cleanup running AFTER the replacement + // already captured+cleared the refs. It must find empty slots (no-op). + await replacePreviousInstanceAndOwner(); + + expect(clearCount).toBe(1); + expect(unmountCount).toBe(1); + expect(disposeCount).toBe(1); + }); + + it('cleanup that throws still clears refs so a second call is a no-op', async () => { + let clearCount = 0; + let unmountCount = 0; + let disposeCount = 0; + const instance = { + clear() { + clearCount++; + throw new Error('clear boom'); + }, + unmount() { + unmountCount++; + }, + }; + const owner = { + async dispose() { + disposeCount++; + }, + }; + + __setTrackedInstanceAndOwnerForTesting(instance, owner); + + // The first replacement captures+clears refs BEFORE calling + // cleanupInstanceAndOwner, so even though clear throws the refs are gone. + await expect(replacePreviousInstanceAndOwner()).rejects.toThrow( + 'clear boom', + ); + + // Second call finds empty slots — no double dispose despite the throw. + await replacePreviousInstanceAndOwner(); + + expect(clearCount).toBe(1); + expect(unmountCount).toBe(1); + expect(disposeCount).toBe(1); + }); + + it('replacement clears owner before dispose throws so second call skips it', async () => { + let disposeCount = 0; + const owner = { + async dispose() { + disposeCount++; + throw new Error('dispose boom'); + }, + }; + + __setTrackedInstanceAndOwnerForTesting(undefined, owner); + + await expect(replacePreviousInstanceAndOwner()).rejects.toThrow( + 'dispose boom', + ); + + // Refs already cleared despite the throw — second call is a no-op. + await replacePreviousInstanceAndOwner(); + + expect(disposeCount).toBe(1); + }); +}); diff --git a/packages/cli/src/ui/hooks/perf/interactivePerfRuntime.behavior.test.ts b/packages/cli/src/ui/hooks/perf/interactivePerfRuntime.behavior.test.ts new file mode 100644 index 0000000000..f8e9522287 --- /dev/null +++ b/packages/cli/src/ui/hooks/perf/interactivePerfRuntime.behavior.test.ts @@ -0,0 +1,385 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * P12 behavioral tests for the interactive perf runtime owner/factory + * (EVIDENCE-AC1, AC2, AC12 — integration spine). + * + * Real PerfSink + real PerfRetention + real OperationLifecycleRegistry + real + * observers + real filesystem. No mock theatre — the factory constructs the + * actual integrated pipeline. Asserts stable outputs, not mock calls. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import { promises as fsp } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { + createInteractivePerfRuntime, + resolveRuntimeVersion, + type InteractivePerfRuntimeOptions, +} from './interactivePerfRuntime.js'; +import type { OperationIdentitySnapshot } from '../agentStream/operationLifecycle.js'; +import { + setInteractiveStdoutObserver, + setInteractiveRenderObserver, + getInteractiveStdoutObserver, + getInteractiveRenderObserver, +} from '../../inkRenderOptions.js'; +import { + getPerfPhaseObserver, + setPerfPhaseObserver, +} from '@vybestack/llxprt-code-telemetry/perf/perfPhaseObserver.js'; +import { + readPerfRecords, + type PerfOperationRecord, +} from '@vybestack/llxprt-code-telemetry/perf/perfRecords.js'; + +let dir: string; + +function fixtureIdentity( + overrides: Partial = {}, +): OperationIdentitySnapshot { + return { + session_id: 'sess-test', + runtime_id: 'rt-test', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'hash-test', + llxprt_version: '0.11.0', + git_sha: 'abc1234', + runtime: 'bun-1.3.14', + platform: 'darwin', + provider: 'test-provider', + model: 'test-model', + terminal_cols: 80, + terminal_rows: 24, + render_mode: 'incremental', + ...overrides, + }; +} + +function identityProvider(snap: OperationIdentitySnapshot) { + return { snapshot: () => snap }; +} + +function makeOptions( + overrides: Partial = {}, +): InteractivePerfRuntimeOptions & { perfDir: string } { + return { + enabled: true, + memoryEnabled: false, + perfDir: dir, + identityProvider: identityProvider(fixtureIdentity()), + ...overrides, + }; +} + +beforeEach(() => { + dir = fs.mkdtempSync(join(tmpdir(), 'perf-owner-')); + // Reset global observers to a clean state before each test. + setInteractiveStdoutObserver(null); + setInteractiveRenderObserver(null); + setPerfPhaseObserver(null); +}); + +afterEach(async () => { + // Clean up any global observers left behind. + setInteractiveStdoutObserver(null); + setInteractiveRenderObserver(null); + setPerfPhaseObserver(null); + try { + await fsp.rm(dir, { recursive: true, force: true }); + } catch { + // ignore + } +}); + +describe('interactivePerfRuntime — disabled mode (AC-2)', () => { + it('returns null when disabled', () => { + const runtime = createInteractivePerfRuntime( + makeOptions({ enabled: false }), + ); + expect(runtime).toBe(null); + }); + + it('creates no perf directory, file, or claim', async () => { + const perfDir = join(dir, 'perf'); + createInteractivePerfRuntime(makeOptions({ enabled: false, perfDir })); + // The perf directory must not exist (no directory creation side effect). + expect(fs.existsSync(perfDir)).toBe(false); + }); + + it('installs no observer and allocates no ring/controller', () => { + createInteractivePerfRuntime(makeOptions({ enabled: false })); + expect(getInteractiveStdoutObserver()).toBe(null); + expect(getInteractiveRenderObserver()).toBe(null); + expect(getPerfPhaseObserver()).toBe(null); + }); +}); + +describe('interactivePerfRuntime — enabled mode constructs full pipeline (AC-1)', () => { + it('constructs a non-null runtime with registry and snapshot capability', () => { + const runtime = createInteractivePerfRuntime(makeOptions()); + expect(runtime).not.toBe(null); + expect(runtime!.registry).toBeDefined(); + expect(runtime!.snapshotCapability).toBeDefined(); + }); + + it('start() creates the claim file and installs observers', async () => { + const runtime = createInteractivePerfRuntime(makeOptions()); + await runtime!.start(); + + try { + // Claim file exists in the perf dir. + const files = fs.readdirSync(dir); + expect(files.some((f) => f.endsWith('.claim'))).toBe(true); + + // Observers are installed. + expect(getInteractiveStdoutObserver()).not.toBe(null); + expect(getInteractiveRenderObserver()).not.toBe(null); + expect(getPerfPhaseObserver()).not.toBe(null); + } finally { + await runtime!.dispose(); + } + }); + + it('enabled with memory constructs a memory controller', () => { + const runtime = createInteractivePerfRuntime( + makeOptions({ memoryEnabled: true }), + ); + expect(runtime!.memoryController).not.toBe(null); + }); + + it('enabled without memory has null memory controller', () => { + const runtime = createInteractivePerfRuntime( + makeOptions({ memoryEnabled: false }), + ); + expect(runtime!.memoryController).toBe(null); + }); +}); + +describe('interactivePerfRuntime — operation recording (AC-1)', () => { + it('produces exactly one valid operation record per operation', async () => { + const runtime = createInteractivePerfRuntime(makeOptions()); + await runtime!.start(); + + const ac = new AbortController(); + runtime!.registry.begin(ac.signal, 'sess#agentic-loop#uuid-1'); + await runtime!.registry.finalise(ac.signal, 'completed'); + + await runtime!.dispose(); + + const files = fs.readdirSync(dir).filter((f) => f.endsWith('.jsonl')); + expect(files.length).toBe(1); + const result = await readPerfRecords(join(dir, files[0])); + const ops = result.records.filter((r) => r.record_type === 'operation'); + expect(ops).toHaveLength(1); + expect(ops[0].status).toBe('completed'); + expect(ops[0].operation_id).toBe('sess#agentic-loop#uuid-1'); + }); + + it('produces separate records for separate operations', async () => { + const runtime = createInteractivePerfRuntime(makeOptions()); + await runtime!.start(); + + const ac1 = new AbortController(); + runtime!.registry.begin(ac1.signal, 'sess#agentic-loop#uuid-1'); + await runtime!.registry.finalise(ac1.signal, 'completed'); + + const ac2 = new AbortController(); + runtime!.registry.begin(ac2.signal, 'sess#agentic-loop#uuid-2'); + await runtime!.registry.finalise(ac2.signal, 'completed'); + + await runtime!.dispose(); + + const files = fs.readdirSync(dir).filter((f) => f.endsWith('.jsonl')); + expect(files.length).toBe(1); + const result = await readPerfRecords(join(dir, files[0])); + const ops = result.records.filter((r) => r.record_type === 'operation'); + expect(ops).toHaveLength(2); + }); +}); + +describe('interactivePerfRuntime — live snapshot (AC-12)', () => { + it('snapshot capability returns active operation during an active operation', async () => { + let mono = 1000; + const runtime = createInteractivePerfRuntime( + makeOptions({ + identityProvider: identityProvider( + fixtureIdentity({ provider: 'openai', model: 'gpt-4o' }), + ), + monotonicNow: () => mono, + }), + ); + await runtime!.start(); + + const ac = new AbortController(); + runtime!.registry.begin(ac.signal, 'sess#agentic-loop#uuid-1'); + + mono = 2500; + const activeOp = runtime!.snapshotCapability.getActiveOperationSummary(); + expect(activeOp).not.toBe(null); + expect(activeOp!.provider).toBe('openai'); + expect(activeOp!.model).toBe('gpt-4o'); + expect(activeOp!.elapsedMs).toBe(1500); + + await runtime!.registry.finalise(ac.signal, 'completed'); + await runtime!.dispose(); + }); + + it('snapshot capability returns null active operation when idle', async () => { + const runtime = createInteractivePerfRuntime(makeOptions()); + await runtime!.start(); + expect(runtime!.snapshotCapability.getActiveOperationSummary()).toBe(null); + await runtime!.dispose(); + }); + + it('snapshot capability returns null memory samples when memory disabled', async () => { + const runtime = createInteractivePerfRuntime( + makeOptions({ memoryEnabled: false }), + ); + await runtime!.start(); + expect(runtime!.snapshotCapability.getMemorySnapshot()).toBe(null); + await runtime!.dispose(); + }); + + it('snapshot capability returns memory samples when memory enabled', async () => { + const runtime = createInteractivePerfRuntime( + makeOptions({ memoryEnabled: true }), + ); + await runtime!.start(); + // No samples yet — ring is empty. + expect(runtime!.snapshotCapability.getMemorySnapshot()).toEqual([]); + await runtime!.dispose(); + }); +}); + +describe('interactivePerfRuntime — clean disposal (AC-1)', () => { + it('dispose removes the claim file', async () => { + const runtime = createInteractivePerfRuntime(makeOptions()); + await runtime!.start(); + + const claimsBefore = fs + .readdirSync(dir) + .filter((f) => f.endsWith('.claim')); + expect(claimsBefore.length).toBe(1); + + await runtime!.dispose(); + + const claimsAfter = fs.readdirSync(dir).filter((f) => f.endsWith('.claim')); + expect(claimsAfter.length).toBe(0); + }); + + it('dispose clears observers', async () => { + const runtime = createInteractivePerfRuntime(makeOptions()); + await runtime!.start(); + expect(getInteractiveStdoutObserver()).not.toBe(null); + + await runtime!.dispose(); + expect(getInteractiveStdoutObserver()).toBe(null); + expect(getInteractiveRenderObserver()).toBe(null); + expect(getPerfPhaseObserver()).toBe(null); + }); + + it('dispose preserves parseable data on disk', async () => { + const runtime = createInteractivePerfRuntime(makeOptions()); + await runtime!.start(); + + const ac = new AbortController(); + runtime!.registry.begin(ac.signal, 'sess#agentic-loop#uuid-1'); + await runtime!.registry.finalise(ac.signal, 'completed'); + + await runtime!.dispose(); + + const files = fs.readdirSync(dir).filter((f) => f.endsWith('.jsonl')); + expect(files.length).toBe(1); + const result = await readPerfRecords(join(dir, files[0])); + expect(result.counts.parsed).toBe(1); + expect(result.counts.malformed).toBe(0); + }); +}); + +describe('interactivePerfRuntime — observer exercise (AC-6)', () => { + it('stdout observer callback accumulates bytes for active operation', async () => { + const runtime = createInteractivePerfRuntime(makeOptions()); + await runtime!.start(); + + const ac = new AbortController(); + runtime!.registry.begin(ac.signal, 'sess#agentic-loop#uuid-1'); + + // Exercise the installed stdout observer directly through the seam. + const stdoutObserver = getInteractiveStdoutObserver(); + expect(stdoutObserver).not.toBe(null); + stdoutObserver!.onWrite(100, 0.5); + + await runtime!.registry.finalise(ac.signal, 'completed'); + await runtime!.dispose(); + + const files = fs.readdirSync(dir).filter((f) => f.endsWith('.jsonl')); + const result = await readPerfRecords(join(dir, files[0])); + const op = result.records.find( + (r): r is PerfOperationRecord => r.record_type === 'operation', + ); + expect(op).toBeDefined(); + expect(op?.stdout_bytes).toBe(100); + expect(op?.stdout_write_calls).toBe(1); + }); + + it('render observer callback accumulates render time for active operation', async () => { + const runtime = createInteractivePerfRuntime(makeOptions()); + await runtime!.start(); + + const ac = new AbortController(); + runtime!.registry.begin(ac.signal, 'sess#agentic-loop#uuid-1'); + + const renderObserver = getInteractiveRenderObserver(); + expect(renderObserver).not.toBe(null); + renderObserver!.onRender(3.5); + + await runtime!.registry.finalise(ac.signal, 'completed'); + await runtime!.dispose(); + + const files = fs.readdirSync(dir).filter((f) => f.endsWith('.jsonl')); + const result = await readPerfRecords(join(dir, files[0])); + const op = result.records.find( + (r): r is PerfOperationRecord => r.record_type === 'operation', + ); + expect(op).toBeDefined(); + expect(op?.ink_render_ms).toBe(3.5); + expect(op?.ink_render_count).toBe(1); + }); +}); + +describe('interactivePerfRuntime — default-off zero side effects (AC-2)', () => { + it('disabled produces zero lifecycle records', () => { + const runtime = createInteractivePerfRuntime( + makeOptions({ enabled: false }), + ); + expect(runtime).toBe(null); + // No records possible — no registry exists. + }); + + it('disabled path does not interact with performance.now or timers', () => { + // The disabled factory returns null before UUID/sink/retention/observer + // construction. There is nothing to dispose. + const runtime = createInteractivePerfRuntime( + makeOptions({ enabled: false }), + ); + expect(runtime).toBe(null); + }); +}); + +describe('resolveRuntimeVersion — canonical format (no leading v)', () => { + it('produces a runtime string without a leading v on the version segment', () => { + const value = resolveRuntimeVersion(); + expect(value.startsWith('bun-') || value.startsWith('node-')).toBe(true); + const segment = value.split('-').slice(1).join('-'); + expect(segment.length).toBeGreaterThan(0); + expect(segment.startsWith('v')).toBe(false); + }); +}); diff --git a/packages/cli/src/ui/hooks/perf/interactivePerfRuntime.startup.behavior.test.ts b/packages/cli/src/ui/hooks/perf/interactivePerfRuntime.startup.behavior.test.ts new file mode 100644 index 0000000000..5e8330f96c --- /dev/null +++ b/packages/cli/src/ui/hooks/perf/interactivePerfRuntime.startup.behavior.test.ts @@ -0,0 +1,345 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * P12 behavioral tests for InteractivePerfRuntime.start startup transaction + * rollback (Item 2). Proves deterministic rollback on sink.start failure and + * observer-install conflict, with no leaked artifacts/observers/timers. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import { promises as fsp } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { + createInteractivePerfRuntime, + type InteractivePerfRuntimeOptions, +} from './interactivePerfRuntime.js'; +import type { OperationIdentitySnapshot } from '../agentStream/operationLifecycle.js'; +import { + setInteractiveStdoutObserver, + setInteractiveRenderObserver, + getInteractiveStdoutObserver, + getInteractiveRenderObserver, +} from '../../inkRenderOptions.js'; +import { + getPerfPhaseObserver, + setPerfPhaseObserver, +} from '@vybestack/llxprt-code-telemetry/perf/perfPhaseObserver.js'; +import type { + PerfRetentionFilesystem, + PerfScheduler, + PerfTimerHandle, +} from '@vybestack/llxprt-code-telemetry/perf/index.js'; + +let dir: string; + +function fixtureIdentity(): OperationIdentitySnapshot { + return { + session_id: 'sess-startup', + runtime_id: 'rt-startup', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'hash-startup', + llxprt_version: '0.11.0', + git_sha: 'abc1234', + runtime: 'bun-1.3.14', + platform: 'darwin-arm64', + provider: 'test-provider', + model: 'test-model', + terminal_cols: 80, + terminal_rows: 24, + render_mode: 'incremental', + }; +} + +function makeOptions( + overrides: Partial = {}, +): InteractivePerfRuntimeOptions & { perfDir: string } { + return { + enabled: true, + memoryEnabled: false, + perfDir: dir, + identityProvider: { snapshot: () => fixtureIdentity() }, + ...overrides, + }; +} + +beforeEach(() => { + dir = fs.mkdtempSync(join(tmpdir(), 'perf-startup-')); + setInteractiveStdoutObserver(null); + setInteractiveRenderObserver(null); + setPerfPhaseObserver(null); +}); + +afterEach(async () => { + setInteractiveStdoutObserver(null); + setInteractiveRenderObserver(null); + setPerfPhaseObserver(null); + try { + await fsp.rm(dir, { recursive: true, force: true }); + } catch { + // ignore + } +}); + +/** + * Filesystem port whose ensureDir throws a non-errno (internal) error so + * retention.start() fails fast (rethrows non-errno) rather than fail-opening. + * The claim is never created (ensureDir fails before openExclusive), and the + * timer is never started, proving no leaked artifact/observer/timer. + */ +class InternalErrorRetentionFs implements PerfRetentionFilesystem { + async ensureDir(): Promise { + throw new Error('internal retention start corruption'); + } + async openExclusive(): Promise { + throw new Error('should not reach openExclusive'); + } + async utimes(): Promise { + throw new Error('should not reach utimes'); + } + async readdir(): Promise { + throw new Error('should not reach readdir'); + } + async stat(): Promise<{ size: number; mtimeMs: number }> { + throw new Error('should not reach stat'); + } + async unlink(): Promise { + throw new Error('should not reach unlink'); + } +} + +describe('InteractivePerfRuntime.start — startup rollback (Item 2)', () => { + it('rolls back and rejects on sink.start failure (non-errno internal error)', async () => { + const runtime = createInteractivePerfRuntime( + makeOptions({ + __retentionFsForTesting: new InternalErrorRetentionFs(), + }), + ); + expect(runtime).not.toBe(null); + + let startError: unknown = null; + try { + await runtime!.start(); + } catch (err) { + startError = err; + } + expect(startError).not.toBe(null); + // The original startup error is surfaced. When rollback also encounters + // cleanup errors, an AggregateError is used; when only the startup error + // occurred, the plain Error is surfaced directly. Either way the original + // message is present. + const err = startError as Error; + const messages = + err instanceof AggregateError + ? (err.errors as unknown[]).map((e) => (e as Error).message) + : [err.message]; + expect(messages.some((m) => m.includes('internal retention start'))).toBe( + true, + ); + + // No observers installed. + expect(getInteractiveStdoutObserver()).toBe(null); + expect(getInteractiveRenderObserver()).toBe(null); + expect(getPerfPhaseObserver()).toBe(null); + + // No claim file or perf jsonl leaked. + const files = fs.readdirSync(dir); + expect(files.some((f) => f.endsWith('.claim'))).toBe(false); + expect(files.some((f) => f.endsWith('.jsonl'))).toBe(false); + + await expect(runtime!.dispose()).resolves.toBeUndefined(); + await expect(runtime!.dispose()).resolves.toBeUndefined(); + }); + + it('rolls back and rejects on observer-install conflict (owner B)', async () => { + // Start owner A first (installs observers owned by A's registry). + const ownerA = createInteractivePerfRuntime(makeOptions()); + await ownerA!.start(); + expect(getInteractiveStdoutObserver()).not.toBe(null); + expect(getInteractiveRenderObserver()).not.toBe(null); + expect(getPerfPhaseObserver()).not.toBe(null); + + const filesAfterA = fs.readdirSync(dir); + expect(filesAfterA.some((f) => f.endsWith('.claim'))).toBe(true); + + // Owner B: same perfDir, a different runUuid so its claim won't conflict. + // B starts its sink (creates its own claim + timer), then conflicts on + // observer ownership because A's observers are still installed. + const ownerB = createInteractivePerfRuntime( + makeOptions({ runUuid: '00000000-0000-4000-8000-000000000000' }), + ); + expect(ownerB).not.toBe(null); + + let bStartError: unknown = null; + try { + await ownerB!.start(); + } catch (err) { + bStartError = err; + } + expect(bStartError).not.toBe(null); + // The observer-conflict error is surfaced. When rollback also encounters + // cleanup errors, an AggregateError is used; when only the startup error + // occurred (B's claim cleanup is fail-open errno or succeeds), the plain + // Error is surfaced directly. Either way the original message is present. + const bErr = bStartError as Error; + const bMessages = + bErr instanceof AggregateError + ? (bErr.errors as unknown[]).map((e) => (e as Error).message) + : [bErr.message]; + expect(bMessages.some((m) => m.includes('observer is already'))).toBe(true); + + // B's claim must be removed (no leaked claim from B). + const filesAfterB = fs.readdirSync(dir); + const claimsAfterB = filesAfterB.filter((f) => f.endsWith('.claim')); + expect(claimsAfterB.length).toBe(1); + // The surviving claim is A's, not B's. + expect( + claimsAfterB.some((f) => + f.includes('00000000-0000-4000-8000-000000000000'), + ), + ).toBe(false); + + // A's observers remain owned (not clobbered by B's failed install). + expect(getInteractiveStdoutObserver()).toBe(ownerA!.registry); + expect(getInteractiveRenderObserver()).toBe(ownerA!.registry); + expect(getPerfPhaseObserver()).toBe(ownerA!.registry); + + // Clean up A. + await ownerA!.dispose(); + expect(getInteractiveStdoutObserver()).toBe(null); + expect(getInteractiveRenderObserver()).toBe(null); + expect(getPerfPhaseObserver()).toBe(null); + + await expect(ownerB!.dispose()).resolves.toBeUndefined(); + await expect(ownerB!.dispose()).resolves.toBeUndefined(); + + // No B claim leaked. + const finalFiles = fs.readdirSync(dir); + expect( + finalFiles.some((f) => + f.includes('00000000-0000-4000-8000-000000000000'), + ), + ).toBe(false); + }); +}); + +// --------------------------------------------------------------------------- +// Item 4: Timer cleanup evidence via counting scheduler +// --------------------------------------------------------------------------- + +/** + * Counting scheduler that records setInterval and clear calls so tests can + * PROVE timer cleanup rather than inferring from claim absence. + */ +class CountingScheduler implements PerfScheduler { + setIntervalCount = 0; + clearCount = 0; + callback: (() => Promise) | null = null; + + setInterval(callback: () => Promise, _ms: number): PerfTimerHandle { + this.setIntervalCount++; + this.callback = callback; + return { + unref: () => {}, + clear: () => { + this.clearCount++; + }, + }; + } +} + +describe('InteractivePerfRuntime — timer cleanup evidence via counting scheduler (Item 4)', () => { + it('dispose() clears the timer (clearCount increments)', async () => { + const scheduler = new CountingScheduler(); + const runtime = createInteractivePerfRuntime( + makeOptions({ + runUuid: '00000000-0000-4000-8000-000000000001', + __schedulerForTesting: scheduler, + }), + ); + await runtime!.start(); + + // Timer was started. + expect(scheduler.setIntervalCount).toBe(1); + expect(scheduler.clearCount).toBe(0); + + await runtime!.dispose(); + + // Timer was cleared on dispose. + expect(scheduler.clearCount).toBe(1); + }); + + it('startup rollback clears the timer when retention.start fails', async () => { + const scheduler = new CountingScheduler(); + const runtime = createInteractivePerfRuntime( + makeOptions({ + runUuid: '00000000-0000-4000-8000-000000000002', + __schedulerForTesting: scheduler, + __retentionFsForTesting: new InternalErrorRetentionFs(), + }), + ); + expect(runtime).not.toBe(null); + + // start() should reject because retention.start fails. + let startError: unknown = null; + try { + await runtime!.start(); + } catch (err) { + startError = err; + } + expect(startError).not.toBe(null); + + // Timer was never started (retention.start fails before scheduler). + expect(scheduler.setIntervalCount).toBe(0); + expect(scheduler.clearCount).toBe(0); + + // dispose is a no-op. + await runtime!.dispose(); + expect(scheduler.clearCount).toBe(0); + }); + + it('owner A/B: A timer cleared on A dispose, B timer cleared on B dispose', async () => { + const schedulerA = new CountingScheduler(); + const schedulerB = new CountingScheduler(); + + // Start owner A with its own scheduler. + const ownerA = createInteractivePerfRuntime( + makeOptions({ + runUuid: '00000000-0000-4000-8000-000000000003', + __schedulerForTesting: schedulerA, + }), + ); + await ownerA!.start(); + expect(schedulerA.setIntervalCount).toBe(1); + expect(schedulerA.clearCount).toBe(0); + + // Dispose A — timer cleared. + await ownerA!.dispose(); + expect(schedulerA.clearCount).toBe(1); + + // Start owner B with its own scheduler (A is already gone). + const ownerB = createInteractivePerfRuntime( + makeOptions({ + runUuid: '00000000-0000-4000-8000-000000000004', + __schedulerForTesting: schedulerB, + }), + ); + await ownerB!.start(); + expect(schedulerB.setIntervalCount).toBe(1); + expect(schedulerB.clearCount).toBe(0); + + // Dispose B — timer cleared. + await ownerB!.dispose(); + expect(schedulerB.clearCount).toBe(1); + + // A's timer was cleared once, B's timer was cleared once. + expect(schedulerA.clearCount).toBe(1); + expect(schedulerB.clearCount).toBe(1); + }); +}); diff --git a/packages/cli/src/ui/hooks/perf/interactivePerfRuntime.ts b/packages/cli/src/ui/hooks/perf/interactivePerfRuntime.ts new file mode 100644 index 0000000000..6610d18f70 --- /dev/null +++ b/packages/cli/src/ui/hooks/perf/interactivePerfRuntime.ts @@ -0,0 +1,500 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Interactive perf runtime owner/factory (P12, issue #3167). + * + * Constructible at the CLI composition boundary. Disabled mode returns null + * BEFORE any construction (no UUID, no directory creation, no sink/retention/ + * claim/registry/ring/controller/observer/performance.now/memoryUsage/timer). + * + * Enabled mode owns: run UUID, PerfRetention, PerfSink, + * OperationLifecycleRegistry, optional MemoryTelemetryController, stdout/render + * observer installation, live snapshot capability, and deterministic + * disposal/draining. + * + * Canonical production directory is join(Storage.getGlobalLogDir(), 'perf'). + * + * Not a global singleton: the composition root constructs one per rendered + * instance. Disposal order stops new observations and preserves accepted + * writes: identity-safe registry observer clear/dispose/drain, memory-controller + * drain, then sink/retention disposal. + */ + +import { randomUUID } from 'node:crypto'; +import { join } from 'node:path'; +import { Storage } from '@vybestack/llxprt-code-settings'; +import { + PerfSink, + PerfRetention, +} from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import type { + PerfSinkFilesystem, + PerfRetentionFilesystem, + PerfScheduler, +} from '@vybestack/llxprt-code-telemetry/perf/index.js'; +import type { + PerfSnapshotSample, + PerfSnapshotCapability, + PerfSelfHealth, +} from '../../commands/perfCommand.js'; +import { OperationLifecycleRegistry } from '../agentStream/operationLifecycle.js'; +import type { + OperationIdentityProvider, + OperationIdentitySnapshot, +} from '../agentStream/operationLifecycle.js'; +import { MemoryTelemetryController } from '../memoryTrend/memoryTelemetry.js'; + +// --------------------------------------------------------------------------- +// Identity provider factory +// --------------------------------------------------------------------------- + +/** + * Truly immutable fields captured exactly once after enablement. These do + * not change for the process lifetime (foreground parent/subagent are null). + */ +export interface InteractivePerfImmutableInputs { + readonly sessionId: string; + readonly runtimeId: string; + readonly projectHash: string; + readonly cliVersion: string; + readonly gitSha: string; + readonly runtime: string; + /** + * Platform with architecture, e.g. 'darwin-arm64'. Includes process.arch + * honestly rather than only process.platform. + */ + readonly platform: string; +} + +/** + * Getter-capable inputs for fields that may change between operations. + * provider/model/terminal geometry are read fresh at each registry.begin + * via these getters so persisted records show each operation's actual values. + */ +export interface InteractivePerfMutableInputs { + readonly provider: () => string; + readonly model: () => string; + readonly terminalCols: () => number; + readonly terminalRows: () => number; + readonly renderMode: () => string; +} + +/** + * Raw identity inputs collected from real runtime/config/build APIs at the + * composition boundary. Immutable fields are captured once; mutable fields + * are provided as getters so each operation snapshots current values. + */ +export interface InteractivePerfIdentityInputs { + readonly sessionId: string; + readonly runtimeId: string; + readonly projectHash: string; + readonly cliVersion: string; + readonly gitSha: string; + readonly runtime: string; + readonly platform: string; + readonly provider: string; + readonly model: string; + readonly terminalCols: number; + readonly terminalRows: number; + readonly renderMode: string; +} + +/** + * Creates an identity provider that snapshots CURRENT provider/model/terminal + * geometry at each call (registry.begin) rather than freezing startup values. + * Immutable fields (session/runtime/project/build) are fixed once after + * enablement. + */ +export function createIdentityProvider( + inputs: InteractivePerfIdentityInputs, +): OperationIdentityProvider { + return createIdentityProviderFromGetters( + { + sessionId: inputs.sessionId, + runtimeId: inputs.runtimeId, + projectHash: inputs.projectHash, + cliVersion: inputs.cliVersion, + gitSha: inputs.gitSha, + runtime: inputs.runtime, + platform: inputs.platform, + }, + { + provider: () => inputs.provider, + model: () => inputs.model, + terminalCols: () => inputs.terminalCols, + terminalRows: () => inputs.terminalRows, + renderMode: () => inputs.renderMode, + }, + ); +} + +/** + * Creates an identity provider from getter-capable mutable inputs. The + * immutable fields are fixed once; provider/model/terminal geometry are + * read fresh at each snapshot() call so persisted records reflect each + * operation's actual values. + */ +export function createIdentityProviderFromGetters( + immutable: InteractivePerfImmutableInputs, + mutable: InteractivePerfMutableInputs, +): OperationIdentityProvider { + return { + snapshot: (): OperationIdentitySnapshot => ({ + session_id: immutable.sessionId, + runtime_id: immutable.runtimeId, + parent_runtime_id: null, + subagent_name: null, + project_hash: immutable.projectHash, + llxprt_version: immutable.cliVersion, + git_sha: immutable.gitSha, + runtime: immutable.runtime, + platform: immutable.platform, + provider: mutable.provider(), + model: mutable.model(), + terminal_cols: mutable.terminalCols(), + terminal_rows: mutable.terminalRows(), + render_mode: mutable.renderMode(), + }), + }; +} + +/** + * Resolves the honest platform string: '-', + * e.g. 'darwin-arm64'. Includes architecture honestly rather than only + * process.platform. + */ +export function resolvePlatformArch(): string { + return `${process.platform}-${process.arch}`; +} + +/** + * Derives the exact render-mode value from actual config/settings. Matches + * the logic in inkRenderOptions.ts: + * - screen-reader → 'screen-reader' + * - alternateBuffer + incremental → 'incremental' + * - alternateBuffer (no incremental) → 'alt-buffer' + * - neither → 'plain' + */ +export function resolveRenderMode( + isScreenReader: boolean, + useAlternateBuffer: boolean, + incrementalRendering: boolean, +): string { + if (isScreenReader) return 'screen-reader'; + if (useAlternateBuffer) { + return incrementalRendering ? 'incremental' : 'alt-buffer'; + } + return 'plain'; +} + +/** + * Builds the runtime version string: 'bun-' when running under Bun, + * 'node-' otherwise. + */ +export function resolveRuntimeVersion(): string { + const bunGlobal = (globalThis as { Bun?: { version: string } }).Bun; + if (bunGlobal !== undefined) { + return `bun-${bunGlobal.version}`; + } + return `node-${process.version.replace(/^v/, '')}`; +} + +// --------------------------------------------------------------------------- +// Options +// --------------------------------------------------------------------------- + +export interface InteractivePerfRuntimeOptions { + /** Master enable. When false, the factory returns null with zero effects. */ + readonly enabled: boolean; + /** Memory sub-enable (requires enabled). When false, no memory controller. */ + readonly memoryEnabled: boolean; + /** Perf directory. Defaults to join(Storage.getGlobalLogDir(), 'perf'). */ + readonly perfDir?: string; + /** Identity provider (P12 constructs this from real runtime/config/build APIs). */ + readonly identityProvider: OperationIdentityProvider; + /** Test override for the run UUID. Defaults to crypto.randomUUID(). */ + readonly runUuid?: string; + /** Test override for monotonic clock. */ + readonly monotonicNow?: () => number; + /** Test override for wall clock. */ + readonly wallNow?: () => number; + /** Test override for memory sampler. */ + readonly memoryNow?: () => NodeJS.MemoryUsage; + /** + * Package-private test seam: a PerfSink filesystem port. When provided, + * buildRuntimeComponents injects it into the PerfSink so tests can + * deterministically fail sink.start() with a non-errno internal error. + */ + readonly __sinkFsForTesting?: PerfSinkFilesystem; + /** + * Package-private test seam: a PerfRetention filesystem port. When + * provided, buildRuntimeComponents injects it into the PerfRetention so + * tests can deterministically fail retention.start() (claim creation) with + * a non-errno internal error, proving startup rollback. + */ + readonly __retentionFsForTesting?: PerfRetentionFilesystem; + /** + * Package-private test seam: a PerfScheduler port. When provided, + * buildRuntimeComponents injects it into PerfRetention so tests can use a + * counting scheduler that proves timer.clear() is called on startup + * rollback and on dispose. + */ + readonly __schedulerForTesting?: PerfScheduler; +} + +// --------------------------------------------------------------------------- +// Runtime handle +// --------------------------------------------------------------------------- + +export interface InteractivePerfRuntime { + readonly registry: OperationLifecycleRegistry; + readonly memoryController: MemoryTelemetryController | null; + readonly snapshotCapability: PerfSnapshotCapability; + /** + * Starts sink/retention (creates claim, starts maintenance interval) and + * installs observers. Must be called and awaited BEFORE inkRenderOptions() + * is evaluated so the observer seams are live before the first render. + */ + start(): Promise; + /** + * Ordered disposal: registry observer clear/drain, memory-controller drain, + * then sink/retention disposal (drains writes + stops maintenance + removes + * claim). Surfaces internal failures rather than swallowing them. + */ + dispose(): Promise; +} + +// --------------------------------------------------------------------------- +// Factory +// --------------------------------------------------------------------------- + +/** + * Creates the interactive perf runtime. Returns null when disabled — BEFORE + * any UUID generation, directory creation, sink, retention, claim, registry, + * ring/controller, observer creation, performance.now, memoryUsage, or timer + * interaction. No side effects in the disabled path. + */ +export function createInteractivePerfRuntime( + options: InteractivePerfRuntimeOptions, +): InteractivePerfRuntime | null { + if (!options.enabled) { + return null; + } + + const { sink, memoryController, registry, snapshotCapability } = + buildRuntimeComponents(options); + + let started = false; + let disposed = false; + + const start = async (): Promise => { + if (started || disposed) return; + started = true; + try { + await sink.start(); + registry.installObservers(); + } catch (startupError) { + const rolledBack = await rollbackStartup( + startupError, + registry, + memoryController, + sink, + ); + disposed = true; + throw rolledBack; + } + }; + + const dispose = async (): Promise => { + if (disposed) return; + disposed = true; + const errors: unknown[] = []; + try { + await registry.dispose(); + } catch (err) { + errors.push(err); + } + if (memoryController !== null) { + try { + await memoryController.drain(); + } catch (err) { + errors.push(err); + } + } + try { + await sink.dispose(); + } catch (err) { + errors.push(err); + } + if (errors.length === 1) throw errors[0]; + if (errors.length > 1) { + throw new AggregateError(errors, 'interactive perf runtime disposal'); + } + }; + + return { + registry, + memoryController, + snapshotCapability, + start, + dispose, + }; +} + +/** + * Constructs the concrete runtime components (retention, sink, memory + * controller, registry, snapshot capability) from options. The enabled check + * is already done; this builds the real pipeline. Extracted to keep + * createInteractivePerfRuntime within the max-lines-per-function limit. + */ +function buildRuntimeComponents(options: InteractivePerfRuntimeOptions): { + sink: PerfSink; + retention: PerfRetention; + memoryController: MemoryTelemetryController | null; + registry: OperationLifecycleRegistry; + snapshotCapability: PerfSnapshotCapability; +} { + const perfDir = options.perfDir ?? getDefaultPerfDir(); + const runUuid = options.runUuid ?? randomUUID(); + const retention = new PerfRetention({ + dir: perfDir, + runUuid, + maintenanceIntervalMs: 60_000, + ...(options.__retentionFsForTesting !== undefined + ? { fs: options.__retentionFsForTesting } + : {}), + ...(options.__schedulerForTesting !== undefined + ? { scheduler: options.__schedulerForTesting } + : {}), + }); + const sink = new PerfSink({ + dir: perfDir, + runUuid, + retention, + ...(options.__sinkFsForTesting !== undefined + ? { fs: options.__sinkFsForTesting } + : {}), + }); + const memoryController = options.memoryEnabled + ? new MemoryTelemetryController({ + sink, + monotonicNow: options.monotonicNow, + wallNow: options.wallNow, + memoryNow: options.memoryNow, + }) + : null; + const registry = new OperationLifecycleRegistry({ + identityProvider: options.identityProvider, + sink, + retention, + monotonicNow: options.monotonicNow, + wallNow: options.wallNow, + memorySampler: memoryController ?? undefined, + }); + const snapshotCapability = createSnapshotCapability( + registry, + memoryController, + sink, + retention, + ); + return { sink, retention, memoryController, registry, snapshotCapability }; +} + +/** + * Rolls back whatever the start() owner created after a sink.start or + * observer-install failure: registry observers+drain, memory drain, then + * sink/retention disposal (timer + claim). Always runs all rollback steps; + * surfaces the original startup error and any internal cleanup errors via + * AggregateError. External errno cleanup remains fail-open inside sink/ + * retention. + */ +async function rollbackStartup( + startupError: unknown, + registry: OperationLifecycleRegistry, + memoryController: MemoryTelemetryController | null, + sink: PerfSink, +): Promise { + const cleanupErrors: unknown[] = []; + try { + await registry.dispose(); + } catch (err) { + cleanupErrors.push(err); + } + if (memoryController !== null) { + try { + await memoryController.drain(); + } catch (err) { + cleanupErrors.push(err); + } + } + try { + await sink.dispose(); + } catch (err) { + cleanupErrors.push(err); + } + if (cleanupErrors.length === 0) { + return startupError; + } + return new AggregateError( + [startupError, ...cleanupErrors], + 'interactive perf runtime startup rolled back', + ); +} + +// --------------------------------------------------------------------------- +// Canonical directory +// --------------------------------------------------------------------------- + +/** + * The canonical production perf directory: exactly + * join(Storage.getGlobalLogDir(), 'perf'). No other path is used. + */ +export function getDefaultPerfDir(): string { + return join(Storage.getGlobalLogDir(), 'perf'); +} + +// --------------------------------------------------------------------------- +// Snapshot capability adapter +// --------------------------------------------------------------------------- + +/** + * Snapshot capability adapter. Exposes read-only active-process self-health + * through the injected sink/retention capability: sink.lastWriteErrorCode + * (null = no error, string errno = last error) and retention.evictionCount + * (0 = none). These are known values — the report distinguishes them from + * undefined (unavailable) which occurs when no active runtime exists. + */ +function createSnapshotCapability( + registry: OperationLifecycleRegistry, + memoryController: MemoryTelemetryController | null, + sink: PerfSink, + retention: PerfRetention, +): PerfSnapshotCapability { + return { + getMemorySnapshot(): readonly PerfSnapshotSample[] | null { + if (memoryController === null) return null; + return memoryController.snapshot().map((s) => ({ + rss: s.rss, + heapUsed: s.heapUsed, + external: s.external, + arrayBuffers: s.arrayBuffers, + uptimeMs: s.uptimeMs, + msSinceLastOperation: s.msSinceLastOperation, + timestampMs: s.timestampMs, + })); + }, + getActiveOperationSummary() { + return registry.getActiveOperationSnapshot(); + }, + getSelfHealth(): PerfSelfHealth { + return { + lastWriteErrorCode: sink.lastWriteErrorCode, + evictionCount: retention.evictionCount, + }; + }, + }; +} diff --git a/packages/cli/src/ui/hooks/useMemoryMonitor.test.tsx b/packages/cli/src/ui/hooks/useMemoryMonitor.test.tsx index 81be5bad00..de22275b9e 100644 --- a/packages/cli/src/ui/hooks/useMemoryMonitor.test.tsx +++ b/packages/cli/src/ui/hooks/useMemoryMonitor.test.tsx @@ -16,7 +16,7 @@ import { MessageType } from '../types.js'; describe('useMemoryMonitor', () => { const addItem = vi.fn(); - const memoryUsageSpy = vi.spyOn(process, 'memoryUsage'); + const rssSpy = vi.spyOn(process.memoryUsage, 'rss'); beforeEach(() => { vi.useFakeTimers(); @@ -28,18 +28,14 @@ describe('useMemoryMonitor', () => { }); it('does not emit a warning when usage is below the threshold', () => { - memoryUsageSpy.mockReturnValue({ - rss: MEMORY_WARNING_THRESHOLD_BYTES / 4, - } as NodeJS.MemoryUsage); + rssSpy.mockReturnValue(MEMORY_WARNING_THRESHOLD_BYTES / 4); renderHook(() => useMemoryMonitor({ addItem })); vi.advanceTimersByTime(MEMORY_CHECK_INTERVAL_MS * 2); expect(addItem).not.toHaveBeenCalled(); }); it('emits a warning once when usage exceeds the threshold', () => { - memoryUsageSpy.mockReturnValue({ - rss: MEMORY_WARNING_THRESHOLD_BYTES * 1.2, - } as NodeJS.MemoryUsage); + rssSpy.mockReturnValue(MEMORY_WARNING_THRESHOLD_BYTES * 1.2); renderHook(() => useMemoryMonitor({ addItem })); vi.advanceTimersByTime(MEMORY_CHECK_INTERVAL_MS); diff --git a/packages/cli/src/ui/hooks/useMemoryMonitor.ts b/packages/cli/src/ui/hooks/useMemoryMonitor.ts index 28f0905011..ace5e877c8 100644 --- a/packages/cli/src/ui/hooks/useMemoryMonitor.ts +++ b/packages/cli/src/ui/hooks/useMemoryMonitor.ts @@ -7,35 +7,112 @@ import { useEffect } from 'react'; import process from 'node:process'; import { MessageType, type HistoryItemWithoutId } from '../types.js'; +import type { MemoryTelemetryController } from './memoryTrend/memoryTelemetry.js'; export const MEMORY_WARNING_THRESHOLD_BYTES = 7 * 1024 * 1024 * 1024; // 7GB export const MEMORY_CHECK_INTERVAL_MS = 60 * 1000; // 1 minute -interface UseMemoryMonitorOptions { +/** + * Package-private ports for deterministic testing. The real implementation + * uses `globalThis.setInterval` / `globalThis.clearInterval` / + * `process.memoryUsage`. Tests inject a controllable scheduler and memory + * function via {@link __setMemoryMonitorPortsForTesting}. + */ +export interface MemoryMonitorPorts { + setInterval: (handler: () => void, ms: number) => unknown; + clearInterval: (id: unknown) => void; + memoryUsage: () => NodeJS.MemoryUsage; + /** + * Cheaper RSS-only sampler used on the disabled (no controller) path so the + * warning check avoids allocating a full MemoryUsage object every tick. + * Defaults to process.memoryUsage.rss(). + */ + rssBytes: () => number; +} + +const realPorts: MemoryMonitorPorts = { + setInterval: (h, ms) => globalThis.setInterval(h, ms), + clearInterval: (id) => + globalThis.clearInterval(id as ReturnType), + memoryUsage: () => process.memoryUsage(), + rssBytes: () => process.memoryUsage.rss(), +}; + +let __portsForTesting: MemoryMonitorPorts | null = null; + +/** + * Package-private test seam. Inject controllable ports for deterministic + * timer/memory behavior, or pass `null` to restore the real ports. + */ +export function __setMemoryMonitorPortsForTesting( + ports: MemoryMonitorPorts | null, +): void { + __portsForTesting = ports; +} + +/** + * Package-private test seam exposing the real production default ports so + * tests can assert behavior of the actual default (e.g. that `rssBytes` uses + * the cheap `process.memoryUsage.rss()` rather than the full object). + */ +export function __getRealMemoryMonitorPortsForTesting(): MemoryMonitorPorts { + return realPorts; +} + +export interface UseMemoryMonitorOptions { addItem: (item: HistoryItemWithoutId, timestamp: number) => void; + /** + * Optional memory telemetry controller. When present (perf+memory enabled), + * each 60 s tick records the full memory sample to the ring and writes a + * `memory_sample` record. When absent, the hook retains its warn-only + * behaviour. P12 wires this based on real settings. + */ + memoryController?: MemoryTelemetryController; } -export function useMemoryMonitor({ addItem }: UseMemoryMonitorOptions): void { +export function useMemoryMonitor({ + addItem, + memoryController, +}: UseMemoryMonitorOptions): void { useEffect(() => { - const intervalId = setInterval(() => { - const rssUsage = process.memoryUsage().rss; - if (rssUsage > MEMORY_WARNING_THRESHOLD_BYTES) { + const ports = __portsForTesting ?? realPorts; + // Warn-once latch, separated from the sampling loop (DEFECT 1 fix): + // the interval continues regardless of whether a warning has fired. + let warnedOnce = false; + + // Shared high-memory warning so the telemetry-enabled and disabled paths + // cannot drift. Fires once via the latch. + const maybeWarn = (rss: number): void => { + if (rss > MEMORY_WARNING_THRESHOLD_BYTES && !warnedOnce) { addItem( { type: MessageType.WARNING, text: `High memory usage detected: ${( - rssUsage / + rss / (1024 * 1024 * 1024) ).toFixed(2)} GB. ` + 'If the CLI exits unexpectedly, please run `/bug` to report it.', }, Date.now(), ); - clearInterval(intervalId); + warnedOnce = true; + } + }; + + const intervalId = ports.setInterval(() => { + if (memoryController !== undefined) { + // Telemetry-enabled tick: exactly one full process.memoryUsage() + // capture, reused for warning + ring + persistence. + const sample = ports.memoryUsage(); + maybeWarn(sample.rss); + memoryController.recordTickSample(sample); + } else { + // Disabled path: cheaper rss-only check (no full MemoryUsage object). + maybeWarn(ports.rssBytes()); } }, MEMORY_CHECK_INTERVAL_MS); - return () => clearInterval(intervalId); - }, [addItem]); + return () => ports.clearInterval(intervalId); + }, [addItem, memoryController]); } diff --git a/packages/cli/src/ui/inkRenderOptions.observer.behavior.test.ts b/packages/cli/src/ui/inkRenderOptions.observer.behavior.test.ts new file mode 100644 index 0000000000..8ebdd0cee7 --- /dev/null +++ b/packages/cli/src/ui/inkRenderOptions.observer.behavior.test.ts @@ -0,0 +1,174 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, it, expect, beforeEach } from 'bun:test'; +import { + inkRenderOptions, + getInteractiveStdio, + setInteractiveStdoutObserver, + setInteractiveRenderObserver, +} from './inkRenderOptions.js'; +import { + type StdoutWriteObserver, + writeToStdout, +} from '@vybestack/llxprt-code-core'; + +const baseConfig = { getScreenReader: () => false }; +const baseSettings = { merged: { ui: {} } }; + +describe('interactive stdio — lazy cache + stdout observer', () => { + beforeEach(() => { + setInteractiveStdoutObserver(null); + setInteractiveRenderObserver(null); + }); + + it('getInteractiveStdio returns the same cached instance on repeated calls', () => { + const a = getInteractiveStdio(); + const b = getInteractiveStdio(); + expect(a).toBe(b); + }); + + it('setting a stdout observer before first build carries it into the stdio', () => { + let count = 0; + const observer: StdoutWriteObserver = { onWrite: () => count++ }; + setInteractiveStdoutObserver(observer); + const { stdout } = getInteractiveStdio(); + stdout.write(''); + expect(count).toBe(1); + }); + + it('setting a different stdout observer invalidates the cache', () => { + let first = 0; + let second = 0; + const obs1: StdoutWriteObserver = { onWrite: () => first++ }; + const obs2: StdoutWriteObserver = { onWrite: () => second++ }; + + setInteractiveStdoutObserver(obs1); + const stdioA = getInteractiveStdio(); + stdioA.stdout.write(''); + expect(first).toBe(1); + + setInteractiveStdoutObserver(obs2); + const stdioB = getInteractiveStdio(); + expect(stdioB).not.toBe(stdioA); + + stdioB.stdout.write(''); + expect(second).toBe(1); + expect(first).toBe(1); + + stdioA.stdout.write(''); + expect(second).toBe(2); + expect(first).toBe(1); + }); + + it('clearing the observer detaches an already-built stdout proxy', () => { + let count = 0; + setInteractiveStdoutObserver({ onWrite: () => count++ }); + const { stdout } = getInteractiveStdio(); + + setInteractiveStdoutObserver(null); + stdout.write(''); + + expect(count).toBe(0); + }); + + it('setting the same stdout observer reuses the cached instance', () => { + const observer: StdoutWriteObserver = { onWrite: () => {} }; + setInteractiveStdoutObserver(observer); + const first = getInteractiveStdio(); + setInteractiveStdoutObserver(observer); + const second = getInteractiveStdio(); + expect(second).toBe(first); + }); + + it('clearing to null when already null reuses the cached instance', () => { + const first = getInteractiveStdio(); + setInteractiveStdoutObserver(null); + const second = getInteractiveStdio(); + expect(second).toBe(first); + }); + + it('default-off: no observer set means stdout proxy write is the unobserved writeToStdout', () => { + const { stdout } = getInteractiveStdio(); + expect(Object.is(stdout.write, writeToStdout)).toBe(true); + }); +}); + +describe('interactive render observer — onRender wiring', () => { + beforeEach(() => { + setInteractiveStdoutObserver(null); + setInteractiveRenderObserver(null); + }); + + it('no render observer set means onRender is not wired (default-off)', () => { + const opts = inkRenderOptions(baseConfig, baseSettings); + expect(opts.onRender).toBeUndefined(); + }); + + it('wires onRender to the render observer, forwarding Ink renderTime', () => { + let captured = -1; + setInteractiveRenderObserver({ onRender: (ms) => (captured = ms) }); + const opts = inkRenderOptions(baseConfig, baseSettings); + expect(typeof opts.onRender).toBe('function'); + opts.onRender?.({ renderTime: 7.5 }); + expect(captured).toBe(7.5); + }); + + it('clearing the render observer detaches existing options and new wiring', () => { + let count = 0; + setInteractiveRenderObserver({ onRender: () => count++ }); + const activeOptions = inkRenderOptions(baseConfig, baseSettings); + + setInteractiveRenderObserver(null); + activeOptions.onRender?.({ renderTime: 1 }); + const clearedOptions = inkRenderOptions(baseConfig, baseSettings); + + expect(count).toBe(0); + expect(clearedOptions.onRender).toBeUndefined(); + }); + + it('render passes are counted distinctly from stdout write calls', () => { + let renderCount = 0; + let writeCount = 0; + setInteractiveRenderObserver({ onRender: () => renderCount++ }); + setInteractiveStdoutObserver({ onWrite: () => writeCount++ }); + + const { stdout } = getInteractiveStdio(); + const opts = inkRenderOptions(baseConfig, baseSettings); + + stdout.write(''); + stdout.write(''); + stdout.write(''); + opts.onRender?.({ renderTime: 1 }); + + expect(writeCount).toBe(3); + expect(renderCount).toBe(1); + }); +}); + +describe('inkRenderOptions — existing options preserved with observer seam', () => { + beforeEach(() => { + setInteractiveStdoutObserver(null); + setInteractiveRenderObserver(null); + }); + + it('still returns the base render options (default-off, no onRender)', () => { + const opts = inkRenderOptions( + { getScreenReader: () => true }, + { merged: { ui: { useAlternateBuffer: true } } }, + ); + expect(opts).toStrictEqual( + expect.objectContaining({ + exitOnCtrlC: false, + patchConsole: false, + isScreenReaderEnabled: true, + alternateBuffer: false, + incrementalRendering: false, + }), + ); + expect(opts.onRender).toBeUndefined(); + }); +}); diff --git a/packages/cli/src/ui/inkRenderOptions.ts b/packages/cli/src/ui/inkRenderOptions.ts index e20082fccf..0c67399bab 100644 --- a/packages/cli/src/ui/inkRenderOptions.ts +++ b/packages/cli/src/ui/inkRenderOptions.ts @@ -5,7 +5,25 @@ */ import type { RenderOptions } from 'ink'; -import { createInkStdio } from '@vybestack/llxprt-code-core'; +import { + createInkStdio, + type StdoutWriteObserver, +} from '@vybestack/llxprt-code-core'; + +/** + * The render-metrics payload Ink passes to its onRender callback. Extracted + * from the installed Ink's RenderOptions type so it tracks the actual API + * (verified: the installed @jrichman/ink provides `{ renderTime: number }`). + */ +type InkRenderMetrics = Parameters>[0]; + +/** + * CLI-owned render observer. P06/P07 install an instance to accumulate render + * passes and duration. Ink provides renderTime (a real duration) per pass. + */ +export interface InteractiveRenderObserver { + onRender(renderTimeMs: number): void; +} type InkRenderOptionsConfig = { getScreenReader(): boolean; @@ -20,8 +38,79 @@ type InkRenderOptionsSettings = { }; }; -// Create stdio streams once so they are reused across calls. -const sharedStdio = createInkStdio(); +// --- Lazy cached interactive stdio seam (P05) --- +// Replaces the former eager module-scope createInkStdio() call so that an +// optional stdout observer can be installed before the first render. Zed's +// separate direct createInkStdio() call stays observer-free and uncounted. +let interactiveStdio: ReturnType | null = null; +let interactiveStdoutObserver: StdoutWriteObserver | null = null; +const forwardingStdoutObserver: StdoutWriteObserver = { + onWrite(encodedBytes, syncDurationMs): void { + interactiveStdoutObserver?.onWrite(encodedBytes, syncDurationMs); + }, +}; + +// --- Optional render observer (P05) --- +let interactiveRenderObserver: InteractiveRenderObserver | null = null; + +/** + * Installs (or clears) the optional stdout observer on the interactive Ink + * instance. Must be called before the first render (i.e. before + * {@link getInteractiveStdio} builds the cache). Setting a different observer + * invalidates the cache so the next build carries it; setting the same value + * is a no-op that reuses the cached instance. + * + * Default-off: never calling this means no observer is installed and no + * counting work occurs. + */ +export function setInteractiveStdoutObserver( + observer: StdoutWriteObserver | null, +): void { + if (observer === interactiveStdoutObserver) { + return; + } + interactiveStdoutObserver = observer; + interactiveStdio = null; +} + +/** + * Returns the cached interactive stdio, building it lazily on first access + * with whatever stdout observer (if any) was installed beforehand. + */ +export function getInteractiveStdio(): ReturnType { + return (interactiveStdio ??= createInkStdio( + interactiveStdoutObserver === null ? undefined : forwardingStdoutObserver, + )); +} + +/** + * Installs (or clears) the optional render observer wired to Ink's onRender + * callback on the interactive instance. P06/P07 install an instance to record + * render passes and duration. Default-off: never calling this means no onRender + * wiring is added to the returned RenderOptions. + */ +export function setInteractiveRenderObserver( + observer: InteractiveRenderObserver | null, +): void { + interactiveRenderObserver = observer; +} + +/** + * Returns the currently installed interactive render observer, or null when + * none is installed. Used by the perf registry for identity-safe disposal + * (clear only if the observer still points at this registry). + */ +export function getInteractiveRenderObserver(): InteractiveRenderObserver | null { + return interactiveRenderObserver; +} + +/** + * Returns the currently installed stdout observer, or null when none is + * installed. Used by the perf registry for identity-safe disposal. + */ +export function getInteractiveStdoutObserver(): StdoutWriteObserver | null { + return interactiveStdoutObserver; +} /** * @plan PLAN-20251215-OLDUI-SCROLL.P04 @@ -37,13 +126,26 @@ export const inkRenderOptions = ( const incrementalRendering = useAlternateBuffer && settings.merged.ui.incrementalRendering !== false; - return { - stdout: sharedStdio.stdout, - stderr: sharedStdio.stderr, + const stdio = getInteractiveStdio(); + + const options: RenderOptions = { + stdout: stdio.stdout, + stderr: stdio.stderr, exitOnCtrlC: false, patchConsole: false, isScreenReaderEnabled, alternateBuffer: useAlternateBuffer, incrementalRendering, }; + + // Render observer is wired only when installed (default-off). The callback + // resolves the active observer at invocation time so disposal detaches + // already-built render options. + if (interactiveRenderObserver !== null) { + options.onRender = (metrics: InkRenderMetrics) => { + interactiveRenderObserver?.onRender(metrics.renderTime); + }; + } + + return options; }; diff --git a/packages/core/package.json b/packages/core/package.json index 24e7cfb872..7649116993 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -232,6 +232,10 @@ "bun": "./src/telemetry/sdk.ts", "import": "./dist/src/telemetry/sdk.js" }, + "./perf/perfPhaseObserver.js": { + "bun": "./src/perf/perfPhaseObserver.ts", + "import": "./dist/src/perf/perfPhaseObserver.js" + }, "./telemetry/constants.js": { "bun": "./src/telemetry/constants.ts", "import": "./dist/src/telemetry/constants.js" diff --git a/packages/core/src/config/config.ts b/packages/core/src/config/config.ts index d11f9c3396..fd0f1ca7f5 100644 --- a/packages/core/src/config/config.ts +++ b/packages/core/src/config/config.ts @@ -34,10 +34,7 @@ import { type SchedulerOptions, } from './schedulerSingleton.js'; import { initializeLsp } from './lspIntegration.js'; -import { - applyConfigParams, - type ConfigConstructorTarget, -} from './configConstructor.js'; +import * as configConstructor from './configConstructor.js'; import { ConfigBase } from './configBase.js'; import { buildNewContentGeneratorConfig, @@ -125,7 +122,10 @@ export class Config extends ConfigBase { constructor(params: ConfigParameters) { super(); - applyConfigParams(this as unknown as ConfigConstructorTarget, params); + configConstructor.applyConfigParams( + this as unknown as configConstructor.ConfigConstructorTarget, + params, + ); this.syncModeDerivedPolicyRules(this.approvalMode); this.cachedEffectiveTrust = this.isTrustedFolder(); this.liveTrustTransitionLifecycle = new LiveTrustTransitionLifecycle({ @@ -587,14 +587,14 @@ export class Config extends ConfigBase { } getTelemetrySettings(): TelemetrySettings { - return { ...this.telemetrySettings }; + return configConstructor.withClonedPerf(this.telemetrySettings); } updateTelemetrySettings(settings: Partial): void { - this.telemetrySettings = { - ...this.telemetrySettings, - ...settings, - }; + this.telemetrySettings = configConstructor.mergeTelemetrySettings( + this.telemetrySettings, + settings, + ); // If we have a provider manager, update its config to trigger re-wrapping if (this.providerManager) { diff --git a/packages/core/src/config/configBaseCore.ts b/packages/core/src/config/configBaseCore.ts index a4de050069..e201c0795c 100644 --- a/packages/core/src/config/configBaseCore.ts +++ b/packages/core/src/config/configBaseCore.ts @@ -67,6 +67,7 @@ import type { ToolRecord } from './toolRegistryFactory.js'; import type { LspState } from './lspIntegration.js'; import type { ApprovalMode, MCPServerConfig } from './configTypes.js'; import type { ImageOperationRunner } from '../services/image/imageCapability.js'; +import { resolvePerfSettings } from './configConstructor.js'; import { type AccessibilitySettings, type BugCommandSettings, @@ -640,6 +641,21 @@ export abstract class ConfigBaseCore { getTelemetryEnabled(): boolean { return this.telemetrySettings.enabled ?? false; } + /** + * Effective perf telemetry master switch. Delegates to resolvePerfSettings + * so gating policy lives in exactly one place. + */ + getTelemetryPerfEnabled(): boolean { + return resolvePerfSettings(this.telemetrySettings).enabled; + } + /** + * Effective perf memory flag. Master-gated: returns false when the perf + * master switch (getTelemetryPerfEnabled) is off, regardless of the stored + * memory value. Delegates to resolvePerfSettings. + */ + getTelemetryPerfMemory(): boolean { + return resolvePerfSettings(this.telemetrySettings).memory; + } getTelemetryLogPromptsEnabled(): boolean { return this.telemetrySettings.logPrompts ?? true; } diff --git a/packages/core/src/config/configConstructor.ts b/packages/core/src/config/configConstructor.ts index 462427e811..c1e6124852 100644 --- a/packages/core/src/config/configConstructor.ts +++ b/packages/core/src/config/configConstructor.ts @@ -315,11 +315,32 @@ function applyTelemetrySettings( config.usageStatisticsEnabled = params.usageStatisticsEnabled ?? true; } -function resolveTelemetrySettings( +/** + * Returns a shallow copy of `settings` with the nested perf sub-object + * defensively cloned, so mutating the result (or its perf) cannot reach the + * source. P09 copy policy: isolation by cloning on every ingress and egress — + * never by freezing. Used by resolveTelemetrySettings (constructor ingress), + * Config.updateTelemetrySettings (update ingress), and + * Config.getTelemetrySettings (egress). + */ +export function withClonedPerf(settings: TelemetrySettings): TelemetrySettings { + const { perf, ...rest } = settings; + return { ...rest, ...(perf ? { perf: { ...perf } } : {}) }; +} + +export function mergeTelemetrySettings( + current: TelemetrySettings, + update: Partial, +): TelemetrySettings { + return withClonedPerf({ ...current, ...update }); +} + +export function resolveTelemetrySettings( telemetry: TelemetrySettings | undefined, ): TelemetrySettings { - return { - ...(telemetry ?? {}), + const { perf, ...rest } = telemetry ?? {}; + return withClonedPerf({ + ...rest, enabled: telemetry?.enabled ?? false, logPrompts: telemetry?.logPrompts ?? true, outfile: telemetry?.outfile, @@ -330,7 +351,30 @@ function resolveTelemetrySettings( redactUrls: telemetry?.redactUrls ?? false, redactEmails: telemetry?.redactEmails ?? false, redactPersonalInfo: telemetry?.redactPersonalInfo ?? false, - }; + ...(perf ? { perf } : {}), + }); +} + +/** + * Pure resolver for perf telemetry settings (D2). + * + * Returns the effective perf state from a TelemetrySettings object. + * Both fields default to false. When `enabled` is false, memory is + * forced to false regardless of its configured value (master gates memory). + * + * Does not mutate the input. Returns a fresh object so callers cannot + * affect subsequent resolutions by mutating the result. + */ +export function resolvePerfSettings(settings: TelemetrySettings | undefined): { + enabled: boolean; + memory: boolean; +} { + const enabled = settings?.perf?.enabled ?? false; + const memory = settings?.perf?.memory ?? false; + if (!enabled) { + return { enabled: false, memory: false }; + } + return { enabled: true, memory }; } function applyFileFilteringSettings( diff --git a/packages/core/src/config/configPerfGetters.behavior.test.ts b/packages/core/src/config/configPerfGetters.behavior.test.ts new file mode 100644 index 0000000000..ba07c18363 --- /dev/null +++ b/packages/core/src/config/configPerfGetters.behavior.test.ts @@ -0,0 +1,121 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + * + * Behavioral evidence for the Config-level perf read API: + * getTelemetryPerfEnabled (master switch) and getTelemetryPerfMemory + * (master-gated memory flag). Both delegate to resolvePerfSettings so the + * gating policy lives in one place. These tests exercise the real Config. + */ + +import { describe, it, expect } from 'bun:test'; +import { Config } from './config.js'; +import type { ConfigParameters } from './config.js'; + +function makeConfig(telemetry?: ConfigParameters['telemetry']): Config { + return new Config({ + sessionId: 'perf-getters-session', + targetDir: '.', + cwd: '.', + debugMode: false, + model: 'test-model', + usageStatisticsEnabled: false, + telemetry, + }); +} + +describe('Config.getTelemetryPerfEnabled / getTelemetryPerfMemory', () => { + describe('defaults', () => { + it('perf enabled defaults to false when perf is absent', () => { + const config = makeConfig({ enabled: true }); + expect(config.getTelemetryPerfEnabled()).toBe(false); + }); + + it('perf memory defaults to false when perf is absent', () => { + const config = makeConfig({ enabled: true }); + expect(config.getTelemetryPerfMemory()).toBe(false); + }); + + it('both default to false for an empty perf object', () => { + const config = makeConfig({ perf: {} }); + expect(config.getTelemetryPerfEnabled()).toBe(false); + expect(config.getTelemetryPerfMemory()).toBe(false); + }); + }); + + describe('master switch', () => { + it('reflects perf.enabled when on', () => { + const config = makeConfig({ perf: { enabled: true } }); + expect(config.getTelemetryPerfEnabled()).toBe(true); + }); + + it('reflects perf.enabled when off', () => { + const config = makeConfig({ perf: { enabled: false } }); + expect(config.getTelemetryPerfEnabled()).toBe(false); + }); + }); + + describe('memory is master-gated', () => { + it('memory is false when the master switch is off, even if memory is configured on', () => { + const config = makeConfig({ perf: { enabled: false, memory: true } }); + expect(config.getTelemetryPerfMemory()).toBe(false); + }); + + it('memory is false when only memory is set and master is absent (defaults off)', () => { + const config = makeConfig({ perf: { memory: true } }); + expect(config.getTelemetryPerfMemory()).toBe(false); + }); + + it('memory is true only when the master switch is on and memory is on', () => { + const config = makeConfig({ perf: { enabled: true, memory: true } }); + expect(config.getTelemetryPerfMemory()).toBe(true); + }); + + it('memory is false when the master switch is on but memory is off', () => { + const config = makeConfig({ perf: { enabled: true, memory: false } }); + expect(config.getTelemetryPerfMemory()).toBe(false); + }); + }); + + describe('reflects updates', () => { + it('getters reflect an update that enables the master switch and memory', () => { + const config = makeConfig(); + expect(config.getTelemetryPerfEnabled()).toBe(false); + expect(config.getTelemetryPerfMemory()).toBe(false); + + config.updateTelemetrySettings({ perf: { enabled: true, memory: true } }); + expect(config.getTelemetryPerfEnabled()).toBe(true); + expect(config.getTelemetryPerfMemory()).toBe(true); + }); + + it('disabling the master switch gates memory back to false', () => { + const config = makeConfig({ perf: { enabled: true, memory: true } }); + config.updateTelemetrySettings({ + perf: { enabled: false, memory: true }, + }); + + expect(config.getTelemetryPerfEnabled()).toBe(false); + expect(config.getTelemetryPerfMemory()).toBe(false); + }); + }); + + describe('return types', () => { + it('always returns booleans across the full state space', () => { + const cases: Array = [ + undefined, + { enabled: true }, + { perf: {} }, + { perf: { enabled: true } }, + { perf: { enabled: false } }, + { perf: { enabled: true, memory: true } }, + { perf: { enabled: false, memory: true } }, + ]; + for (const telemetry of cases) { + const config = makeConfig(telemetry); + expect(typeof config.getTelemetryPerfEnabled()).toBe('boolean'); + expect(typeof config.getTelemetryPerfMemory()).toBe('boolean'); + } + }); + }); +}); diff --git a/packages/core/src/config/configTypes.ts b/packages/core/src/config/configTypes.ts index 656142d552..112b515b51 100644 --- a/packages/core/src/config/configTypes.ts +++ b/packages/core/src/config/configTypes.ts @@ -114,6 +114,18 @@ export interface IntrospectionAgentSettings { enabled?: boolean; } +/** + * Client-side performance telemetry settings (D2). + * + * The persisted shape is nested: `telemetry.perf.enabled` (master) and + * `telemetry.perf.memory`, both default false. `telemetry.perf` itself is + * an object, never a boolean. Memory is effective only when enabled is true. + */ +export interface PerfTelemetrySettings { + enabled?: boolean; + memory?: boolean; +} + export interface TelemetrySettings { enabled?: boolean; logPrompts?: boolean; @@ -140,6 +152,7 @@ export interface TelemetrySettings { enableDataRetention?: boolean; conversationExpirationDays?: number; maxConversationsStored?: number; + perf?: PerfTelemetrySettings; } /** diff --git a/packages/core/src/config/perfSettings.behavior.test.ts b/packages/core/src/config/perfSettings.behavior.test.ts new file mode 100644 index 0000000000..decda6baa3 --- /dev/null +++ b/packages/core/src/config/perfSettings.behavior.test.ts @@ -0,0 +1,179 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + * + * Behavioral tests for resolvePerfSettings and the perf-related + * TelemetrySettings resolution. EVIDENCE-AC2: default-off, master-gates-memory, + * input immutability, and nested-return copy isolation. + */ + +import { describe, it, expect } from 'bun:test'; +import { resolvePerfSettings } from './configConstructor.js'; +import type { TelemetrySettings } from './configTypes.js'; + +describe('resolvePerfSettings', () => { + describe('default-off (absent settings)', () => { + it('resolves undefined to fully disabled', () => { + expect(resolvePerfSettings(undefined)).toEqual({ + enabled: false, + memory: false, + }); + }); + + it('resolves an empty object to fully disabled', () => { + expect(resolvePerfSettings({})).toEqual({ + enabled: false, + memory: false, + }); + }); + + it('resolves a telemetry object with no perf key to fully disabled', () => { + const settings: TelemetrySettings = { enabled: true }; + expect(resolvePerfSettings(settings)).toEqual({ + enabled: false, + memory: false, + }); + }); + + it('resolves a telemetry object with an empty perf object to fully disabled', () => { + const settings: TelemetrySettings = { perf: {} }; + expect(resolvePerfSettings(settings)).toEqual({ + enabled: false, + memory: false, + }); + }); + }); + + describe('enabled only (master on, memory omitted)', () => { + it('resolves enabled:true, memory absent to { enabled: true, memory: false }', () => { + const settings: TelemetrySettings = { perf: { enabled: true } }; + expect(resolvePerfSettings(settings)).toEqual({ + enabled: true, + memory: false, + }); + }); + }); + + describe('memory gated off (master off, memory on)', () => { + it('resolves enabled:false, memory:true to fully disabled (master gates memory)', () => { + const settings: TelemetrySettings = { + perf: { enabled: false, memory: true }, + }; + expect(resolvePerfSettings(settings)).toEqual({ + enabled: false, + memory: false, + }); + }); + + it('resolves enabled absent, memory:true to fully disabled (master gates memory)', () => { + const settings: TelemetrySettings = { perf: { memory: true } }; + expect(resolvePerfSettings(settings)).toEqual({ + enabled: false, + memory: false, + }); + }); + }); + + describe('both on', () => { + it('resolves enabled:true, memory:true to { enabled: true, memory: true }', () => { + const settings: TelemetrySettings = { + perf: { enabled: true, memory: true }, + }; + expect(resolvePerfSettings(settings)).toEqual({ + enabled: true, + memory: true, + }); + }); + }); + + describe('false overrides', () => { + it('resolves enabled:false, memory:false to fully disabled', () => { + const settings: TelemetrySettings = { + perf: { enabled: false, memory: false }, + }; + expect(resolvePerfSettings(settings)).toEqual({ + enabled: false, + memory: false, + }); + }); + + it('resolves enabled:true, memory:false to enabled without memory', () => { + const settings: TelemetrySettings = { + perf: { enabled: true, memory: false }, + }; + expect(resolvePerfSettings(settings)).toEqual({ + enabled: true, + memory: false, + }); + }); + }); + + describe('input immutability', () => { + it('does not mutate the caller settings object', () => { + const settings: TelemetrySettings = { + perf: { enabled: true, memory: true }, + }; + const snapshot = JSON.parse(JSON.stringify(settings)); + resolvePerfSettings(settings); + expect(settings).toEqual(snapshot); + }); + + it('does not mutate the perf sub-object', () => { + const perf = { enabled: true, memory: true }; + const settings: TelemetrySettings = { perf }; + resolvePerfSettings(settings); + expect(perf).toEqual({ enabled: true, memory: true }); + }); + + it('does not mutate a settings object with enabled:false, memory:true', () => { + const settings: TelemetrySettings = { + perf: { enabled: false, memory: true }, + }; + const snapshot = JSON.parse(JSON.stringify(settings)); + resolvePerfSettings(settings); + expect(settings).toEqual(snapshot); + }); + }); + + describe('nested-return copy isolation', () => { + it('returns a fresh object whose mutation does not affect subsequent calls', () => { + const settings: TelemetrySettings = { + perf: { enabled: true, memory: true }, + }; + const result1 = resolvePerfSettings(settings); + result1.enabled = false; + result1.memory = false; + const result2 = resolvePerfSettings(settings); + expect(result2).toEqual({ enabled: true, memory: true }); + }); + + it('returns primitive booleans (no shared reference to input perf)', () => { + const perf = { enabled: true, memory: false }; + const settings: TelemetrySettings = { perf }; + const result = resolvePerfSettings(settings); + // The returned object is a new object; mutating it does not change the input + result.enabled = false; + expect(perf.enabled).toBe(true); + expect(settings.perf?.enabled).toBe(true); + }); + }); + + describe('return type safety', () => { + it('always returns enabled as a boolean', () => { + const cases: Array = [ + undefined, + {}, + { perf: {} }, + { perf: { enabled: true } }, + { perf: { enabled: false } }, + { perf: { enabled: true, memory: true } }, + ]; + for (const settings of cases) { + const result = resolvePerfSettings(settings); + expect(typeof result.enabled).toBe('boolean'); + expect(typeof result.memory).toBe('boolean'); + } + }); + }); +}); diff --git a/packages/core/src/config/telemetryConfigCopy.behavior.test.ts b/packages/core/src/config/telemetryConfigCopy.behavior.test.ts new file mode 100644 index 0000000000..151edd1c87 --- /dev/null +++ b/packages/core/src/config/telemetryConfigCopy.behavior.test.ts @@ -0,0 +1,141 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + * + * Behavioral evidence for Config-level perf copy isolation. P09 gap: the + * nested perf sub-object must be defensively cloned on ingress (constructor + * resolution and updateTelemetrySettings) and on egress (getTelemetrySettings) + * so that mutating a returned or supplied perf object can never reach internal + * state. Isolation is provided by copying, not by freezing. + * + * These tests exercise the real Config public methods end-to-end. + */ + +import { describe, it, expect } from 'bun:test'; +import { Config } from './config.js'; +import type { ConfigParameters } from './config.js'; + +function makeConfig(telemetry?: ConfigParameters['telemetry']): Config { + return new Config({ + sessionId: 'perf-copy-session', + targetDir: '.', + cwd: '.', + debugMode: false, + model: 'test-model', + usageStatisticsEnabled: false, + telemetry, + }); +} + +describe('Config telemetry perf copy isolation', () => { + describe('constructor/get copy isolation', () => { + it('getTelemetrySettings returns a perf object that is a copy of the internal reference', () => { + const config = makeConfig({ + perf: { enabled: true, memory: true }, + }); + const got = config.getTelemetrySettings(); + + expect(got.perf).toEqual({ enabled: true, memory: true }); + // The returned perf must not be the internal reference. + expect(got.perf).not.toBe( + (config as unknown as { telemetrySettings: { perf?: unknown } }) + .telemetrySettings.perf, + ); + }); + + it('mutating the returned perf does not affect a subsequent get', () => { + const config = makeConfig({ + perf: { enabled: true, memory: true }, + }); + const got = config.getTelemetrySettings(); + got.perf!.enabled = false; + got.perf!.memory = false; + + const gotAgain = config.getTelemetrySettings(); + expect(gotAgain.perf).toEqual({ enabled: true, memory: true }); + }); + + it('two consecutive gets return independent perf objects', () => { + const config = makeConfig({ + perf: { enabled: true, memory: false }, + }); + const first = config.getTelemetrySettings(); + const second = config.getTelemetrySettings(); + + expect(first.perf).not.toBe(second.perf); + first.perf!.enabled = false; + expect(second.perf?.enabled).toBe(true); + }); + + it('returned perf is not frozen — isolation is by copy, not by freeze', () => { + const config = makeConfig({ + perf: { enabled: true, memory: true }, + }); + const got = config.getTelemetrySettings(); + + expect(Object.isFrozen(got.perf)).toBe(false); + // Mutation succeeds (no throw) but must not leak into internal state. + expect(() => { + got.perf!.enabled = false; + }).not.toThrow(); + expect(config.getTelemetrySettings().perf?.enabled).toBe(true); + }); + }); + + describe('update/get copy isolation', () => { + it('updateTelemetrySettings clones a provided perf so caller mutation cannot affect internal state', () => { + const config = makeConfig(); + const callerPerf = { enabled: true, memory: true }; + config.updateTelemetrySettings({ perf: callerPerf }); + + // The stored perf must not be the caller's reference. + expect( + (config as unknown as { telemetrySettings: { perf?: unknown } }) + .telemetrySettings.perf, + ).not.toBe(callerPerf); + + // Mutating the caller object after update has no effect. + callerPerf.enabled = false; + callerPerf.memory = false; + expect(config.getTelemetrySettings().perf).toEqual({ + enabled: true, + memory: true, + }); + }); + + it('a perf obtained via get, then mutated, does not leak back through update', () => { + const config = makeConfig({ + perf: { enabled: true, memory: true }, + }); + const snapshot = config.getTelemetrySettings(); + // Hand the (already-isolated) perf back in via update, then mutate it. + config.updateTelemetrySettings({ perf: snapshot.perf }); + snapshot.perf!.enabled = false; + + expect(config.getTelemetrySettings().perf?.enabled).toBe(true); + }); + + it('omitting perf in update retains the previously-cloned internal perf', () => { + const config = makeConfig({ + perf: { enabled: true, memory: true }, + }); + config.updateTelemetrySettings({ logPrompts: false }); + + expect(config.getTelemetrySettings().perf).toEqual({ + enabled: true, + memory: true, + }); + }); + + it('providing a perf replaces it entirely — enabled/memory are not deep-merged', () => { + const config = makeConfig({ + perf: { enabled: true, memory: true }, + }); + // New perf omits memory: shallow replacement, not a merge. + config.updateTelemetrySettings({ perf: { enabled: true } }); + + expect(config.getTelemetrySettings().perf).toEqual({ enabled: true }); + }); + }); +}); diff --git a/packages/core/src/config/telemetrySettingsCopy.behavior.test.ts b/packages/core/src/config/telemetrySettingsCopy.behavior.test.ts new file mode 100644 index 0000000000..44276fac44 --- /dev/null +++ b/packages/core/src/config/telemetrySettingsCopy.behavior.test.ts @@ -0,0 +1,76 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + * + * Behavioral tests for resolveTelemetrySettings copy isolation of the + * nested perf sub-object. EVIDENCE-AC2: getTelemetrySettings must not leak + * nested mutable state; resolveTelemetrySettings must not mutate caller. + * + * The resolved perf object is a defensive clone (not frozen). Copy isolation + * is provided by cloning on ingress (resolveTelemetrySettings and + * Config.updateTelemetrySettings) and on egress (Config.getTelemetrySettings). + * Tests assert reference-separation and that cloning — not freezing — provides + * isolation. + */ + +import { describe, it, expect } from 'bun:test'; +import { resolveTelemetrySettings } from './configConstructor.js'; +import type { TelemetrySettings } from './configTypes.js'; + +describe('resolveTelemetrySettings — perf copy isolation', () => { + it('returns a perf object that is a copy, not the caller reference', () => { + const input: TelemetrySettings = { + perf: { enabled: true, memory: false }, + }; + const resolved = resolveTelemetrySettings(input); + expect(resolved.perf).toEqual({ enabled: true, memory: false }); + expect(resolved.perf).not.toBe(input.perf); + }); + + it('the resolved perf is a mutable copy — isolation is by cloning, not freezing', () => { + const input: TelemetrySettings = { + perf: { enabled: true, memory: true }, + }; + const resolved = resolveTelemetrySettings(input); + // Copy policy, not freeze: the resolved perf is not frozen. + expect(Object.isFrozen(resolved.perf)).toBe(false); + // Mutation succeeds (no throw), yet cloning still isolates the input. + expect(() => { + (resolved.perf as { enabled: boolean }).enabled = false; + }).not.toThrow(); + expect(input.perf?.enabled).toBe(true); + }); + + it('mutations to the input perf after resolution do not affect the resolved copy', () => { + const input: TelemetrySettings = { + perf: { enabled: true, memory: false }, + }; + const resolved = resolveTelemetrySettings(input); + input.perf!.enabled = false; + expect(resolved.perf?.enabled).toBe(true); + }); + + it('does not mutate the caller settings object', () => { + const input: TelemetrySettings = { + enabled: true, + perf: { enabled: true, memory: true }, + }; + const snapshot = JSON.parse(JSON.stringify(input)); + resolveTelemetrySettings(input); + expect(input).toEqual(snapshot); + }); + + it('resolves undefined perf to undefined (absent, not a fabricated object)', () => { + const resolved = resolveTelemetrySettings({ enabled: true }); + expect(resolved.perf).toBeUndefined(); + }); + + it('preserves perf fields from input through resolution', () => { + const input: TelemetrySettings = { + perf: { enabled: true, memory: true }, + }; + const resolved = resolveTelemetrySettings(input); + expect(resolved.perf).toEqual({ enabled: true, memory: true }); + }); +}); diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 30c3e8f94d..8b57a661af 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -685,6 +685,10 @@ export { // Explicit exports resolve star-export ambiguity between config.js and extensionLoader.js. export type { LlxprtExtension } from './config/configTypes.js'; +// Export the perf settings resolver for downstream phases (P06/P10/P12) +export { resolvePerfSettings } from './config/configConstructor.js'; +export type { PerfTelemetrySettings } from './config/configTypes.js'; + // Export MCP Client Manager — re-exported from @vybestack/llxprt-code-mcp (also available above) // Export models (legacy constants) diff --git a/packages/core/src/perf/perfPhaseObserver.ts b/packages/core/src/perf/perfPhaseObserver.ts new file mode 100644 index 0000000000..4e267064ee --- /dev/null +++ b/packages/core/src/perf/perfPhaseObserver.ts @@ -0,0 +1,29 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Narrow core re-export of the telemetry-owned perf phase observer seam + * (P07, issue #3167). + * + * Layering: telemetry owns the seam; core re-exports it so that packages + * (notably providers) which already depend on core — but do NOT declare a + * dependency on telemetry — can consume it without creating an undeclared + * dependency edge or a cycle. The CLI registry installs an implementation + * when perf is enabled; when absent (default-off), the getter returns null. + * + * No behaviour lives here — this is a pure re-export boundary. + */ + +export { + setPerfPhaseObserver, + getPerfPhaseObserver, +} from '@vybestack/llxprt-code-telemetry/perf/perfPhaseObserver.js'; +export type { + PerfPhaseObserver, + PerfProviderAttemptStartInfo, + PerfProviderAttemptEndInfo, + PerfToolCallCompletedInfo, +} from '@vybestack/llxprt-code-telemetry/perf/perfPhaseObserver.js'; diff --git a/packages/core/src/utils/stdio.observer.behavior.test.ts b/packages/core/src/utils/stdio.observer.behavior.test.ts new file mode 100644 index 0000000000..21135154a4 --- /dev/null +++ b/packages/core/src/utils/stdio.observer.behavior.test.ts @@ -0,0 +1,209 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, it, expect } from 'bun:test'; +import { + createInkStdio, + createObservedStdoutWrite, + writeToStdout, + type StdoutWriteObserver, +} from './stdio.js'; + +type StdoutWriteCallback = (err?: NodeJS.ErrnoException | null) => void; +type StdoutWriteArgs = + | [chunk: Uint8Array | string, callback?: StdoutWriteCallback] + | [ + chunk: Uint8Array | string, + encoding?: BufferEncoding, + callback?: StdoutWriteCallback, + ]; + +const noopUnderlying = (..._args: StdoutWriteArgs): boolean => true; + +const capturingObserver = (): { + observer: StdoutWriteObserver; + bytes: () => number; + duration: () => number; + count: () => number; +} => { + let b = -1; + let d = -1; + let c = 0; + return { + observer: { + onWrite: (encodedBytes, syncDurationMs) => { + b = encodedBytes; + d = syncDurationMs; + c++; + }, + }, + bytes: () => b, + duration: () => d, + count: () => c, + }; +}; + +describe('createObservedStdoutWrite — encoded byte counting', () => { + it('counts Uint8Array bytes by byteLength, not character count', () => { + const { observer, bytes } = capturingObserver(); + const observed = createObservedStdoutWrite(noopUnderlying, observer); + // "你好" encoded as UTF-8 = 6 bytes + const data = new Uint8Array([0xe4, 0xbd, 0xa0, 0xe5, 0xa5, 0xbd]); + observed(data); + expect(bytes()).toBe(6); + }); + + it('counts a multibyte UTF-8 string by encoded byte length', () => { + const { observer, bytes } = capturingObserver(); + const observed = createObservedStdoutWrite(noopUnderlying, observer); + observed('你好'); + expect(bytes()).toBe(6); + }); + + it('uses the supplied BufferEncoding for string byte counting', () => { + const { observer, bytes } = capturingObserver(); + const observed = createObservedStdoutWrite(noopUnderlying, observer); + // "café" is 4 bytes in latin1 but 5 bytes in UTF-8 + observed('café', 'latin1'); + expect(bytes()).toBe(4); + }); + + it('counts 0 bytes for an empty string', () => { + const { observer, bytes } = capturingObserver(); + const observed = createObservedStdoutWrite(noopUnderlying, observer); + observed(''); + expect(bytes()).toBe(0); + }); +}); + +describe('createObservedStdoutWrite — backpressure + callback passthrough', () => { + it('passes through true backpressure from the underlying write', () => { + const underlying = (..._args: StdoutWriteArgs): boolean => true; + const observed = createObservedStdoutWrite(underlying, { + onWrite: () => {}, + }); + expect(observed('x')).toBe(true); + }); + + it('passes through false backpressure from the underlying write', () => { + const underlying = (..._args: StdoutWriteArgs): boolean => false; + const observed = createObservedStdoutWrite(underlying, { + onWrite: () => {}, + }); + expect(observed('x')).toBe(false); + }); + + it('preserves the (chunk, callback) overload by forwarding to underlying', () => { + let cbInvoked = false; + const underlying = (...args: StdoutWriteArgs): boolean => { + const maybeCb = args[args.length - 1]; + if (typeof maybeCb === 'function') { + maybeCb(); + } + return true; + }; + const observed = createObservedStdoutWrite(underlying, { + onWrite: () => {}, + }); + observed('x', () => { + cbInvoked = true; + }); + expect(cbInvoked).toBe(true); + }); + + it('preserves the (chunk, encoding, callback) overload', () => { + let receivedEncoding: string | undefined; + let cbInvoked = false; + const underlying = (...args: StdoutWriteArgs): boolean => { + const maybeEnc = args[1]; + if (typeof maybeEnc === 'string') { + receivedEncoding = maybeEnc; + } + const maybeCb = args[args.length - 1]; + if (typeof maybeCb === 'function') { + maybeCb(); + } + return true; + }; + const observed = createObservedStdoutWrite(underlying, { + onWrite: () => {}, + }); + observed('x', 'utf8', () => { + cbInvoked = true; + }); + expect(receivedEncoding).toBe('utf8'); + expect(cbInvoked).toBe(true); + }); +}); + +describe('createObservedStdoutWrite — duration + call count', () => { + it('measures a finite, non-negative synchronous duration', () => { + const { observer, duration } = capturingObserver(); + const observed = createObservedStdoutWrite(noopUnderlying, observer); + observed('x'); + expect(Number.isFinite(duration())).toBe(true); + expect(duration()).toBeGreaterThanOrEqual(0); + }); + + it('invokes onWrite exactly once per write call', () => { + const { observer, count } = capturingObserver(); + const observed = createObservedStdoutWrite(noopUnderlying, observer); + observed('a'); + observed('b'); + observed('c'); + expect(count()).toBe(3); + }); +}); + +describe('createObservedStdoutWrite — error propagation (D8 fail-fast)', () => { + it('propagates an observer error without swallowing it', () => { + const observer: StdoutWriteObserver = { + onWrite: () => { + throw new Error('observer boom'); + }, + }; + const observed = createObservedStdoutWrite(noopUnderlying, observer); + expect(() => observed('x')).toThrow('observer boom'); + }); + + it('does not invoke the observer when the underlying write throws', () => { + const { observer, count } = capturingObserver(); + const throwingUnderlying = (..._args: StdoutWriteArgs): boolean => { + throw new Error('underlying boom'); + }; + const observed = createObservedStdoutWrite(throwingUnderlying, observer); + expect(() => observed('x')).toThrow('underlying boom'); + expect(count()).toBe(0); + }); +}); + +describe('createInkStdio — observer wiring + default-off', () => { + it('without an observer, stdout proxy write is writeToStdout (identity preserved)', () => { + const { stdout } = createInkStdio(); + expect(stdout.write).toBe(writeToStdout); + }); + + it('with an observer, stdout writes invoke the observer', () => { + const { observer, count } = capturingObserver(); + const { stdout } = createInkStdio(observer); + stdout.write(''); + expect(count()).toBe(1); + }); + + it('stderr writes are never observed even when a stdout observer is set', () => { + const { observer, count } = capturingObserver(); + const { stderr } = createInkStdio(observer); + stderr.write(''); + expect(count()).toBe(0); + }); + + it('Zed path characterization: createInkStdio() with no observer is uncounted', () => { + // Zed calls createInkStdio() directly with no observer — its writes must + // never be counted. This characterizes that contract. + const { stdout } = createInkStdio(); + expect(stdout.write).toBe(writeToStdout); + }); +}); diff --git a/packages/core/src/utils/stdio.ts b/packages/core/src/utils/stdio.ts index 506222df84..51c2b33f6e 100644 --- a/packages/core/src/utils/stdio.ts +++ b/packages/core/src/utils/stdio.ts @@ -6,6 +6,87 @@ import { coreEvents } from './events.js'; +type StdioWriteChunk = Uint8Array | string; +type StdioWriteCallback = (err?: NodeJS.ErrnoException | null) => void; + +/** + * The Node stdout write signature, covering both overloads: + * write(chunk, callback?) and write(chunk, encoding, callback?). + */ +interface StdoutWrite { + (chunk: StdioWriteChunk, callback?: StdioWriteCallback): boolean; + ( + chunk: StdioWriteChunk, + encoding?: BufferEncoding, + callback?: StdioWriteCallback, + ): boolean; +} + +type StdoutWriteArgs = + | [chunk: StdioWriteChunk, callback?: StdioWriteCallback] + | [ + chunk: StdioWriteChunk, + encoding?: BufferEncoding, + callback?: StdioWriteCallback, + ]; + +/** + * Second positional argument to stdout.write: either a BufferEncoding, a + * callback, or absent. + */ +type StdoutEncodingArg = BufferEncoding | StdioWriteCallback | undefined; + +/** + * Observer invoked once per real stdout write, after the write returns. + * Internal observer/programming errors must propagate (D8) — this callback is + * never wrapped in try/catch. Only the filesystem writer fails open. + */ +export interface StdoutWriteObserver { + onWrite(encodedBytes: number, syncDurationMs: number): void; +} + +/** + * Computes the number of encoded bytes for a write chunk: + * - Uint8Array (incl. Buffer): byteLength + * - string: Buffer.byteLength using the supplied encoding (defaults to UTF-8) + */ +function encodedByteLength( + chunk: Uint8Array | string, + encodingOrCallback: StdoutEncodingArg, +): number { + if (typeof chunk === 'string') { + const encoding = + typeof encodingOrCallback === 'string' ? encodingOrCallback : undefined; + return Buffer.byteLength(chunk, encoding); + } + return chunk.byteLength; +} + +/** + * Wraps an underlying stdout write with byte counting and synchronous-duration + * measurement. Exported from this module so behavioral tests can inject a + * deterministic underlying write to verify byte counting, backpressure, + * callback forwarding, and error propagation without touching the real + * process.stdout. + * + * The observer is called directly after the underlying write returns, with no + * try/catch — internal/programming errors fail fast (D8). If the underlying + * write throws synchronously, no observer sample is produced. + */ +export function createObservedStdoutWrite( + underlyingWrite: StdoutWrite, + observer: StdoutWriteObserver, +): StdoutWrite { + return function observedStdoutWrite(...args: StdoutWriteArgs): boolean { + const encodedBytes = encodedByteLength(args[0], args[1]); + const start = performance.now(); + const ok = Reflect.apply(underlyingWrite, undefined, args) as boolean; + const syncDurationMs = performance.now() - start; + observer.onWrite(encodedBytes, syncDurationMs); + return ok; + } as StdoutWrite; +} + // Capture the original stdout and stderr write methods before any monkey patching occurs. const originalStdoutWrite = process.stdout.write.bind(process.stdout); const originalStderrWrite = process.stderr.write.bind(process.stderr); @@ -37,9 +118,16 @@ const handleStderrError = (err: NodeJS.ErrnoException) => { * Writes to the real stdout, bypassing any monkey patching on process.stdout.write. */ export function writeToStdout( - ...args: Parameters -): boolean { - return originalStdoutWrite(...args); + chunk: StdioWriteChunk, + callback?: StdioWriteCallback, +): boolean; +export function writeToStdout( + chunk: StdioWriteChunk, + encoding?: BufferEncoding, + callback?: StdioWriteCallback, +): boolean; +export function writeToStdout(...args: StdoutWriteArgs): boolean { + return Reflect.apply(originalStdoutWrite, undefined, args) as boolean; } /** @@ -108,7 +196,7 @@ export function patchStdio(): () => void { * Also adds error event handlers to prevent EPIPE crashes when output is piped * to a process that exits early. */ -export function createInkStdio() { +export function createInkStdio(observer?: StdoutWriteObserver) { // Remove any existing handlers to avoid duplicates, then re-add. // Handlers are defined at module scope so the same references are used. process.stdout.removeListener('error', handleStdoutError); @@ -117,10 +205,17 @@ export function createInkStdio() { process.stdout.on('error', handleStdoutError); process.stderr.on('error', handleStderrError); + // When no observer is present, the write delegate is writeToStdout itself — + // preserving the exact function identity and behaviour of today. When an + // observer is supplied, a single observed wrapper is created and reused. + const stdoutWrite = observer + ? createObservedStdoutWrite(writeToStdout, observer) + : writeToStdout; + const inkStdout = new Proxy(process.stdout, { get(target, prop, receiver) { if (prop === 'write') { - return writeToStdout; + return stdoutWrite; } const value = Reflect.get(target, prop, receiver); if (typeof value === 'function') { diff --git a/packages/providers/src/__tests__/attemptRecorder.perf.behavior.test.ts b/packages/providers/src/__tests__/attemptRecorder.perf.behavior.test.ts new file mode 100644 index 0000000000..b519299246 --- /dev/null +++ b/packages/providers/src/__tests__/attemptRecorder.perf.behavior.test.ts @@ -0,0 +1,341 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Behavioral tests proving the AttemptRecorder invokes the PerfPhaseObserver + * at exact lifecycle boundaries (P07, EVIDENCE-AC5). + * + * Real AttemptRecorder lifecycle (onAttemptStart/onAttemptEnd), real + * PerfPhaseObserver seam. No mock theater. + * + * Proves: + * - Provider attempt start/end boundaries from real lifecycle + * - Retries → multiple attempt intervals + * - Consumer abort → end boundary with 'aborted' status + * - D8: observer invoked outside try/catch (fail-fast on observer error) + * - SDK-disabled mode still notifies (observer invoked regardless of SDK state) + * - Default-off: null observer → no notification + * - Attempt boundary timestamps are the real monotonic start/end + */ + +import { describe, it, expect, beforeEach, afterEach, vi } from 'bun:test'; +import * as sdk from '@vybestack/llxprt-code-telemetry/telemetry/sdk.js'; +import { AttemptRecorder } from '../logging/attemptRecorder.js'; +import { + setPerfPhaseObserver, + getPerfPhaseObserver, + type PerfPhaseObserver, + type PerfProviderAttemptStartInfo, + type PerfProviderAttemptEndInfo, +} from '@vybestack/llxprt-code-telemetry/perf/perfPhaseObserver.js'; + +function createRecorder(wrapperOwned = true): AttemptRecorder { + return new AttemptRecorder({ + providerName: 'test-provider', + defaultModelName: 'test-model', + config: undefined, + logicalRequestId: 'req-perf-test', + wrapperOwned, + }); +} + +function capturingObserver(): { + observer: PerfPhaseObserver; + starts: PerfProviderAttemptStartInfo[]; + ends: PerfProviderAttemptEndInfo[]; +} { + const starts: PerfProviderAttemptStartInfo[] = []; + const ends: PerfProviderAttemptEndInfo[] = []; + const observer: PerfPhaseObserver = { + onProviderAttemptStart: (info) => starts.push(info), + onProviderAttemptEnd: (info) => ends.push(info), + onToolCallCompleted: () => undefined, + }; + return { observer, starts, ends }; +} + +describe('AttemptRecorder perf phase observer (P07)', () => { + beforeEach(() => { + vi.spyOn(sdk, 'isTelemetrySdkInitialized').mockReturnValue(false); + setPerfPhaseObserver(null); + }); + + afterEach(() => { + vi.restoreAllMocks(); + setPerfPhaseObserver(null); + }); + + it('notifies observer on attempt start and end (wrapperOwned, success)', () => { + const { observer, starts, ends } = capturingObserver(); + setPerfPhaseObserver(observer); + + const recorder = createRecorder(true); + recorder.ensureAttemptStarted(); + recorder.recordTokenBearingChunk( + recorder.getCurrentAttemptId()!, + undefined, + 'hello', + ); + recorder.finalizeAttempt('success', 'test-model'); + + expect(starts).toHaveLength(1); + expect(ends).toHaveLength(1); + expect(starts[0].attemptId).toBe(ends[0].attemptId); + expect(ends[0].status).toBe('success'); + expect(ends[0].endMs).toBeGreaterThanOrEqual(starts[0].startMs); + }); + + it('notifies on consumer abort with "aborted" status', () => { + const { observer, ends } = capturingObserver(); + setPerfPhaseObserver(observer); + + const recorder = createRecorder(true); + recorder.ensureAttemptStarted(); + recorder.finalizeAttempt('aborted', 'test-model'); + + expect(ends).toHaveLength(1); + expect(ends[0].status).toBe('aborted'); + }); + + it('notifies on error with "error" status', () => { + const { observer, ends } = capturingObserver(); + setPerfPhaseObserver(observer); + + const recorder = createRecorder(true); + recorder.ensureAttemptStarted(); + recorder.finalizeAttempt('error', 'test-model', undefined, 'boom'); + + expect(ends).toHaveLength(1); + expect(ends[0].status).toBe('error'); + }); + + it('notifies once per attempt (dedup by attemptId via hasEmittedTerminal)', () => { + const { observer, ends } = capturingObserver(); + setPerfPhaseObserver(observer); + + const recorder = createRecorder(true); + recorder.ensureAttemptStarted(); + recorder.finalizeAttempt('success', 'test-model'); + // Second finalize is a no-op (hasEmittedTerminal guard). + recorder.finalizeAttempt('error', 'test-model'); + + expect(ends).toHaveLength(1); + }); + + it('notifies for each retry attempt (external lifecycle owner)', () => { + const { observer, starts, ends } = capturingObserver(); + setPerfPhaseObserver(observer); + + const recorder = createRecorder(false); + // External lifecycle owner fires start/end per attempt. + recorder.onAttemptStart({ + requestStartMs: 1000, + attemptId: 'a1', + attemptIndex: 0, + }); + recorder.onAttemptEnd({ + attemptId: 'a1', + attemptIndex: 0, + start: 1000, + completionMs: 2000, + firstTokenMs: 1100, + lastTokenMs: 1900, + status: 'error', + providerName: 'test-provider', + modelName: 'test-model', + inputTokens: 100, + outputTokens: 50, + cachedTokens: 0, + thoughtsTokens: 0, + toolTokens: 0, + errorMessage: 'retry me', + }); + + recorder.onAttemptStart({ + requestStartMs: 2100, + attemptId: 'a2', + attemptIndex: 1, + }); + recorder.onAttemptEnd({ + attemptId: 'a2', + attemptIndex: 1, + start: 2100, + completionMs: 3000, + firstTokenMs: 2200, + lastTokenMs: 2900, + status: 'success', + providerName: 'test-provider', + modelName: 'test-model', + inputTokens: 200, + outputTokens: 80, + cachedTokens: 0, + thoughtsTokens: 0, + toolTokens: 0, + }); + + expect(starts).toHaveLength(2); + expect(ends).toHaveLength(2); + expect(ends[0].attemptId).toBe('a1'); + expect(ends[1].attemptId).toBe('a2'); + expect(ends[0].status).toBe('error'); + expect(ends[1].status).toBe('success'); + // Interval boundaries from the real lifecycle start/end. + expect(ends[0].startMs).toBe(1000); + expect(ends[0].endMs).toBe(2000); + expect(ends[1].startMs).toBe(2100); + expect(ends[1].endMs).toBe(3000); + }); + + it('preserves token counts at the boundary', () => { + const { observer, ends } = capturingObserver(); + setPerfPhaseObserver(observer); + + const recorder = createRecorder(false); + recorder.onAttemptStart({ + requestStartMs: 1000, + attemptId: 'tok-1', + attemptIndex: 0, + }); + recorder.onAttemptEnd({ + attemptId: 'tok-1', + attemptIndex: 0, + start: 1000, + completionMs: 2000, + firstTokenMs: null, + lastTokenMs: null, + status: 'success', + providerName: 'test-provider', + modelName: 'test-model', + inputTokens: 500, + outputTokens: 120, + cachedTokens: 10, + thoughtsTokens: 5, + toolTokens: 3, + }); + + expect(ends[0].inputTokens).toBe(500); + expect(ends[0].outputTokens).toBe(120); + // cachedTokens/thoughtsTokens/toolTokens are intentionally not forwarded + // to the perf observer — only input/output totals are tracked. + expect(ends[0]).not.toHaveProperty('cachedTokens'); + }); + + it('SDK-disabled mode still notifies (observer invoked before SDK gate)', () => { + vi.spyOn(sdk, 'isTelemetrySdkInitialized').mockReturnValue(false); + const { observer, ends } = capturingObserver(); + setPerfPhaseObserver(observer); + + const recorder = createRecorder(true); + recorder.ensureAttemptStarted(); + recorder.finalizeAttempt('success', 'test-model'); + + // The observer must fire even when the SDK is disabled. + expect(ends).toHaveLength(1); + }); + + it('default-off: null observer produces no notification and no crash', () => { + setPerfPhaseObserver(null); + expect(getPerfPhaseObserver()).toBeNull(); + + const recorder = createRecorder(true); + recorder.ensureAttemptStarted(); + recorder.finalizeAttempt('success', 'test-model'); + // No crash, no observer to notify. + }); + + it('D8: observer error propagates (fail-fast, not swallowed)', () => { + const throwingObserver: PerfPhaseObserver = { + onProviderAttemptStart: () => undefined, + onProviderAttemptEnd: () => { + throw new Error('observer internal error'); + }, + onToolCallCompleted: () => undefined, + }; + setPerfPhaseObserver(throwingObserver); + + const recorder = createRecorder(true); + recorder.ensureAttemptStarted(); + + // The observer error must propagate, NOT be swallowed by the + // emitAttemptRecord try/catch. + expect(() => recorder.finalizeAttempt('success', 'test-model')).toThrow( + 'observer internal error', + ); + }); + + it('D8: observer error on start propagates (fail-fast)', () => { + const throwingObserver: PerfPhaseObserver = { + onProviderAttemptStart: () => { + throw new Error('observer start error'); + }, + onProviderAttemptEnd: () => undefined, + onToolCallCompleted: () => undefined, + }; + setPerfPhaseObserver(throwingObserver); + + const recorder = createRecorder(true); + expect(() => recorder.ensureAttemptStarted()).toThrow( + 'observer start error', + ); + }); + + it('carries the logicalRequestId as promptId on start and end (D1 correlation)', () => { + const { observer, starts, ends } = capturingObserver(); + setPerfPhaseObserver(observer); + + const recorder = new AttemptRecorder({ + providerName: 'test-provider', + defaultModelName: 'test-model', + config: undefined, + logicalRequestId: 'sess#agentic-loop#logical-req-1', + wrapperOwned: true, + }); + recorder.ensureAttemptStarted(); + recorder.finalizeAttempt('success', 'test-model'); + + expect(starts).toHaveLength(1); + expect(ends).toHaveLength(1); + expect(starts[0].promptId).toBe('sess#agentic-loop#logical-req-1'); + expect(ends[0].promptId).toBe('sess#agentic-loop#logical-req-1'); + }); + + it('carries the logicalRequestId as promptId for external lifecycle owner', () => { + const { observer, starts, ends } = capturingObserver(); + setPerfPhaseObserver(observer); + + const recorder = new AttemptRecorder({ + providerName: 'test-provider', + defaultModelName: 'test-model', + config: undefined, + logicalRequestId: 'sess#agentic-loop#ext-req', + wrapperOwned: false, + }); + recorder.onAttemptStart({ + requestStartMs: 0, + attemptId: 'ext-a1', + attemptIndex: 0, + }); + recorder.onAttemptEnd({ + attemptId: 'ext-a1', + attemptIndex: 0, + start: 0, + completionMs: 100, + firstTokenMs: null, + lastTokenMs: null, + status: 'success', + providerName: 'test-provider', + modelName: 'test-model', + inputTokens: 0, + outputTokens: 0, + cachedTokens: 0, + thoughtsTokens: 0, + toolTokens: 0, + }); + + expect(starts[0].promptId).toBe('sess#agentic-loop#ext-req'); + expect(ends[0].promptId).toBe('sess#agentic-loop#ext-req'); + }); +}); diff --git a/packages/providers/src/logging/attemptRecorder.ts b/packages/providers/src/logging/attemptRecorder.ts index 4e49fc3eac..dee9c3b0b7 100644 --- a/packages/providers/src/logging/attemptRecorder.ts +++ b/packages/providers/src/logging/attemptRecorder.ts @@ -13,6 +13,7 @@ import type { Config } from '@vybestack/llxprt-code-core/config/config.js'; import { logApiError } from '@vybestack/llxprt-code-core/telemetry/loggers.js'; import { ApiErrorEvent } from '@vybestack/llxprt-code-core/telemetry/types.js'; import { DebugLogger } from '@vybestack/llxprt-code-core/debug/index.js'; +import { getPerfPhaseObserver } from '@vybestack/llxprt-code-core/perf/perfPhaseObserver.js'; import type { UsageStats } from '@vybestack/llxprt-code-core/services/history/IContent.js'; import { type ResponseTokenCounts, @@ -99,7 +100,11 @@ export interface AttemptRecorderOptions { * - Stable nonempty attempt IDs provided by the lifecycle owner * - Monotonic timestamps (performance.now-based) * - No phantom attempt when an external owner is present - * - Fail-open: listener/export errors never propagate into the stream path + * - Fail-open: emit/export errors (emitAttemptRecord, emitMetricsTelemetry, + * logApiError) never propagate into the stream path. Perf phase observer + * invocations are the exception: they propagate fail-fast per D8 so a + * programming error in the observer is surfaced immediately rather than + * silently corrupting telemetry. */ export class AttemptRecorder implements AttemptLifecycleObserver { private readonly logger = new DebugLogger('llxprt:attempt:recorder'); @@ -152,6 +157,18 @@ export class AttemptRecorder implements AttemptLifecycleObserver { }); this.attemptOrder.push(attemptId); this.attemptCounter++; + + // Perf phase observer (P07): notify at the exact attempt-start boundary. + // Default-off: null observer short-circuits. Invoked directly (no + // try/catch) so internal errors propagate fail-fast (D8). + const perfObserver = getPerfPhaseObserver(); + if (perfObserver !== null) { + perfObserver.onProviderAttemptStart({ + attemptId, + promptId: this.logicalRequestId, + startMs: requestStartMs, + }); + } } /** @@ -234,6 +251,27 @@ export class AttemptRecorder implements AttemptLifecycleObserver { // Prune terminal attempts to prevent unbounded memory growth in // long-lived recorder instances. this.pruneTerminalAttempts(); + + // Perf phase observer (P07): notify at the exact attempt-end boundary. + // Invoked AFTER the try/catch above so an internal observer/programming + // error propagates fail-fast (D8) rather than being swallowed by the + // emitAttemptRecord catch. The observer fires before any SDK/export + // gate so SDK-disabled mode still notifies. Default-off: null observer + // short-circuits. + const perfObserver = getPerfPhaseObserver(); + if (perfObserver !== null) { + const completionMs = + sanitizeTimestamp(info.completionMs) ?? performance.now(); + perfObserver.onProviderAttemptEnd({ + attemptId, + promptId: this.logicalRequestId, + startMs: attempt.requestStartMs, + endMs: completionMs, + status: info.status, + inputTokens: info.inputTokens, + outputTokens: info.outputTokens, + }); + } } /** Maximum number of terminal attempts to retain before pruning. */ diff --git a/packages/telemetry/index.ts b/packages/telemetry/index.ts index bdd5ddbab2..58eec73ef0 100644 --- a/packages/telemetry/index.ts +++ b/packages/telemetry/index.ts @@ -10,6 +10,9 @@ export * from './src/debug/index.js'; // Telemetry module export * from './src/telemetry/index.js'; +// Perf telemetry module (issue #3167) +export * from './src/perf/index.js'; + // Utilities export { safeJsonStringify } from './src/utils/safeJsonStringify.js'; export { LLXPRT_DIR } from './src/utils/paths.js'; diff --git a/packages/telemetry/package.json b/packages/telemetry/package.json index 90e48e6317..770f969c49 100644 --- a/packages/telemetry/package.json +++ b/packages/telemetry/package.json @@ -88,6 +88,26 @@ "bun": "./src/telemetry/sessionMetricsAggregator.ts", "import": "./dist/src/telemetry/sessionMetricsAggregator.js" }, + "./perf/index.js": { + "bun": "./src/perf/index.ts", + "import": "./dist/src/perf/index.js" + }, + "./perf/perfRecords.js": { + "bun": "./src/perf/perfRecords.ts", + "import": "./dist/src/perf/perfRecords.js" + }, + "./perf/perfPhaseObserver.js": { + "bun": "./src/perf/perfPhaseObserver.ts", + "import": "./dist/src/perf/perfPhaseObserver.js" + }, + "./perf/perfSlopeBridge.js": { + "bun": "./src/perf/perfSlopeBridge.ts", + "import": "./dist/src/perf/perfSlopeBridge.js" + }, + "./telemetry/intervalUnion.js": { + "bun": "./src/telemetry/intervalUnion.ts", + "import": "./dist/src/telemetry/intervalUnion.js" + }, "./telemetry/tool-call-decision.js": { "bun": "./src/telemetry/tool-call-decision.ts", "import": "./dist/src/telemetry/tool-call-decision.js" diff --git a/packages/telemetry/src/perf/PerfSink.ts b/packages/telemetry/src/perf/PerfSink.ts new file mode 100644 index 0000000000..8bc3de88d3 --- /dev/null +++ b/packages/telemetry/src/perf/PerfSink.ts @@ -0,0 +1,372 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * PerfSink — serialized no-drop perf telemetry writer (P04B, D4). + * + * A constructible (non-singleton) writer that does NOT inherit FileOutput. + * Uses a serialized promise chain: one record per operation, own back-pressure, + * no bounded queue, no drop counter. + * + * File layout: one exclusive-created 0600 file per run UUID per UTC record + * day: `perf-YYYYMMDD-.jsonl`. The day comes from each record's `ts` + * and rolls on the next serialized record whose day differs. Empty sink + * creates no file. Dispose drains all accepted writes. + * + * Error policy (D8): schema/programming/serialization errors fail fast to the + * caller (synchronous throw before queueing). Only filesystem + * create/append/close errors fail-open and are rate-limited. + * + * Decision: FileOutput is left untouched. The only overlap with FileOutput is + * directory-creation and file-append, which are 2-line primitives too trivial + * to extract. FileOutput does not use exclusive-open (`wx`), has a different + * naming scheme, and carries a bounded/drop queue that PerfSink must not + * inherit (D4). Extraction would broaden scope and risk the singleton/debug + * behavior — not warranted. + */ + +import { promises as fs } from 'node:fs'; +import { join } from 'node:path'; +import { PerfRecordSchema } from './perfRecords.js'; +import { requireValidRunUuid } from './perfArtifacts.js'; +import type { PerfRetention } from './retention.js'; + +// --------------------------------------------------------------------------- +// Filesystem port (D6 — package-private for deterministic fault injection) +// --------------------------------------------------------------------------- + +/** + * Narrow filesystem port used by PerfSink. The default implementation uses + * real `node:fs/promises`; tests inject {@link FaultInjectingPerfFilesystem} + * or a custom implementation to produce deterministic EACCES/EROFS/ENOSPC + * failures at the append boundary without filling a disk or relying on chmod. + */ +export interface PerfSinkFilesystem { + ensureDir(dir: string): Promise; + openExclusive(path: string, mode: number): Promise; + appendFile(path: string, data: string, mode: number): Promise; +} + +/** Default filesystem port using real `node:fs/promises`. */ +class RealPerfFilesystem implements PerfSinkFilesystem { + async ensureDir(dir: string): Promise { + try { + await fs.access(dir); + } catch { + await fs.mkdir(dir, { recursive: true, mode: 0o700 }); + } + } + + async openExclusive(filePath: string, mode: number): Promise { + const handle = await fs.open(filePath, 'wx', mode); + await handle.close(); + } + + async appendFile( + filePath: string, + data: string, + mode: number, + ): Promise { + await fs.appendFile(filePath, data, { encoding: 'utf8', mode }); + } +} + +/** + * Deterministic fault-injecting filesystem port. Fails the configured method + * with the given errno code on every call, delegating all other methods to the + * real implementation. Used by fault-injection tests (D6) — never fills a real + * disk or relies on chmod. + */ +export class FaultInjectingPerfFilesystem implements PerfSinkFilesystem { + private readonly real = new RealPerfFilesystem(); + + constructor( + private readonly fault: { + readonly failMethod: 'appendFile' | 'openExclusive' | 'ensureDir'; + readonly code: 'EACCES' | 'EROFS' | 'ENOSPC'; + }, + ) {} + + async ensureDir(dir: string): Promise { + if (this.fault.failMethod === 'ensureDir') { + throw this.makeError(); + } + await this.real.ensureDir(dir); + } + + async openExclusive(filePath: string, mode: number): Promise { + if (this.fault.failMethod === 'openExclusive') { + throw this.makeError(); + } + await this.real.openExclusive(filePath, mode); + } + + async appendFile( + filePath: string, + data: string, + mode: number, + ): Promise { + if (this.fault.failMethod === 'appendFile') { + throw this.makeError(); + } + await this.real.appendFile(filePath, data, mode); + } + + private makeError(): NodeJS.ErrnoException { + const err = new Error( + `fault-injected ${this.fault.code}`, + ) as NodeJS.ErrnoException; + err.code = this.fault.code; + return err; + } +} + +// --------------------------------------------------------------------------- +// Helpers +// --------------------------------------------------------------------------- + +/** + * Extracts a UTC YYYYMMDD day key from an ISO 8601 timestamp string. + */ +function utcDayKey(ts: string): string { + const date = new Date(ts); + const year = date.getUTCFullYear(); + const month = String(date.getUTCMonth() + 1).padStart(2, '0'); + const day = String(date.getUTCDate()).padStart(2, '0'); + return `${year}${month}${day}`; +} + +/** + * Determines whether an error carries a Node.js errno code, indicating a + * filesystem persistence failure (fail-open). Errors without an errno code + * are programming errors and must propagate (fail fast). + */ +function isErrnoError(err: unknown): boolean { + if (!(err instanceof Error)) return false; + return typeof (err as NodeJS.ErrnoException).code === 'string'; +} + +/** + * Extracts the errno code from an error, or 'UNKNOWN' if none. Shared by the + * self-health surface and the diagnostic message so they cannot drift. + */ +function extractErrnoCode(err: unknown): string { + return err instanceof Error + ? ((err as NodeJS.ErrnoException).code ?? 'UNKNOWN') + : 'UNKNOWN'; +} + +const DEFAULT_DIAG_RATE_LIMIT_MS = 60_000; + +function defaultDiagnostic(message: string): void { + process.stderr.write(`${message}\n`); +} + +// --------------------------------------------------------------------------- +// PerfSink +// --------------------------------------------------------------------------- + +export interface PerfSinkOptions { + readonly dir: string; + readonly runUuid: string; + readonly fs?: PerfSinkFilesystem; + readonly retention?: PerfRetention; + readonly diagRateLimitMs?: number; + readonly onDiagnostic?: (message: string) => void; +} + +export class PerfSink { + private readonly sinkDir: string; + private readonly runUuid: string; + private readonly fsPort: PerfSinkFilesystem; + private readonly retention: PerfRetention | null; + private readonly diagRateLimitMs: number; + private readonly onDiagnostic: (message: string) => void; + + private fileDayKey: string | null = null; + private currentPath: string | null = null; + private bytesSinceStat = 0; + private writeChain: Promise = Promise.resolve(); + private lastDiagMs = 0; + private disposed = false; + /** Paths this instance successfully created with an exclusive open. */ + private readonly createdPaths = new Set(); + // P11 self-health: the errno code of the latest filesystem write failure + // in THIS process, or null if no write has failed. Narrow read-only state + // for the inspect/report self-health surface — NOT persisted. + private latestWriteErrorCode: string | null = null; + + constructor(options: PerfSinkOptions) { + this.sinkDir = options.dir; + this.runUuid = requireValidRunUuid(options.runUuid); + this.fsPort = options.fs ?? new RealPerfFilesystem(); + this.retention = options.retention ?? null; + this.diagRateLimitMs = + options.diagRateLimitMs ?? DEFAULT_DIAG_RATE_LIMIT_MS; + this.onDiagnostic = options.onDiagnostic ?? defaultDiagnostic; + } + + /** + * Starts the optional retention owner (creates the per-run claim file and + * the one owned maintenance interval). Must be awaited before writing if a + * retention owner was provided, so the runtime (P12) can await it before + * installing observers. If no retention owner was provided, this is a no-op. + * + * An empty started sink creates ONLY its claim — no perf JSONL. + */ + async start(): Promise { + if (this.retention !== null) { + await this.retention.start(); + } + } + + /** + * Validates and serializes the record through the schema, then queues a + * serialized filesystem append. + * + * Schema/programming/serialization errors throw synchronously (fail fast). + * Filesystem errors are caught and rate-limited (fail-open) — the returned + * promise always resolves for filesystem errors. + * + * After dispose, returns the current chain without queueing a new write. + */ + write(record: unknown): Promise { + if (this.disposed) { + return this.writeChain; + } + + // Fail fast: validate + serialize BEFORE queueing. These throw + // synchronously to the caller — they are NOT filesystem errors. + const validated = PerfRecordSchema.parse(record); + const payload = JSON.stringify(validated) + '\n'; + const dayKey = utcDayKey(validated.ts); + + // Serialize the append through the no-drop promise chain. + this.writeChain = this.writeChain.then(async () => { + try { + await this.appendPayload(payload, dayKey); + } catch (err) { + if (isErrnoError(err)) { + this.emitDiagnostic(err); + } else { + throw err; + } + } + }); + + return this.writeChain; + } + + /** + * Blocks further writes deterministically, drains all accepted writes, + * then stops maintenance and removes the claim cleanly (if a retention + * owner was provided). Always runs BOTH the write-chain drain and the + * retention disposal, aggregating internal failures via AggregateError so + * an internal write rejection cannot skip claim/timer cleanup. + */ + async dispose(): Promise { + this.disposed = true; + const errors: unknown[] = []; + try { + await this.writeChain; + } catch (err) { + errors.push(err); + } + if (this.retention !== null) { + try { + await this.retention.dispose(); + } catch (err) { + errors.push(err); + } + } + if (errors.length === 1) throw errors[0]!; + if (errors.length > 1) { + throw new AggregateError(errors, 'PerfSink disposal'); + } + } + + get byteCount(): number { + return this.bytesSinceStat; + } + + /** + * P11 self-health: the errno code of the latest filesystem write failure in + * THIS process (e.g. 'EACCES'), or null if no write has failed. Used by the + * inspect/report self-health surface to surface the current-process write + * health. Not persisted. + */ + get lastWriteErrorCode(): string | null { + return this.latestWriteErrorCode; + } + + // ----------------------------------------------------------------------- + // Private + // ----------------------------------------------------------------------- + + private async appendPayload(payload: string, dayKey: string): Promise { + if (dayKey !== this.fileDayKey || this.currentPath === null) { + await this.rollToNewFile(dayKey); + } + const target = this.currentPath; + if (target === null) { + throw new Error( + 'PerfSink internal error: currentPath is null after roll', + ); + } + await this.fsPort.appendFile(target, payload, 0o600); + this.bytesSinceStat += Buffer.byteLength(payload); + } + + private async rollToNewFile(dayKey: string): Promise { + const name = `perf-${dayKey}-${this.runUuid}.jsonl`; + const filePath = join(this.sinkDir, name); + + // Filesystem operations only — any error here is a filesystem error. + await this.fsPort.ensureDir(this.sinkDir); + try { + await this.fsPort.openExclusive(filePath, 0o600); + this.createdPaths.add(filePath); + } catch (err) { + // Non-monotonic timestamps can revisit an earlier UTC-day file. Re-adopt + // only paths this sink previously created; preserve exclusive creation + // for every unknown path. + if ( + !(err instanceof Error) || + (err as NodeJS.ErrnoException).code !== 'EEXIST' || + !this.createdPaths.has(filePath) + ) { + throw err; + } + } + + // State advances ONLY after successful exclusive open (or re-adoption of + // a path this run already owns). + this.currentPath = filePath; + this.bytesSinceStat = 0; + this.fileDayKey = dayKey; + + // Roll boundary triggers rate-limited maintenance so a 24/7 process + // that never restarts still bounds growth. maybeMaintain handles its + // own filesystem errors (fail-open); only internal errors propagate. + if (this.retention !== null) { + await this.retention.maybeMaintain(Date.now()); + } + } + + private emitDiagnostic(err: unknown): void { + const now = Date.now(); + // P11 self-health: always record the latest error code even if the + // diagnostic message is rate-limited. The code is surfaced by the + // inspect/report self-health surface. + const code = extractErrnoCode(err); + this.latestWriteErrorCode = code; + if (now - this.lastDiagMs < this.diagRateLimitMs) { + return; + } + this.lastDiagMs = now; + this.onDiagnostic(`perf telemetry write failed: ${code}`); + } +} diff --git a/packages/telemetry/src/perf/index.ts b/packages/telemetry/src/perf/index.ts new file mode 100644 index 0000000000..15c5693d29 --- /dev/null +++ b/packages/telemetry/src/perf/index.ts @@ -0,0 +1,118 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +export { + PERF_SCHEMA_VERSION, + PERF_RECORD_TYPE_OPERATION, + PERF_RECORD_TYPE_MEMORY_SAMPLE, + PERF_TERMINAL_STATUSES, + PerfOperationRecordSchema, + PerfMemorySampleRecordSchema, + PerfRecordSchema, + deriveOperationId, + joinKeyFromPromptId, + classifyPerfLine, + parsePerfRecord, + readPerfRecords, + streamPerfRecords, +} from './perfRecords.js'; +export type { + PerfTerminalStatus, + PerfOperationRecord, + PerfMemorySampleRecord, + PerfRecord, + PerfLineClassification, + PerfStreamEntry, + PerfReaderCounts, + PerfReaderResult, +} from './perfRecords.js'; +export { PerfSink } from './PerfSink.js'; +export type { PerfSinkOptions, PerfSinkFilesystem } from './PerfSink.js'; + +// Retention + claim lifecycle (P08, D3/D5/D6). +// The filesystem and scheduler port types are exported for the CLI composition +// root. The deterministic fault injector remains package-private to tests. +export { + PERF_MAX_BYTES, + PERF_MAX_FILES, + PERF_MAINTENANCE_INTERVAL_MS, + PERF_CLAIM_LEASE_MS, + PERF_DIAG_RATE_LIMIT_MS, + PerfRetention, +} from './retention.js'; +export type { PerfRetentionOptions } from './retention.js'; +export type { PerfRetentionFilesystem } from './retention.js'; +export type { PerfScheduler, PerfTimerHandle } from './retention.js'; + +// Perf phase observer seam (P07). Default-off module-level subscription. +export { + setPerfPhaseObserver, + getPerfPhaseObserver, +} from './perfPhaseObserver.js'; +export type { + PerfPhaseObserver, + PerfProviderAttemptStartInfo, + PerfProviderAttemptEndInfo, + PerfToolCallCompletedInfo, +} from './perfPhaseObserver.js'; + +// P11: directory consumer, report, inspect, delete, shared artifacts. +export { streamPerfDirectory, consumePerfDirectory } from './perfConsumer.js'; +export type { + PerfConsumerEntry, + PerfConsumerCounts, + PerfConsumerResult, +} from './perfConsumer.js'; +export { + buildReport, + assembleReport, + formatReport, + joinTokenRowsByOperation, +} from './perfReport.js'; +export type { + PerfTokenUsageRow, + ReportDimensions, + ReportBuildIdentity, + ReportFileMemorySlopes, + ReportGroup, + BaselineComparison, + ReportGroupWithBaseline, + ReportSelfHealth, + ReportResult, + P50MetricKey, +} from './perfReport.js'; +export { perfInspect, formatInspect } from './perfInspect.js'; +export type { PerfInspectResult, PerfSkippedCounts } from './perfInspect.js'; +export { perfDelete, formatDeleteResult } from './perfDelete.js'; +export type { + PerfDeleteOptions, + PerfDeleteResult, + PerfDeleteFilesystem, +} from './perfDelete.js'; + +// D1 token-usage streaming and aggregation stay package-private. The report is +// the production API; same-package behavioral tests import the reader directly. +// +// Artifact parsing/protection functions are package-private (used only by +// PerfRetention and perfDelete within this package). Only the types that appear +// in public-facing signatures are exported. +export type { + ParsedPerfFilename, + ParsedClaimFilename, + ParsedArtifactName, + ClaimProtectionInput, +} from './perfArtifacts.js'; + +// P10/P11: canonical read-time memory slope derivation and types (owned below +// the CLI layer so the report and the live view share one algorithm). +export { + derivePerOperationMemorySlope, + derivePerMinuteMemorySlope, +} from './perfSlopeBridge.js'; +export type { + PerOperationMemorySlope, + PerMinuteMemorySlope, +} from './perfSlopeBridge.js'; diff --git a/packages/telemetry/src/perf/perfArtifacts.ts b/packages/telemetry/src/perf/perfArtifacts.ts new file mode 100644 index 0000000000..b44e73483b --- /dev/null +++ b/packages/telemetry/src/perf/perfArtifacts.ts @@ -0,0 +1,300 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Shared artifact parsing/protection logic for the perf directory (P11, F). + * + * Extracted from {@link PerfRetention} so that retention eviction and + * {@link perfDelete} share the same primitives: what constitutes an owned + * artifact, how to parse day-key / run-UUID from a filename, and how to + * evaluate live-writer and claim freshness. The two operations apply these + * primitives with deliberately different policies: automatic retention + * protects a JSONL only as a live writer (so a 24×7 run can converge), while + * explicit delete additionally protects any JSONL whose run holds a fresh + * claim (see {@link isPerfJsonlProtected}). + * + * File-name conventions (P04B): + * - Perf JSONL: `perf-YYYYMMDD-.jsonl` + * - Claim: `.claim` + * + * Canonical run IDs (F): production run UUIDs are standard `crypto.randomUUID()` + * values. Internal constructor boundaries ({@link PerfRetention}, + * {@link PerfSink}) validate the UUID before joining it into a filesystem + * path, rejecting separators, traversal sequences, and malformed IDs + * (fail-fast). External filename parsing ({@link extractRunUuid}) remains + * tolerant — files on disk are external input and may carry any string. + */ + +/** Regex matching perf JSONL file names: perf-YYYYMMDD-uuid.jsonl */ +export const PERF_FILE_RE = /^perf-(\d{8})-(.+)\.jsonl$/; + +/** Regex matching claim file names: uuid.claim */ +export const CLAIM_FILE_RE = /^(.+)\.claim$/; + +/** + * Path-safe run ID validation (internal boundary, fail-fast). + * + * A run ID is path-safe when it is non-empty, contains no path separators + * (`/` or `\`), no traversal sequence (`..`), no C0 control characters + * (0x00–0x1F), no DEL (0x7F), and no ASCII whitespace. Canonical production + * run UUIDs (standard `crypto.randomUUID()` output) always satisfy this; the + * check is deliberately broader than strict UUID format so it validates the + * actual safety concern (path injection) without rejecting legitimate IDs. + * + * Implemented with explicit char-code tests (not a control-character regex) + * so it is lint-clean while still rejecting control characters per the + * documented contract. + */ +function isRunIdCharUnsafe(code: number): boolean { + if (code === 0x2f || code === 0x5c) return true; // '/' and '\' + if (code <= 0x20 || code === 0x7f) return true; // C0 controls, space, DEL + return false; +} + +function hasUnsafeRunIdChar(value: string): boolean { + for (let i = 0; i < value.length; i++) { + if (isRunIdCharUnsafe(value.charCodeAt(i))) return true; + } + return false; +} + +// --------------------------------------------------------------------------- +// Canonical run ID validation (F — internal boundary, fail-fast) +// --------------------------------------------------------------------------- + +/** + * Returns true if the value is a path-safe run ID (no separators, traversal + * sequences, null bytes, or control characters). Canonical production run + * UUIDs (`crypto.randomUUID()`) always satisfy this. Used at INTERNAL + * constructor boundaries to prevent path injection. + */ +export function isValidRunUuid(value: string): boolean { + if (value.length === 0) return false; + if (value.includes('..')) return false; + return !hasUnsafeRunIdChar(value); +} + +/** + * Validates a run ID at an internal constructor boundary, throwing a + * TypeError if it is not path-safe. Call this BEFORE joining the ID into a + * filesystem path so a programming error (e.g. a path separator in the ID) + * cannot cause directory traversal. + * + * This is fail-fast for internal/programming errors — it does NOT affect + * external-input tolerance (files on disk are parsed by + * {@link extractRunUuid}, which is tolerant). + */ +export function requireValidRunUuid(value: string): string { + if (!isValidRunUuid(value)) { + throw new TypeError( + `Invalid run UUID: contains path separators, traversal, or control characters (got ${JSON.stringify(value)})`, + ); + } + return value; +} + +// --------------------------------------------------------------------------- +// Shared filename parser (F — one strict parser used everywhere) +// --------------------------------------------------------------------------- + +/** A parsed perf JSONL filename. */ +export interface ParsedPerfFilename { + readonly kind: 'perf'; + readonly dayKey: string; + readonly runUuid: string; +} + +/** A parsed claim filename. */ +export interface ParsedClaimFilename { + readonly kind: 'claim'; + readonly runUuid: string; +} + +/** A parsed owned-artifact filename (perf JSONL or claim). */ +export type ParsedArtifactName = ParsedPerfFilename | ParsedClaimFilename; + +/** + * The single shared parser for perf JSONL and claim filenames. Used by + * extraction/counting/retention/delete so their algorithms cannot drift. + * + * Returns `null` for any name that does not match the expected pattern — + * this is the external-input tolerance boundary (files on disk are external). + */ +export function parseArtifactName(name: string): ParsedArtifactName | null { + const perfMatch = name.match(PERF_FILE_RE); + if (perfMatch) { + return { kind: 'perf', dayKey: perfMatch[1], runUuid: perfMatch[2] }; + } + const claimMatch = name.match(CLAIM_FILE_RE); + if (claimMatch) { + return { kind: 'claim', runUuid: claimMatch[1] }; + } + return null; +} + +// --------------------------------------------------------------------------- +// Owned-artifact predicates +// --------------------------------------------------------------------------- + +/** + * Returns true if the name matches a perf JSONL file (`perf-YYYYMMDD-*.jsonl`). + */ +export function isPerfJsonl(name: string): boolean { + return PERF_FILE_RE.test(name); +} + +/** + * Returns true if the name matches a claim file (`*.claim`). + * + * Uses {@link CLAIM_FILE_RE} (the same regex as {@link parseArtifactName}) so + * `isClaimFile` and `parseArtifactName` can never disagree on edge cases like + * `.claim` (no run-UUID prefix), matching how {@link isPerfJsonl} already + * delegates to {@link PERF_FILE_RE}. + */ +export function isClaimFile(name: string): boolean { + return CLAIM_FILE_RE.test(name); +} + +/** + * Returns true if the name is an owned perf artifact (JSONL or claim). + * This is the single source of truth shared by retention and delete. + * Unrelated files in the dedicated directory are never counted/deleted. + */ +export function isOwnedArtifact(name: string): boolean { + return isPerfJsonl(name) || isClaimFile(name); +} + +/** + * Extracts the UTC YYYYMMDD day key from a perf JSONL filename, or null if + * the name does not match the expected pattern. + */ +export function parseDayKeyFromName(name: string): string | null { + const parsed = parseArtifactName(name); + return parsed?.kind === 'perf' ? parsed.dayKey : null; +} + +/** + * Extracts the run UUID from a perf JSONL or claim filename, or null if the + * name does not match the expected pattern. Tolerant of any string value — + * files on disk are external input. + */ +export function extractRunUuid(name: string): string | null { + const parsed = parseArtifactName(name); + return parsed?.runUuid ?? null; +} + +/** Extracts a UTC YYYYMMDD day key from an epoch-millis timestamp. */ +export function utcDayKey(now: number): string { + const date = new Date(now); + const year = date.getUTCFullYear(); + const month = String(date.getUTCMonth() + 1).padStart(2, '0'); + const day = String(date.getUTCDate()).padStart(2, '0'); + return `${year}${month}${day}`; +} + +/** + * Determines whether a perf JSONL file is protected as a live writer: + * its day-key is the current UTC day AND its mtime is within the maintenance + * interval (or materially in the future). + * + * Shared by retention eviction and delete — both must protect the active + * writer. + */ +export function isLiveWriterFile( + name: string, + mtimeMs: number, + now: number, + maintenanceIntervalMs: number, +): boolean { + const dayKey = parseDayKeyFromName(name); + if (dayKey === null) return false; + if (dayKey !== utcDayKey(now)) return false; + return now - mtimeMs <= maintenanceIntervalMs; +} + +/** + * Determines whether a claim file is non-stale (fresh or future-dated). + * A claim is non-stale while (now - mtime) ≤ claimLeaseMs. A future-dated + * claim (negative delta) is also non-stale until it ages past the lease. + * + * Shared by retention eviction and delete — both must respect live claims. + */ +export function isNonStaleClaim( + mtimeMs: number, + now: number, + claimLeaseMs: number, +): boolean { + return now - mtimeMs <= claimLeaseMs; +} + +// --------------------------------------------------------------------------- +// Claim→run→JSONL protection (used by explicit delete; F) +// --------------------------------------------------------------------------- + +/** A stated claim artifact with its run UUID and mtime for protection logic. */ +export interface ClaimProtectionInput { + readonly runUuid: string | null; + readonly mtimeMs: number; +} + +/** + * Collects canonical run IDs from fresh/future claims. A claim is fresh if + * `now - mtime ≤ claimLeaseMs` (future mtimes are also fresh — negative + * delta). + * + * Used by {@link perfDelete} to protect JSONL whose run holds a fresh claim. + */ +export function collectFreshClaimRunUuids( + claims: readonly ClaimProtectionInput[], + now: number, + claimLeaseMs: number, +): Set { + const uuids = new Set(); + for (const claim of claims) { + if ( + claim.runUuid !== null && + isNonStaleClaim(claim.mtimeMs, now, claimLeaseMs) + ) { + uuids.add(claim.runUuid); + } + } + return uuids; +} + +/** + * Determines whether a perf JSONL file is protected from explicit deletion, + * based on live-writer AND claim→run protection. + * + * Used by {@link perfDelete} (perfDelete.ts). Explicit delete deliberately + * protects a JSONL whose run holds a fresh claim, to avoid unlinking a file + * another active process may still be appending. This is intentionally BROADER + * than automatic retention, which protects JSONL only as a live writer + * ({@link isLiveWriterFile}) so a 24×7 process can converge to the eventual + * byte/file caps (see PerfRetention.isProtected). + * + * A JSONL file is protected if ANY of: + * 1. It is a live writer (today's day-key + mtime within maintenance window). + * 2. Its run UUID has a fresh/future claim (in `protectedRunUuids`). + * + * @param protectedRunUuids Run UUIDs with fresh/future claims (from + * {@link collectFreshClaimRunUuids}). + */ +export function isPerfJsonlProtected( + name: string, + mtimeMs: number, + now: number, + maintenanceIntervalMs: number, + protectedRunUuids: ReadonlySet, +): boolean { + if (isLiveWriterFile(name, mtimeMs, now, maintenanceIntervalMs)) { + return true; + } + const runUuid = extractRunUuid(name); + if (runUuid !== null && protectedRunUuids.has(runUuid)) { + return true; + } + return false; +} diff --git a/packages/telemetry/src/perf/perfConsumer.behavior.test.ts b/packages/telemetry/src/perf/perfConsumer.behavior.test.ts new file mode 100644 index 0000000000..77cda9ff67 --- /dev/null +++ b/packages/telemetry/src/perf/perfConsumer.behavior.test.ts @@ -0,0 +1,369 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import { promises as fs } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { + consumePerfDirectory, + streamFileTolerant, + streamPerfDirectory, +} from './perfConsumer.js'; +import type { PerfOperationRecord, PerfStreamEntry } from './perfRecords.js'; + +// --------------------------------------------------------------------------- +// Fixtures +// --------------------------------------------------------------------------- + +function makeOperation( + overrides: Partial = {}, +): PerfOperationRecord { + return { + schema_version: 1, + record_type: 'operation', + ts: '2026-01-01T00:00:00.000Z', + session_id: 'sess-1', + operation_id: 'op-1', + runtime_id: 'rt-1', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'hash-1', + llxprt_version: '0.10.0', + git_sha: 'abc1234', + runtime: 'cli', + platform: 'darwin', + provider: 'test-provider', + model: 'test-model', + context_tokens: 1000, + output_tokens: 500, + terminal_cols: 80, + terminal_rows: 24, + render_mode: 'ink', + concurrent_instances: 1, + status: 'completed', + client_prepare_ms: 10, + stream_handler_ms: 100, + ink_render_ms: 5, + ink_render_count: 3, + stdout_bytes: 1024, + stdout_write_calls: 5, + stdout_write_sync_ms: 2, + client_finalize_ms: 8, + provider_attempts: 1, + provider_attempt_sum_ms: 200, + provider_union_ms: 200, + tool_calls: 2, + tool_call_sum_ms: 50, + tool_union_ms: 50, + agent_activity_union_ms: 250, + operation_elapsed_ms: 1000, + approval_wait_ms: 0, + unclassified_elapsed_ms: 0, + session_operation_index: 0, + uptime_ms: 5000, + ...overrides, + }; +} + +async function makeTempDir(): Promise { + const dir = join( + tmpdir(), + `perf-test-${Date.now()}-${Math.random().toString(36).slice(2)}`, + ); + await fs.mkdir(dir, { recursive: true }); + return dir; +} + +async function writeJsonl( + dir: string, + name: string, + lines: string[], +): Promise { + const content = lines.join('\n'); + await fs.writeFile( + join(dir, name), + content + (content.endsWith('\n') ? '' : '\n'), + 'utf8', + ); +} + +// --------------------------------------------------------------------------- +// Tests +// --------------------------------------------------------------------------- + +describe('PerfConsumer (P11, AC-9)', () => { + let dir: string; + + beforeEach(async () => { + dir = await makeTempDir(); + }); + + afterEach(async () => { + await fs.rm(dir, { recursive: true, force: true }); + }); + + it('parses valid v1 operation records from sorted perf files', async () => { + const op1 = makeOperation({ + operation_id: 'op-1', + session_operation_index: 0, + }); + const op2 = makeOperation({ + operation_id: 'op-2', + session_operation_index: 1, + }); + await writeJsonl(dir, 'perf-20260101-aaaa1111.jsonl', [ + JSON.stringify(op1), + JSON.stringify(op2), + ]); + + const { entries, counts } = await consumePerfDirectory(dir); + + const okEntries = entries.filter((e) => e.entry.kind === 'ok'); + expect(okEntries).toHaveLength(2); + expect(counts.parsed).toBe(2); + expect(counts.files).toBe(1); + expect(counts.bytes).toBeGreaterThan(0); + }); + + it('includes source file and run UUID identity in entries', async () => { + const op = makeOperation(); + await writeJsonl(dir, 'perf-20260101-deadbeef.jsonl', [JSON.stringify(op)]); + + const { entries } = await consumePerfDirectory(dir); + const okEntry = entries.find((e) => e.entry.kind === 'ok'); + + expect(okEntry).toBeDefined(); + expect(okEntry!.sourceFile).toBe('perf-20260101-deadbeef.jsonl'); + expect(okEntry!.runUuid).toBe('deadbeef'); + }); + + it('reads files in sorted order (file streaming, one at a time)', async () => { + await writeJsonl(dir, 'perf-20260101-zzzz.jsonl', [ + JSON.stringify(makeOperation({ operation_id: 'from-z' })), + ]); + await writeJsonl(dir, 'perf-20260101-aaaa.jsonl', [ + JSON.stringify(makeOperation({ operation_id: 'from-a' })), + ]); + + const order: string[] = []; + for await (const entry of streamPerfDirectory(dir)) { + if ( + entry.entry.kind === 'ok' && + entry.entry.record.record_type === 'operation' + ) { + order.push(entry.sourceFile); + } + } + + // Sorted: aaaa before zzzz + expect(order).toEqual([ + 'perf-20260101-aaaa.jsonl', + 'perf-20260101-zzzz.jsonl', + ]); + }); + + it('missing directory is an empty dataset (not an error)', async () => { + const { entries, counts } = await consumePerfDirectory( + join(dir, 'does-not-exist'), + ); + + expect(entries).toHaveLength(0); + expect(counts.parsed).toBe(0); + expect(counts.files).toBe(0); + expect(counts.bytes).toBe(0); + }); + + it('skips and counts future version (v999) records', async () => { + await writeJsonl(dir, 'perf-20260101-futr.jsonl', [ + JSON.stringify({ + ...makeOperation(), + schema_version: 999, + }), + ]); + + const { counts } = await consumePerfDirectory(dir); + + expect(counts.futureVersion).toBe(1); + expect(counts.parsed).toBe(0); + }); + + it('skips and counts unversioned records (no schema_version or record_type)', async () => { + await writeJsonl(dir, 'perf-20260101-unv.jsonl', [ + JSON.stringify({ foo: 'bar', baz: 42 }), + ]); + + const { counts } = await consumePerfDirectory(dir); + + expect(counts.unversioned).toBe(1); + expect(counts.parsed).toBe(0); + }); + + it('skips and counts malformed JSON lines', async () => { + await writeJsonl(dir, 'perf-20260101-bad.jsonl', [ + '{ this is not valid json', + JSON.stringify(makeOperation()), + ]); + + const { counts } = await consumePerfDirectory(dir); + + expect(counts.malformed).toBe(1); + expect(counts.parsed).toBe(1); + }); + + it('counts truncated final line (no trailing newline, unparseable)', async () => { + // Write a valid record, then a partial unterminated line without trailing newline + const valid = JSON.stringify(makeOperation()); + const partial = '{"schema_version":1,"record_type":"operation","ts":"2026'; + await fs.writeFile( + join(dir, 'perf-20260101-trunc.jsonl'), + valid + '\n' + partial, // no trailing newline on partial + 'utf8', + ); + + const { counts } = await consumePerfDirectory(dir); + + expect(counts.parsed).toBe(1); + expect(counts.truncated).toBe(1); + }); + + it('counts blank lines', async () => { + await writeJsonl(dir, 'perf-20260101-blank.jsonl', [ + '', + ' ', + JSON.stringify(makeOperation()), + ]); + + const { counts } = await consumePerfDirectory(dir); + + expect(counts.blank).toBe(2); + expect(counts.parsed).toBe(1); + }); + + it('does not parse claim files', async () => { + await writeJsonl(dir, 'perf-20260101-data.jsonl', [ + JSON.stringify(makeOperation()), + ]); + // Write a claim file that looks like it could be parsed + await fs.writeFile(join(dir, 'someuuid.claim'), '', 'utf8'); + + const { counts } = await consumePerfDirectory(dir); + + expect(counts.files).toBe(1); // only perf-*.jsonl + expect(counts.parsed).toBe(1); + }); + + it('aggregates counts across multiple files', async () => { + await writeJsonl(dir, 'perf-20260101-aaa.jsonl', [ + JSON.stringify(makeOperation({ operation_id: 'a1' })), + 'bad json', + ]); + await writeJsonl(dir, 'perf-20260102-bbb.jsonl', [ + JSON.stringify(makeOperation({ operation_id: 'b1' })), + JSON.stringify({ foo: 'unversioned' }), + JSON.stringify({ + ...makeOperation(), + schema_version: 999, + }), + ]); + + const { counts } = await consumePerfDirectory(dir); + + expect(counts.files).toBe(2); + expect(counts.parsed).toBe(2); + expect(counts.malformed).toBe(1); + expect(counts.unversioned).toBe(1); + expect(counts.futureVersion).toBe(1); + }); + + it('streamPerfDirectory yields entries lazily (before reading all files)', async () => { + await writeJsonl(dir, 'perf-20260101-stream.jsonl', [ + JSON.stringify(makeOperation({ operation_id: 's1' })), + JSON.stringify(makeOperation({ operation_id: 's2' })), + ]); + + let firstYield = false; + for await (const entry of streamPerfDirectory(dir)) { + if (entry.entry.kind === 'ok') { + firstYield = true; + break; // prove we can break early (lazy iteration) + } + } + + expect(firstYield).toBe(true); + }); + + it('tolerates a file evicted (ENOENT) during streaming without aborting the directory', async () => { + const opA = makeOperation({ operation_id: 'op-a' }); + const opB = makeOperation({ operation_id: 'op-b' }); + const fileA = 'perf-20260101-aaaa.jsonl'; + const fileB = 'perf-20260101-zzzz.jsonl'; + await writeJsonl(dir, fileA, [JSON.stringify(opA)]); + await writeJsonl(dir, fileB, [JSON.stringify(opB)]); + + const ids: string[] = []; + let evictedB = false; + for await (const entry of streamPerfDirectory(dir)) { + if ( + entry.entry.kind === 'ok' && + entry.entry.record.record_type === 'operation' + ) { + ids.push(entry.entry.record.operation_id); + } + if (!evictedB) { + await fs.rm(join(dir, fileB), { force: true }); + evictedB = true; + } + } + + expect(evictedB).toBe(true); + expect(ids).toEqual(['op-a']); + }); + + it('tolerates ENOENT when a statted file is deleted before stream open', async () => { + const filePath = join(dir, 'perf-20260101-open-race.jsonl'); + await writeJsonl(dir, 'perf-20260101-open-race.jsonl', [ + JSON.stringify(makeOperation()), + ]); + await fs.stat(filePath); + + const iterator = streamFileTolerant(filePath); + await fs.rm(filePath); + + expect((await iterator.next()).done).toBe(true); + }); + + it('propagates non-ENOENT stream errors', async () => { + const denied = Object.assign(new Error('permission denied'), { + code: 'EACCES', + }); + async function* deniedStream(): AsyncGenerator { + yield Promise.reject(denied); + } + + await expect( + streamFileTolerant('unused', deniedStream).next(), + ).rejects.toBe(denied); + }); + + it('closes the underlying stream when its consumer stops early', async () => { + let closed = 0; + async function* source(): AsyncGenerator { + try { + yield { kind: 'blank' }; + yield { kind: 'blank' }; + } finally { + closed += 1; + } + } + + const stream = streamFileTolerant('unused', source); + expect((await stream.next()).done).toBe(false); + await stream.return(undefined); + + expect(closed).toBe(1); + }); +}); diff --git a/packages/telemetry/src/perf/perfConsumer.ts b/packages/telemetry/src/perf/perfConsumer.ts new file mode 100644 index 0000000000..f57a7ec73b --- /dev/null +++ b/packages/telemetry/src/perf/perfConsumer.ts @@ -0,0 +1,255 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Cross-platform directory streaming consumer for the perf JSONL directory + * (P11, REQ-3167-9). + * + * Reads sorted `perf-*.jsonl` files one at a time (no gzip, no shell pipeline, + * no argument-limit breakage). Each yielded entry carries source-file / run-UUID + * identity so the report computes per-file memory slopes and never accidentally + * pools data across process uptimes / session indices. + * + * A missing directory is an empty dataset (fail open). Other genuine filesystem + * errors propagate so the caller (report / inspect / delete) can fail open at + * the external-inspection boundary and reflect the error in self-health. + * + * The consumer does NOT parse claim files. + */ + +import { promises as fsp } from 'node:fs'; +import { join } from 'node:path'; +import { streamPerfRecords, type PerfStreamEntry } from './perfRecords.js'; +import { isPerfJsonl, extractRunUuid } from './perfArtifacts.js'; + +/** + * A classified perf line annotated with its source file and run-UUID identity. + * The run UUID is parsed from the filename (`perf-YYYYMMDD-.jsonl`) so + * memory slopes can be computed per run/file without pooling. + */ +export interface PerfConsumerEntry { + readonly entry: PerfStreamEntry; + readonly sourceFile: string; + readonly runUuid: string; +} + +/** + * Aggregate counters across all files in the directory. Extends the per-file + * reader counts with file/byte totals for the inspect surface. + */ +export interface PerfConsumerCounts { + readonly parsed: number; + readonly malformed: number; + readonly futureVersion: number; + readonly unversioned: number; + readonly truncated: number; + readonly blank: number; + readonly files: number; + readonly bytes: number; +} + +export interface PerfConsumerResult { + readonly entries: readonly PerfConsumerEntry[]; + readonly counts: PerfConsumerCounts; +} + +/** + * Returns true if an error carries the given Node errno code. + */ +function hasErrnoCode(err: unknown, code: string): boolean { + return err instanceof Error && (err as NodeJS.ErrnoException).code === code; +} + +type PerfRecordStreamFactory = ( + filePath: string, +) => AsyncGenerator; + +async function nextPerfEntry( + iterator: AsyncIterator, +): Promise | null> { + try { + return await iterator.next(); + } catch (err) { + if (hasErrnoCode(err, 'ENOENT')) return null; + throw err; + } +} + +/** + * Streams one perf file, skipping it if it was evicted between `stat` and + * `open` (or mid-read) by a concurrent retention sweep in another process. + * Only ENOENT — a genuinely external file disappearance — is tolerated; every + * other error propagates so the caller can surface it. + * + * @internal Exported only from this module for deterministic boundary tests. + */ +export async function* streamFileTolerant( + filePath: string, + streamRecords: PerfRecordStreamFactory = streamPerfRecords, +): AsyncGenerator { + const iterator = streamRecords(filePath)[Symbol.asyncIterator](); + let exhausted = false; + try { + for (;;) { + const next = await nextPerfEntry(iterator); + if (next === null || next.done === true) { + exhausted = true; + return; + } + yield next.value; + } + } finally { + if (!exhausted) { + await iterator.return(undefined); + } + } +} + +/** + * Extracts the run UUID from a perf JSONL filename, throwing on null. + * + * `streamPerfDirectory`/`consumePerfDirectory` only enumerate names that pass + * `isPerfJsonl`, so `extractRunUuid` can never legitimately return null here. + * A null result is an internal invariant violation (a name slipped through that + * does not match `perf-YYYYMMDD-.jsonl`) and must fail fast rather than + * be masked by an invented `unknown` identity. + */ +function requireRunUuid(name: string): string { + const uuid = extractRunUuid(name); + if (uuid === null) { + throw new Error( + `Internal invariant violation: extractRunUuid returned null for perf JSONL name '${name}'`, + ); + } + return uuid; +} + +interface PerfFileInfo { + readonly name: string; + readonly path: string; + readonly bytes: number; +} + +/** + * Lists sorted `perf-*.jsonl` files in a directory, returning name + path + + * byte size. A missing directory (ENOENT) yields nothing — an empty dataset. + * Other errors propagate. + */ +async function* listPerfFiles(dir: string): AsyncGenerator { + let names: string[]; + try { + names = await fsp.readdir(dir); + } catch (err) { + if (hasErrnoCode(err, 'ENOENT')) return; // missing dir = empty dataset + throw err; + } + + const sorted = names.filter(isPerfJsonl).sort(); + for (const name of sorted) { + const filePath = join(dir, name); + let bytes = 0; + try { + const stat = await fsp.stat(filePath); + bytes = stat.size; + } catch (err) { + if (!hasErrnoCode(err, 'ENOENT')) throw err; + // ENOENT race (file deleted between readdir and stat) — skip. + continue; + } + yield { name, path: filePath, bytes }; + } +} + +/** + * Streams perf consumer entries from all `perf-*.jsonl` files in a directory, + * one file at a time. Each entry carries its source file name and run UUID. + * + * A missing directory yields nothing (empty dataset). Other genuine filesystem + * errors propagate. + */ +export async function* streamPerfDirectory( + dir: string, +): AsyncGenerator { + for await (const file of listPerfFiles(dir)) { + const runUuid = requireRunUuid(file.name); + for await (const entry of streamFileTolerant(file.path)) { + yield { entry, sourceFile: file.name, runUuid }; + } + } +} + +/** + * Accumulating directory consumer: streams all `perf-*.jsonl` files and + * accumulates every parsed entry with aggregate counts. This is NOT an + * algorithmically bounded collector — it retains all entries in memory so the + * report can group and compute p50/slopes across the full dataset. A bounded + * variant would lose the longitudinal comparison the report needs. + * + * Missing directory = empty result. Other genuine filesystem errors + * propagate. + */ +export async function consumePerfDirectory( + dir: string, +): Promise { + const entries: PerfConsumerEntry[] = []; + let parsed = 0; + let malformed = 0; + let futureVersion = 0; + let unversioned = 0; + let truncated = 0; + let blank = 0; + let files = 0; + let bytes = 0; + + for await (const file of listPerfFiles(dir)) { + files += 1; + bytes += file.bytes; + const runUuid = requireRunUuid(file.name); + for await (const entry of streamFileTolerant(file.path)) { + entries.push({ entry, sourceFile: file.name, runUuid }); + switch (entry.kind) { + case 'ok': + parsed += 1; + break; + case 'malformed': + malformed += 1; + break; + case 'future_version': + futureVersion += 1; + break; + case 'unversioned': + unversioned += 1; + break; + case 'truncated': + truncated += 1; + break; + case 'blank': + blank += 1; + break; + default: { + const _exhaustive: never = entry; + throw new Error( + `Internal invariant violation: unhandled PerfStreamEntry kind: ${JSON.stringify(_exhaustive)}`, + ); + } + } + } + } + + return { + entries, + counts: { + parsed, + malformed, + futureVersion, + unversioned, + truncated, + blank, + files, + bytes, + }, + }; +} diff --git a/packages/telemetry/src/perf/perfDelete.behavior.test.ts b/packages/telemetry/src/perf/perfDelete.behavior.test.ts new file mode 100644 index 0000000000..50a1189fd7 --- /dev/null +++ b/packages/telemetry/src/perf/perfDelete.behavior.test.ts @@ -0,0 +1,412 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import { promises as fs } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { perfDelete } from './perfDelete.js'; +import type { PerfDeleteFilesystem } from './perfDelete.js'; + +async function makeTempDir(): Promise { + const dir = join( + tmpdir(), + `perf-delete-${Date.now()}-${Math.random().toString(36).slice(2)}`, + ); + await fs.mkdir(dir, { recursive: true }); + return dir; +} + +function utcDayKey(now: number): string { + const d = new Date(now); + const y = d.getUTCFullYear(); + const m = String(d.getUTCMonth() + 1).padStart(2, '0'); + const day = String(d.getUTCDate()).padStart(2, '0'); + return `${y}${m}${day}`; +} + +async function writeFile( + dir: string, + name: string, + content: string, + mtimeMs?: number, +): Promise { + await fs.writeFile(join(dir, name), content, 'utf8'); + if (mtimeMs !== undefined) { + const d = new Date(mtimeMs); + await fs.utimes(join(dir, name), d, d); + } +} + +describe('PerfDelete (P11, AC-9, D3)', () => { + let dir: string; + const now = Date.parse('2026-01-15T12:00:00.000Z'); + const dayKey = utcDayKey(now); + const maintenanceIntervalMs = 60_000; + const claimLeaseMs = 180_000; + + beforeEach(async () => { + dir = await makeTempDir(); + }); + + afterEach(async () => { + await fs.rm(dir, { recursive: true, force: true }); + }); + + it('deletes stale perf JSONL files and stale claims', async () => { + // Old file (past day) — eligible + await writeFile(dir, `perf-20260101-old.jsonl`, '{"schema_version":1}\n'); + // Stale claim (mtime is old, past lease) + await writeFile(dir, 'stale-uuid.claim', '', now - claimLeaseMs - 10_000); + + const result = await perfDelete({ + dir, + now, + maintenanceIntervalMs, + claimLeaseMs, + }); + + expect(result.deleted).toBe(2); + expect(result.protected).toBe(0); + expect(result.failed).toBe(0); + expect(result.deletedFiles).toContain('perf-20260101-old.jsonl'); + expect(result.deletedFiles).toContain('stale-uuid.claim'); + + // Verify files are actually gone + const remaining = await fs.readdir(dir); + expect(remaining).toHaveLength(0); + }); + + it('protects current UTC-day file with recent mtime (active writer)', async () => { + // Today's file with recent mtime + await writeFile( + dir, + `perf-${dayKey}-active.jsonl`, + '{"schema_version":1}\n', + now - 10_000, // 10s ago, within maintenance window + ); + + const result = await perfDelete({ + dir, + now, + maintenanceIntervalMs, + claimLeaseMs, + }); + + expect(result.deleted).toBe(0); + expect(result.protected).toBe(1); + expect(result.protectedFiles).toContain(`perf-${dayKey}-active.jsonl`); + + // File still exists + const remaining = await fs.readdir(dir); + expect(remaining).toContain(`perf-${dayKey}-active.jsonl`); + }); + + it('protects perf JSONL whose run UUID has a non-stale claim', async () => { + // Old-day perf file (would normally be eligible) + await writeFile( + dir, + 'perf-20260101-protected.jsonl', + '{"schema_version":1}\n', + ); + // Fresh claim for the same run UUID + await writeFile( + dir, + 'protected.claim', + '', + now - 10_000, // fresh, within lease + ); + + const result = await perfDelete({ + dir, + now, + maintenanceIntervalMs, + claimLeaseMs, + }); + + // The perf JSONL is protected by the fresh claim + expect(result.protectedFiles).toContain('perf-20260101-protected.jsonl'); + expect(result.protectedFiles).toContain('protected.claim'); + expect(result.deleted).toBe(0); + }); + + it('protects non-stale/future-dated claims (lease)', async () => { + // Fresh claim + await writeFile(dir, 'fresh.claim', '', now - 1000); + // Future-dated claim + await writeFile(dir, 'future.claim', '', now + 60_000); + // Stale claim + await writeFile(dir, 'stale.claim', '', now - claimLeaseMs - 10_000); + + const result = await perfDelete({ + dir, + now, + maintenanceIntervalMs, + claimLeaseMs, + }); + + expect(result.protectedFiles).toContain('fresh.claim'); + expect(result.protectedFiles).toContain('future.claim'); + expect(result.deletedFiles).toContain('stale.claim'); + }); + + it('deletes old perf JSONL whose claim is stale', async () => { + // Old-day perf file + await writeFile(dir, 'perf-20260101-stale-claim.jsonl', '{"v":1}\n'); + // Stale claim for the same run UUID + await writeFile(dir, 'stale-claim.claim', '', now - claimLeaseMs - 10_000); + + const result = await perfDelete({ + dir, + now, + maintenanceIntervalMs, + claimLeaseMs, + }); + + expect(result.deletedFiles).toContain('perf-20260101-stale-claim.jsonl'); + expect(result.deletedFiles).toContain('stale-claim.claim'); + }); + + it('never deletes unrelated files', async () => { + await writeFile(dir, 'random.txt', 'hello'); + await writeFile(dir, 'other.log', 'world'); + await writeFile(dir, 'config.json', '{}'); + // Also a stale perf file to trigger the delete logic + await writeFile(dir, 'perf-20260101-old.jsonl', '{"v":1}\n'); + + const result = await perfDelete({ + dir, + now, + maintenanceIntervalMs, + claimLeaseMs, + }); + + // Only perf-*.jsonl and *.claim are touched + expect(result.deletedFiles).not.toContain('random.txt'); + expect(result.deletedFiles).not.toContain('other.log'); + expect(result.deletedFiles).not.toContain('config.json'); + + // Unrelated files still exist + const remaining = await fs.readdir(dir); + expect(remaining).toContain('random.txt'); + expect(remaining).toContain('other.log'); + expect(remaining).toContain('config.json'); + }); + + it('missing directory is a no-op (fail open)', async () => { + const result = await perfDelete({ + dir: join(dir, 'does-not-exist'), + now, + maintenanceIntervalMs, + claimLeaseMs, + }); + + expect(result.deleted).toBe(0); + expect(result.protected).toBe(0); + expect(result.failed).toBe(0); + }); + + it('external fs failures fail open and are counted', async () => { + await writeFile(dir, 'perf-20260101-old.jsonl', '{"v":1}\n'); + + const failingFs: PerfDeleteFilesystem = { + readdir: async (d: string) => fs.readdir(d), + stat: async (p: string) => { + const s = await fs.stat(p); + return { size: s.size, mtimeMs: s.mtimeMs }; + }, + unlink: async () => { + const err = new Error('EACCES') as NodeJS.ErrnoException; + err.code = 'EACCES'; + throw err; + }, + }; + + const result = await perfDelete({ + dir, + now, + maintenanceIntervalMs, + claimLeaseMs, + fs: failingFs, + }); + + expect(result.failed).toBe(1); + expect(result.deleted).toBe(0); + expect(result.failedFiles).toContain('perf-20260101-old.jsonl'); + }); + + it('internal invalid options (NaN now) fail fast', async () => { + await expect( + perfDelete({ + dir, + now: NaN, + maintenanceIntervalMs, + claimLeaseMs, + }), + ).rejects.toThrow(RangeError); + }); + + it('internal invalid options (negative interval) fail fast', async () => { + await expect( + perfDelete({ + dir, + now, + maintenanceIntervalMs: -1, + claimLeaseMs, + }), + ).rejects.toThrow(RangeError); + }); + + it('deletes stale claims alongside perf JSONL from same run', async () => { + // Old perf file from run "aaa" with stale claim + await writeFile(dir, 'perf-20260101-aaa.jsonl', '{"v":1}\n'); + await writeFile(dir, 'aaa.claim', '', now - claimLeaseMs - 10_000); + + // Old perf file from run "bbb" with fresh claim → protected + await writeFile(dir, 'perf-20260101-bbb.jsonl', '{"v":1}\n'); + await writeFile(dir, 'bbb.claim', '', now - 1000); + + const result = await perfDelete({ + dir, + now, + maintenanceIntervalMs, + claimLeaseMs, + }); + + // aaa's stale perf and claim deleted + expect(result.deletedFiles).toContain('perf-20260101-aaa.jsonl'); + expect(result.deletedFiles).toContain('aaa.claim'); + // bbb's perf and claim protected (fresh claim) + expect(result.protectedFiles).toContain('perf-20260101-bbb.jsonl'); + expect(result.protectedFiles).toContain('bbb.claim'); + }); + + // --- P11: external errno readdir/stat failures are counted as failures --- + + it('EACCES readdir failure is counted as a failure with a directory sentinel', async () => { + await writeFile(dir, 'perf-20260101-old.jsonl', '{"v":1}\n'); + + const failingFs: PerfDeleteFilesystem = { + readdir: async () => { + const err = new Error('permission denied') as NodeJS.ErrnoException; + err.code = 'EACCES'; + throw err; + }, + stat: async (p: string) => { + const s = await fs.stat(p); + return { size: s.size, mtimeMs: s.mtimeMs }; + }, + unlink: async (p: string) => fs.unlink(p), + }; + + const result = await perfDelete({ + dir, + now, + maintenanceIntervalMs, + claimLeaseMs, + fs: failingFs, + }); + + // readdir failed entirely: no files enumerated, but the failure is + // surfaced (not silently zeroed) using a stable directory sentinel. + expect(result.failed).toBe(1); + expect(result.deleted).toBe(0); + expect(result.protected).toBe(0); + expect(result.failedFiles).toHaveLength(1); + expect(typeof result.failedFiles[0]).toBe('string'); + expect(result.failedFiles[0].length).toBeGreaterThan(0); + }); + + it('EROFS stat failure is counted as a failure with the file name', async () => { + await writeFile(dir, 'perf-20260101-old.jsonl', '{"v":1}\n'); + await writeFile(dir, 'perf-20260102-old2.jsonl', '{"v":1}\n'); + + const failingFs: PerfDeleteFilesystem = { + readdir: async (d: string) => fs.readdir(d), + stat: async (p: string) => { + if (p.endsWith('perf-20260101-old.jsonl')) { + const err = new Error( + 'read-only file system', + ) as NodeJS.ErrnoException; + err.code = 'EROFS'; + throw err; + } + const s = await fs.stat(p); + return { size: s.size, mtimeMs: s.mtimeMs }; + }, + unlink: async (p: string) => fs.unlink(p), + }; + + const result = await perfDelete({ + dir, + now, + maintenanceIntervalMs, + claimLeaseMs, + fs: failingFs, + }); + + // The file whose stat failed with EROFS is counted as a failure (named), + // not silently dropped to null. The other eligible file is deleted. + expect(result.failed).toBe(1); + expect(result.failedFiles).toContain('perf-20260101-old.jsonl'); + expect(result.deletedFiles).toContain('perf-20260102-old2.jsonl'); + }); + + it('ENOENT readdir (missing dir) remains a no-op, not a failure', async () => { + const result = await perfDelete({ + dir: join(dir, 'missing-subdir'), + now, + maintenanceIntervalMs, + claimLeaseMs, + }); + + expect(result.failed).toBe(0); + expect(result.deleted).toBe(0); + expect(result.protected).toBe(0); + }); + + it('non-errno readdir failure rejects (internal/programming error)', async () => { + const failingFs: PerfDeleteFilesystem = { + readdir: async () => { + throw new TypeError('programmer bug'); + }, + stat: async () => ({ size: 0, mtimeMs: 0 }), + unlink: async () => {}, + }; + + await expect( + perfDelete({ + dir, + now, + maintenanceIntervalMs, + claimLeaseMs, + fs: failingFs, + }), + ).rejects.toThrow(TypeError); + }); + + it('non-errno stat failure rejects (internal/programming error)', async () => { + await writeFile(dir, 'perf-20260101-old.jsonl', '{"v":1}\n'); + + const failingFs: PerfDeleteFilesystem = { + readdir: async (d: string) => fs.readdir(d), + stat: async () => { + throw new RangeError('internal stat bug'); + }, + unlink: async () => {}, + }; + + await expect( + perfDelete({ + dir, + now, + maintenanceIntervalMs, + claimLeaseMs, + fs: failingFs, + }), + ).rejects.toThrow(RangeError); + }); +}); diff --git a/packages/telemetry/src/perf/perfDelete.ts b/packages/telemetry/src/perf/perfDelete.ts new file mode 100644 index 0000000000..1164c16c91 --- /dev/null +++ b/packages/telemetry/src/perf/perfDelete.ts @@ -0,0 +1,433 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Perf directory deletion with live-writer safety (P11, REQ-3167-8, D3). + * + * Removes owned perf JSONL and stale claim artifacts only. Protects: + * - The current UTC-day perf file with recent/future mtime (active writer). + * - Any perf JSONL whose run UUID has a non-stale / future-dated claim (lease). + * + * Reuses the shared artifact parsing and protection primitives from + * `perfArtifacts.ts`. Explicit delete intentionally applies broader JSONL + * protection than automatic retention by honoring fresh run claims. + * + * Never deletes unrelated files (only `perf-YYYYMMDD-*.jsonl` and `*.claim`). + * External filesystem failures fail open and are counted. Internal invalid + * options fail fast. No broad `rm`. + */ + +import { promises as fsp } from 'node:fs'; +import { join } from 'node:path'; +import { + PERF_MAINTENANCE_INTERVAL_MS, + PERF_CLAIM_LEASE_MS, +} from './retention.js'; +import { + isPerfJsonl, + isClaimFile, + extractRunUuid, + isNonStaleClaim, + collectFreshClaimRunUuids, + isPerfJsonlProtected, +} from './perfArtifacts.js'; + +/** + * Narrow filesystem port for delete operations. Tests inject a custom + * implementation to produce deterministic failures. + */ +export interface PerfDeleteFilesystem { + readdir(dir: string): Promise; + stat(path: string): Promise<{ size: number; mtimeMs: number }>; + unlink(path: string): Promise; +} + +/** Default filesystem port using real `node:fs/promises`. */ +class RealDeleteFilesystem implements PerfDeleteFilesystem { + async readdir(dir: string): Promise { + return fsp.readdir(dir); + } + + async stat(path: string): Promise<{ size: number; mtimeMs: number }> { + const s = await fsp.stat(path); + return { size: s.size, mtimeMs: s.mtimeMs }; + } + + async unlink(path: string): Promise { + await fsp.unlink(path); + } +} + +export interface PerfDeleteOptions { + readonly dir: string; + readonly fs?: PerfDeleteFilesystem; + readonly now?: number; + readonly maintenanceIntervalMs?: number; + readonly claimLeaseMs?: number; +} + +export interface PerfDeleteResult { + readonly deleted: number; + readonly protected: number; + readonly failed: number; + readonly deletedFiles: readonly string[]; + readonly protectedFiles: readonly string[]; + readonly failedFiles: readonly string[]; +} + +function isErrnoError(err: unknown): boolean { + if (!(err instanceof Error)) return false; + return typeof (err as NodeJS.ErrnoException).code === 'string'; +} + +function hasErrnoCode(err: unknown, code: string): boolean { + return err instanceof Error && (err as NodeJS.ErrnoException).code === code; +} + +/** + * Stable sentinel used as the failed-file name when a directory-level + * filesystem failure (e.g. EACCES/EROFS on readdir) occurs and there is no + * individual filename to attribute the failure to. + */ +const DIRECTORY_SENTINEL = ''; + +/** + * Deletes owned perf JSONL and stale claim artifacts from a directory, + * respecting live-writer safety. + * + * - A missing directory is a no-op (fail open, returns zero counts). + * - External filesystem failures fail open and are counted. + * - Internal invalid options (NaN/negative timing) fail fast. + * + * Explicit-delete protection rules (using primitives from `perfArtifacts.ts`): + * - A perf JSONL whose day-key is today UTC AND mtime is within the + * maintenance interval is protected (active writer). + * - A perf JSONL whose run UUID has a non-stale/future claim is protected; + * automatic retention deliberately does not apply this broader rule. + * - A claim that is non-stale (now - mtime ≤ lease) is protected. + */ +interface StatedArtifact { + readonly name: string; + readonly path: string; + readonly mtimeMs: number; + readonly runUuid: string | null; +} + +interface DeleteAccumulator { + readonly deleted: string[]; + readonly protected: string[]; + readonly failed: string[]; +} + +function emptyAccumulator(): DeleteAccumulator { + return { deleted: [], protected: [], failed: [] }; +} + +function validateDeleteOptions( + now: number, + maintenanceIntervalMs: number, + claimLeaseMs: number, +): void { + if (!Number.isFinite(now)) { + throw new RangeError(`perfDelete: now must be finite (got ${now})`); + } + if (!Number.isFinite(maintenanceIntervalMs) || maintenanceIntervalMs <= 0) { + throw new RangeError( + `perfDelete: maintenanceIntervalMs must be finite positive (got ${maintenanceIntervalMs})`, + ); + } + if (!Number.isFinite(claimLeaseMs) || claimLeaseMs <= 0) { + throw new RangeError( + `perfDelete: claimLeaseMs must be finite positive (got ${claimLeaseMs})`, + ); + } +} + +/** + * Stats a single owned artifact. Returns the stated artifact, or `null` to + * signal a skip (non-owned name or an ENOENT race). Genuine non-ENOENT + * filesystem failures during stat are surfaced via `failures` so they are + * counted rather than silently dropped to null. Internal/programming errors + * (non-errno) rethrow. + */ +async function statOwnedArtifact( + fsPort: PerfDeleteFilesystem, + dir: string, + name: string, + failures: string[], +): Promise { + if (!isPerfJsonl(name) && !isClaimFile(name)) return null; + try { + const statResult = await fsPort.stat(join(dir, name)); + return { + name, + path: join(dir, name), + mtimeMs: statResult.mtimeMs, + runUuid: extractRunUuid(name), + }; + } catch (err) { + if (isErrnoError(err)) { + if (hasErrnoCode(err, 'ENOENT')) { + // ENOENT race (file deleted between readdir and stat) — skip. + return null; + } + // Other errno (EACCES/EROFS/…) — count as a failure, do not delete. + failures.push(name); + return null; + } + throw err; + } +} + +/** + * Phase 1: Stats all owned perf JSONL and claim artifacts. + * ENOENT races skip silently; other errno errors are counted as failures; + * internal errors throw. + */ +async function collectOwnedArtifacts( + fsPort: PerfDeleteFilesystem, + dir: string, + names: readonly string[], +): Promise<{ + readonly jsonl: StatedArtifact[]; + readonly claims: StatedArtifact[]; + readonly statFailures: string[]; +}> { + const jsonl: StatedArtifact[] = []; + const claims: StatedArtifact[] = []; + const statFailures: string[] = []; + + for (const name of names) { + const artifact = await statOwnedArtifact(fsPort, dir, name, statFailures); + if (artifact !== null) { + if (isPerfJsonl(name)) { + jsonl.push(artifact); + } else { + claims.push(artifact); + } + } + } + + return { jsonl, claims, statFailures }; +} + +/** + * Attempts to delete one artifact via unlink, appending to the accumulator. + */ +async function attemptDelete( + fsPort: PerfDeleteFilesystem, + artifact: StatedArtifact, + acc: DeleteAccumulator, +): Promise { + const ok = await safeUnlink(fsPort, artifact.path); + if (ok) { + acc.deleted.push(artifact.name); + } else { + acc.failed.push(artifact.name); + } +} + +/** + * Phase 2: Delete claims. Non-stale claims are protected; stale claims are + * deleted. Returns the set of non-stale claim UUIDs for JSONL protection, + * computed via the centralized {@link collectFreshClaimRunUuids}. + */ +async function deleteClaims( + fsPort: PerfDeleteFilesystem, + claims: readonly StatedArtifact[], + now: number, + claimLeaseMs: number, + acc: DeleteAccumulator, +): Promise> { + const nonStaleUuids = collectFreshClaimRunUuids( + claims.map((c) => ({ runUuid: c.runUuid, mtimeMs: c.mtimeMs })), + now, + claimLeaseMs, + ); + + for (const claim of claims) { + const isProtected = isNonStaleClaim(claim.mtimeMs, now, claimLeaseMs); + if (isProtected) { + acc.protected.push(claim.name); + continue; + } + await attemptDelete(fsPort, claim, acc); + } + + return nonStaleUuids; +} + +/** + * Phase 3: Delete perf JSONL. Protects live-writer files and any file whose + * run UUID has a non-stale claim, via the centralized + * {@link isPerfJsonlProtected}. This claim→JSONL protection is deliberate for + * explicit delete (avoid unlinking a file another active process appends) and + * is intentionally broader than automatic retention, which protects JSONL only + * as a live writer. + */ +async function deletePerfJsonl( + fsPort: PerfDeleteFilesystem, + files: readonly StatedArtifact[], + now: number, + maintenanceIntervalMs: number, + nonStaleClaimUuids: ReadonlySet, + acc: DeleteAccumulator, +): Promise { + for (const file of files) { + const isProtected = isPerfJsonlProtected( + file.name, + file.mtimeMs, + now, + maintenanceIntervalMs, + nonStaleClaimUuids, + ); + + if (isProtected) { + acc.protected.push(file.name); + continue; + } + await attemptDelete(fsPort, file, acc); + } +} + +export async function perfDelete( + options: PerfDeleteOptions, +): Promise { + const dir = options.dir; + const fsPort = options.fs ?? new RealDeleteFilesystem(); + const now = options.now ?? Date.now(); + const maintenanceIntervalMs = + options.maintenanceIntervalMs ?? PERF_MAINTENANCE_INTERVAL_MS; + const claimLeaseMs = options.claimLeaseMs ?? PERF_CLAIM_LEASE_MS; + + validateDeleteOptions(now, maintenanceIntervalMs, claimLeaseMs); + + let names: string[]; + try { + names = await fsPort.readdir(dir); + } catch (err) { + if (isErrnoError(err)) { + if (hasErrnoCode(err, 'ENOENT')) { + // Missing directory is a valid empty dataset — no-op. + return { + deleted: 0, + protected: 0, + failed: 0, + deletedFiles: [], + protectedFiles: [], + failedFiles: [], + }; + } + // Other genuine filesystem failures (EACCES/EROFS/…) fail open but are + // counted. There is no individual filename, so a stable directory + // sentinel identifies the failure. + return { + deleted: 0, + protected: 0, + failed: 1, + deletedFiles: [], + protectedFiles: [], + failedFiles: [DIRECTORY_SENTINEL], + }; + } + throw err; + } + + const { jsonl, claims, statFailures } = await collectOwnedArtifacts( + fsPort, + dir, + names, + ); + + const acc = emptyAccumulator(); + // Surface stat-level filesystem failures before deletion proceeds. + for (const name of statFailures) { + acc.failed.push(name); + } + + const nonStaleClaimUuids = await deleteClaims( + fsPort, + claims, + now, + claimLeaseMs, + acc, + ); + + await deletePerfJsonl( + fsPort, + jsonl, + now, + maintenanceIntervalMs, + nonStaleClaimUuids, + acc, + ); + + return { + deleted: acc.deleted.length, + protected: acc.protected.length, + failed: acc.failed.length, + deletedFiles: acc.deleted, + protectedFiles: acc.protected, + failedFiles: acc.failed, + }; +} + +/** + * Attempts to unlink a file. Returns true on success, false on filesystem + * errors (fail open — counted). Rethrows internal/programming errors. + */ +async function safeUnlink( + fsPort: PerfDeleteFilesystem, + filePath: string, +): Promise { + try { + await fsPort.unlink(filePath); + return true; + } catch (err) { + if (isErrnoError(err)) return false; + throw err; + } +} + +/** + * Formats a delete result into a stable, human-readable string. + */ +export function formatDeleteResult(result: PerfDeleteResult): string { + const lines: string[] = []; + + lines.push('Perf Delete'); + lines.push('==========='); + lines.push(''); + lines.push(`Deleted: ${result.deleted} file(s)`); + lines.push(`Protected (live): ${result.protected} file(s)`); + lines.push(`Failed: ${result.failed} file(s)`); + + if (result.deletedFiles.length > 0) { + lines.push(''); + lines.push('Deleted files:'); + for (const f of result.deletedFiles) { + lines.push(` ${f}`); + } + } + + if (result.protectedFiles.length > 0) { + lines.push(''); + lines.push('Protected files:'); + for (const f of result.protectedFiles) { + lines.push(` ${f}`); + } + } + + if (result.failedFiles.length > 0) { + lines.push(''); + lines.push('Failed files:'); + for (const f of result.failedFiles) { + lines.push(` ${f}`); + } + } + + return lines.join('\n'); +} diff --git a/packages/telemetry/src/perf/perfInspect.behavior.test.ts b/packages/telemetry/src/perf/perfInspect.behavior.test.ts new file mode 100644 index 0000000000..201f6b5e1e --- /dev/null +++ b/packages/telemetry/src/perf/perfInspect.behavior.test.ts @@ -0,0 +1,214 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import { promises as fs } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { perfInspect, formatInspect } from './perfInspect.js'; +import type { + PerfOperationRecord, + PerfMemorySampleRecord, +} from './perfRecords.js'; + +function makeOperation( + overrides: Partial = {}, +): PerfOperationRecord { + return { + schema_version: 1, + record_type: 'operation', + ts: '2026-01-01T00:00:00.000Z', + session_id: 'sess-1', + operation_id: 'op-1', + runtime_id: 'rt-1', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'hash-1', + llxprt_version: '0.10.0', + git_sha: 'abc1234', + runtime: 'cli', + platform: 'darwin', + provider: 'test-provider', + model: 'test-model', + context_tokens: 1000, + output_tokens: 500, + terminal_cols: 80, + terminal_rows: 24, + render_mode: 'ink', + concurrent_instances: 1, + status: 'completed', + client_prepare_ms: 10, + stream_handler_ms: 100, + ink_render_ms: 5, + ink_render_count: 3, + stdout_bytes: 1024, + stdout_write_calls: 5, + stdout_write_sync_ms: 2, + client_finalize_ms: 8, + provider_attempts: 1, + provider_attempt_sum_ms: 200, + provider_union_ms: 200, + tool_calls: 2, + tool_call_sum_ms: 50, + tool_union_ms: 50, + agent_activity_union_ms: 250, + operation_elapsed_ms: 1000, + approval_wait_ms: 0, + unclassified_elapsed_ms: 0, + session_operation_index: 0, + uptime_ms: 5000, + ...overrides, + }; +} + +function makeMemorySample( + overrides: Partial = {}, +): PerfMemorySampleRecord { + return { + schema_version: 1, + record_type: 'memory_sample', + ts: '2026-01-01T00:01:00.000Z', + rss_bytes: 50_000_000, + heap_used_bytes: 20_000_000, + external_bytes: 5_000_000, + array_buffers_bytes: 1_000_000, + uptime_ms: 60_000, + ms_since_last_operation: 30_000, + ...overrides, + }; +} + +async function makeTempDir(): Promise { + return fs.mkdtemp(join(tmpdir(), 'perf-inspect-')); +} + +async function writeJsonl( + dir: string, + name: string, + lines: string[], +): Promise { + const content = lines.join('\n'); + await fs.writeFile( + join(dir, name), + content + (content.endsWith('\n') ? '' : '\n'), + 'utf8', + ); +} + +describe('PerfInspect (P11, AC-9)', () => { + let dir: string; + + beforeEach(async () => { + dir = await makeTempDir(); + }); + + afterEach(async () => { + await fs.rm(dir, { recursive: true, force: true }); + }); + + it('returns dir, schema version, and privacy statement', async () => { + const result = await perfInspect(dir); + + expect(result.dir).toBe(dir); + expect(result.schemaVersion).toBe(1); + expect(result.privacy.localOnly).toBe(true); + expect(result.privacy.defaultOff).toBe(true); + expect(result.privacy.noUpload).toBe(true); + expect(result.privacy.memorySeparatelyOptIn).toBe(true); + }); + + it('counts owned JSONL files and total bytes', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + ]); + await writeJsonl(dir, 'perf-20260102-run2.jsonl', [ + JSON.stringify(makeOperation()), + ]); + + const result = await perfInspect(dir); + + expect(result.fileCount).toBe(2); + expect(result.totalBytes).toBeGreaterThan(0); + }); + + it('counts operations and memory samples separately', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + JSON.stringify(makeOperation({ operation_id: 'op-2' })), + JSON.stringify(makeMemorySample()), + JSON.stringify(makeMemorySample({ ts: '2026-01-01T00:02:00.000Z' })), + ]); + + const result = await perfInspect(dir); + + expect(result.operationCount).toBe(2); + expect(result.memorySampleCount).toBe(2); + }); + + it('includes tolerant skipped breakdown', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + 'bad json', + JSON.stringify({ + ...makeOperation(), + schema_version: 999, + }), + JSON.stringify({ unknown: 'shape' }), + ]); + + const result = await perfInspect(dir); + + expect(result.skipped.malformed).toBe(1); + expect(result.skipped.futureVersion).toBe(1); + expect(result.skipped.unversioned).toBe(1); + expect(result.counts.parsed).toBe(1); + }); + + it('counts claim files', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + ]); + await fs.writeFile(join(dir, 'uuid-aaa.claim'), '', 'utf8'); + await fs.writeFile(join(dir, 'uuid-bbb.claim'), '', 'utf8'); + + const result = await perfInspect(dir); + + expect(result.claimCount).toBe(2); + }); + + it('missing directory returns zero counts (empty dataset)', async () => { + const result = await perfInspect(join(dir, 'does-not-exist')); + + expect(result.fileCount).toBe(0); + expect(result.totalBytes).toBe(0); + expect(result.operationCount).toBe(0); + expect(result.memorySampleCount).toBe(0); + expect(result.claimCount).toBe(0); + }); + + it('formatter includes all key fields', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + ]); + await fs.writeFile(join(dir, 'uuid.claim'), '', 'utf8'); + + const result = await perfInspect(dir); + const text = formatInspect(result); + + expect(text).toContain('Perf Inspect'); + expect(text).toContain('Directory:'); + expect(text).toContain('Schema version:'); + expect(text).toContain('Privacy:'); + expect(text).toContain('local-only'); + expect(text).toContain('default-off'); + expect(text).toContain('memory collection separately opt-in'); + expect(text).toContain('Owned JSONL files:'); + expect(text).toContain('Claim files:'); + expect(text).toContain('operations:'); + expect(text).toContain('memory samples:'); + expect(text).toContain('Skipped breakdown:'); + }); +}); diff --git a/packages/telemetry/src/perf/perfInspect.ts b/packages/telemetry/src/perf/perfInspect.ts new file mode 100644 index 0000000000..407111d234 --- /dev/null +++ b/packages/telemetry/src/perf/perfInspect.ts @@ -0,0 +1,154 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Perf directory inspection (P11, REQ-3167-8, D7). + * + * Surfaces the directory path, schema version, privacy statement, owned JSONL + * file count / bytes, operation and memory-sample counts, tolerant skipped + * breakdown, and claim count. Uses real files. + */ + +import { promises as fsp } from 'node:fs'; +import { PERF_SCHEMA_VERSION } from './perfRecords.js'; +import { isClaimFile } from './perfArtifacts.js'; +import type { PerfConsumerCounts } from './perfConsumer.js'; + +/** + * Counts of lines that were SKIPPED during tolerant reading — every + * non-parsed line EXCEPT truncated (which is surfaced separately). This + * deliberately excludes `parsed` (successfully processed records are NOT + * skipped), so it is a narrower type than {@link PerfReaderCounts}. + */ +export interface PerfSkippedCounts { + readonly malformed: number; + readonly futureVersion: number; + readonly unversioned: number; + readonly truncated: number; + readonly blank: number; +} + +/** The inspect result. */ +export interface PerfInspectResult { + readonly dir: string; + readonly schemaVersion: number; + readonly privacy: { + readonly localOnly: true; + readonly defaultOff: true; + readonly noUpload: true; + readonly memorySeparatelyOptIn: true; + }; + readonly fileCount: number; + readonly totalBytes: number; + readonly operationCount: number; + readonly memorySampleCount: number; + readonly claimCount: number; + /** Skipped (non-parsed) line counts. Excludes `parsed` (successful records). */ + readonly skipped: PerfSkippedCounts; + readonly counts: PerfConsumerCounts; +} + +function hasErrnoCode(err: unknown, code: string): boolean { + return err instanceof Error && (err as NodeJS.ErrnoException).code === code; +} + +/** + * Inspects a perf directory. A missing directory returns zero counts (it is + * a valid empty dataset). Other genuine filesystem errors propagate so the + * caller can fail open. + */ +export async function perfInspect(dir: string): Promise { + // Import here to avoid circular deps at module scope. + const { consumePerfDirectory } = await import('./perfConsumer.js'); + const { entries, counts } = await consumePerfDirectory(dir); + + let operationCount = 0; + let memorySampleCount = 0; + + for (const ce of entries) { + if (ce.entry.kind !== 'ok') continue; + if (ce.entry.record.record_type === 'operation') { + operationCount += 1; + } else { + memorySampleCount += 1; + } + } + + // Count claim files separately (the consumer skips them). + let claimCount = 0; + try { + const names = await fsp.readdir(dir); + claimCount = names.filter(isClaimFile).length; + } catch (err) { + if (!hasErrnoCode(err, 'ENOENT')) throw err; + } + + return { + dir, + schemaVersion: PERF_SCHEMA_VERSION, + privacy: { + localOnly: true, + defaultOff: true, + noUpload: true, + memorySeparatelyOptIn: true, + }, + fileCount: counts.files, + totalBytes: counts.bytes, + operationCount, + memorySampleCount, + claimCount, + skipped: { + malformed: counts.malformed, + futureVersion: counts.futureVersion, + unversioned: counts.unversioned, + truncated: counts.truncated, + blank: counts.blank, + }, + counts, + }; +} + +/** + * Formats an inspect result into a stable, human-readable string. + */ +export function formatInspect(result: PerfInspectResult): string { + const lines: string[] = []; + + lines.push('Perf Inspect'); + lines.push('============'); + lines.push(''); + lines.push(`Directory: ${result.dir}`); + lines.push(`Schema version: ${result.schemaVersion}`); + lines.push(''); + lines.push('Privacy:'); + lines.push(' local-only (never uploaded)'); + lines.push(' default-off'); + lines.push(' memory collection separately opt-in'); + lines.push(''); + lines.push(`Owned JSONL files: ${result.fileCount}`); + lines.push(`Total bytes: ${formatBytes(result.totalBytes)}`); + lines.push(`Claim files: ${result.claimCount}`); + lines.push(''); + lines.push('Record counts:'); + lines.push(` operations: ${result.operationCount}`); + lines.push(` memory samples: ${result.memorySampleCount}`); + lines.push(''); + const s = result.skipped; + lines.push('Skipped breakdown:'); + lines.push(` malformed: ${s.malformed}`); + lines.push(` future version: ${s.futureVersion}`); + lines.push(` unversioned: ${s.unversioned}`); + lines.push(` truncated: ${s.truncated}`); + lines.push(` blank: ${s.blank}`); + + return lines.join('\n'); +} + +function formatBytes(bytes: number): string { + if (bytes < 1024) return `${bytes} B`; + if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KiB`; + return `${(bytes / (1024 * 1024)).toFixed(1)} MiB`; +} diff --git a/packages/telemetry/src/perf/perfPhaseObserver.behavior.test.ts b/packages/telemetry/src/perf/perfPhaseObserver.behavior.test.ts new file mode 100644 index 0000000000..b19063e389 --- /dev/null +++ b/packages/telemetry/src/perf/perfPhaseObserver.behavior.test.ts @@ -0,0 +1,61 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Behavioral tests for the perf phase observer seam (P07, EVIDENCE-AC5). + * + * Proves: default-off (null), set/get, observer callbacks are direct. + * Tests reset the module observer deterministically. + */ + +import { describe, it, expect, afterEach } from 'bun:test'; +import { + setPerfPhaseObserver, + getPerfPhaseObserver, + type PerfPhaseObserver, +} from './perfPhaseObserver.js'; + +describe('PerfPhaseObserver module seam', () => { + afterEach(() => { + setPerfPhaseObserver(null); + }); + + it('returns null by default (default-off)', () => { + setPerfPhaseObserver(null); + expect(getPerfPhaseObserver()).toBeNull(); + }); + + it('returns the installed observer after set', () => { + const observer: PerfPhaseObserver = { + onProviderAttemptStart: () => undefined, + onProviderAttemptEnd: () => undefined, + onToolCallCompleted: () => undefined, + }; + setPerfPhaseObserver(observer); + expect(getPerfPhaseObserver()).toBe(observer); + }); + + it('clears to null after set(null)', () => { + const observer: PerfPhaseObserver = { + onProviderAttemptStart: () => undefined, + onProviderAttemptEnd: () => undefined, + onToolCallCompleted: () => undefined, + }; + setPerfPhaseObserver(observer); + expect(getPerfPhaseObserver()).not.toBeNull(); + setPerfPhaseObserver(null); + expect(getPerfPhaseObserver()).toBeNull(); + }); + + it('disabled logger does not notify (null observer short-circuits callers)', () => { + setPerfPhaseObserver(null); + // Callers check getPerfPhaseObserver() and short-circuit when null. + // Simulating the caller pattern: + const observer = getPerfPhaseObserver(); + expect(observer).toBeNull(); + // No callback is invoked. + }); +}); diff --git a/packages/telemetry/src/perf/perfPhaseObserver.ts b/packages/telemetry/src/perf/perfPhaseObserver.ts new file mode 100644 index 0000000000..64ba4d13d6 --- /dev/null +++ b/packages/telemetry/src/perf/perfPhaseObserver.ts @@ -0,0 +1,121 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Narrow optional perf phase observer seam owned by telemetry (P07, issue #3167). + * + * A module-level subscription point that lower layers (providers' + * AttemptRecorder, telemetry's tool-call logger) invoke at exact lifecycle + * boundaries. The CLI's operation lifecycle registry installs an + * implementation when perf is enabled; when absent (default-off), the getters + * return null and the callers short-circuit with zero allocation. + * + * Layering: telemetry owns the seam. Providers import it (directly or through + * core re-exports); the CLI registry sets it. packages/agents is never + * involved. No agents→telemetry dependency edge is created (AC scope). + * + * D8: observer callbacks are direct and never swallowed. They MUST be invoked + * outside any generic catch-and-log boundary so internal/programming errors + * propagate fail-fast. SDK-disabled mode still notifies (the observer is + * invoked before any SDK/export gate). + */ + +// --------------------------------------------------------------------------- +// Provider attempt lifecycle (AttemptRecorder start/end boundaries) +// --------------------------------------------------------------------------- + +export interface PerfProviderAttemptStartInfo { + /** Stable per-attempt ID (matches AttemptRecorder's attemptId). */ + readonly attemptId: string; + /** + * The AttemptRecorder's logicalRequestId (the agent's prompt/logical request + * identity). The registry derives the operation_id via deriveOperationId so + * the attempt associates to the correct operation — NOT to the foreground op + * by position (concurrent subagent/unrelated requests are not misattributed). + * Continuation IDs collapse via the exact D1 split. + */ + readonly promptId: string; + /** Monotonic timestamp (ms) at the start of the attempt. */ + readonly startMs: number; +} + +export interface PerfProviderAttemptEndInfo { + /** Stable per-attempt ID (matches AttemptRecorder's attemptId). */ + readonly attemptId: string; + /** + * The prompt/logical-request identity (AttemptRecorder.logicalRequestId), + * used to associate the attempt to an operation via deriveOperationId (D1). + */ + readonly promptId: string; + /** Monotonic timestamp (ms) when the attempt started. */ + readonly startMs: number; + /** Monotonic timestamp (ms) at terminal completion. */ + readonly endMs: number; + /** Explicit terminal status. */ + readonly status: 'success' | 'error' | 'aborted'; + /** Input/prompt tokens reported at the boundary, or 0 when unknown. */ + readonly inputTokens: number; + /** Output/completion tokens reported at the boundary, or 0 when unknown. */ + readonly outputTokens: number; +} + +// --------------------------------------------------------------------------- +// Tool call completion (ToolCallEvent logger seam) +// --------------------------------------------------------------------------- + +export interface PerfToolCallCompletedInfo { + /** + * The agent's prompt_id carried by the tool call request. Associated to the + * operation via `deriveOperationId(promptId)` so continuation prompt IDs + * collapse to the shared operation_id (D1). + */ + readonly promptId: string; + /** Unique tool call ID for deduplication, or undefined when absent. */ + readonly callId: string | undefined; + /** Monotonic start timestamp (ms) for interval unioning, or undefined. */ + readonly startMs: number | undefined; + /** Monotonic end timestamp (ms) for interval unioning, or undefined. */ + readonly endMs: number | undefined; + /** Duration (ms) of the tool call. */ + readonly durationMs: number; +} + +// --------------------------------------------------------------------------- +// Observer contract +// --------------------------------------------------------------------------- + +export interface PerfPhaseObserver { + onProviderAttemptStart(info: PerfProviderAttemptStartInfo): void; + onProviderAttemptEnd(info: PerfProviderAttemptEndInfo): void; + onToolCallCompleted(info: PerfToolCallCompletedInfo): void; +} + +// --------------------------------------------------------------------------- +// Module-level seam (default-off: null means no observer installed) +// --------------------------------------------------------------------------- + +let activeObserver: PerfPhaseObserver | null = null; + +/** + * Installs (or clears) the global perf phase observer. The CLI registry calls + * this when perf is enabled; tests call it with null to reset deterministically. + * + * Default-off: when this is never called (or called with null), all getters + * return null and no observer invocation or interval/counter allocation occurs + * in hot telemetry paths. + */ +export function setPerfPhaseObserver(observer: PerfPhaseObserver | null): void { + activeObserver = observer; +} + +/** + * Returns the currently installed perf phase observer, or null when none is + * installed (default-off). Lower-layer callers (AttemptRecorder, tool logger) + * check this and short-circuit when null. + */ +export function getPerfPhaseObserver(): PerfPhaseObserver | null { + return activeObserver; +} diff --git a/packages/telemetry/src/perf/perfReader.join.behavior.test.ts b/packages/telemetry/src/perf/perfReader.join.behavior.test.ts new file mode 100644 index 0000000000..547b408a4c --- /dev/null +++ b/packages/telemetry/src/perf/perfReader.join.behavior.test.ts @@ -0,0 +1,158 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Behavioral tests proving the read-time continuation join (AC-3, D1). + * + * The perf operation record carries only `operation_id` (derived from the + * initial prompt-id prefix). Token-usage / session-recording rows each carry + * their own `prompt_id` (one per send, including continuations). The report + * derives `operation_id` from each token row's `prompt_id` via + * `joinKeyFromPromptId` and joins N continuation rows to the SINGLE perf + * operation — without copying any child id into the perf record. + * + * These tests use the real derivation helpers and a real Map join; no mocks. + */ + +import { describe, it, expect } from 'bun:test'; +import { + deriveOperationId, + joinKeyFromPromptId, + parsePerfRecord, + PerfOperationRecordSchema, +} from './perfRecords.js'; +import type { PerfOperationRecord } from './perfRecords.js'; + +// --------------------------------------------------------------------------- +// A single perf operation record (no child-id arrays — D1) +// --------------------------------------------------------------------------- + +const INITIAL_PROMPT_ID = 'sess-1#agentic-loop#f7e2-aaaa'; + +const PERF_OPERATION_RECORD: Record = { + schema_version: 1, + record_type: 'operation', + ts: '2026-08-08T12:00:00.000Z', + session_id: 'sess-1', + operation_id: deriveOperationId(INITIAL_PROMPT_ID), + runtime_id: 'rt-main', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'sha256:project-hash', + llxprt_version: '0.11.0', + git_sha: 'abc1234', + runtime: 'bun-1.3.14', + platform: 'darwin-arm64', + provider: 'openai', + model: 'gpt-4o', + context_tokens: 1000, + output_tokens: 500, + terminal_cols: 120, + terminal_rows: 40, + render_mode: 'incremental', + concurrent_instances: 1, + status: 'completed', + client_prepare_ms: 5, + stream_handler_ms: 10, + ink_render_ms: 20, + ink_render_count: 3, + stdout_bytes: 4096, + stdout_write_calls: 3, + stdout_write_sync_ms: 2, + client_finalize_ms: 1, + provider_attempts: 1, + provider_attempt_sum_ms: 800, + provider_union_ms: 800, + tool_calls: 2, + tool_call_sum_ms: 300, + tool_union_ms: 280, + agent_activity_union_ms: 1000, + operation_elapsed_ms: 1200, + approval_wait_ms: 0, + unclassified_elapsed_ms: 100, + session_operation_index: 1, + uptime_ms: 50000, +}; + +/** + * Simulated token-usage rows: one prompt_id per send, including the initial + * send and every continuation. These mimic the join input the report receives. + */ +const TOKEN_USAGE_PROMPT_IDS = [ + INITIAL_PROMPT_ID, + `${INITIAL_PROMPT_ID}#continuation#1`, + `${INITIAL_PROMPT_ID}#continuation#2`, + `${INITIAL_PROMPT_ID}#continuation#3`, +]; + +// --------------------------------------------------------------------------- +// Prefix invariant (AC-3 separate behavioural test) +// --------------------------------------------------------------------------- + +describe('operation_id prefix invariant (AC-3)', () => { + it('operation_id equals the initial prompt id (no suffix to strip)', () => { + expect(PERF_OPERATION_RECORD.operation_id).toBe(INITIAL_PROMPT_ID); + }); + + it('every continuation prompt id derives back to the initial prompt id', () => { + for (const promptId of TOKEN_USAGE_PROMPT_IDS) { + expect(joinKeyFromPromptId(promptId)).toBe(INITIAL_PROMPT_ID); + expect(deriveOperationId(promptId)).toBe(INITIAL_PROMPT_ID); + } + }); +}); + +// --------------------------------------------------------------------------- +// Read-time join (AC-3 read-time join evidence, D1) +// --------------------------------------------------------------------------- + +describe('read-time join — N continuation rows → 1 perf operation (D1)', () => { + it('the perf operation record validates and carries no child-id arrays', () => { + const parsed = parsePerfRecord(PERF_OPERATION_RECORD); + expect(parsed).not.toBeNull(); + if (parsed === null) throw new Error('expected a parsed record'); + expect(parsed.record_type).toBe('operation'); + expect('prompt_ids' in parsed).toBe(false); + expect('turn_ids' in parsed).toBe(false); + expect('prompt_ids_total' in parsed).toBe(false); + expect('turn_ids_total' in parsed).toBe(false); + }); + + it('joins every continuation token row to the single perf operation', () => { + // Build the join index the way the report does: operation_id → perf record. + const perfRecord = PerfOperationRecordSchema.parse(PERF_OPERATION_RECORD); + const perfByOperationId = new Map([ + [perfRecord.operation_id, perfRecord], + ]); + + // For each token-usage row, derive the join key and look up the perf op. + const joinedOperations = TOKEN_USAGE_PROMPT_IDS.map((promptId) => { + const operationId = joinKeyFromPromptId(promptId); + return perfByOperationId.get(operationId); + }); + + // Every token row joined to a perf operation. + for (const op of joinedOperations) { + expect(op).toBeDefined(); + } + + // All N rows joined to the SAME single perf operation (by identity). + const uniqueJoined = new Set(joinedOperations); + expect(uniqueJoined.size).toBe(1); + const [sole] = uniqueJoined; + if (sole === undefined) throw new Error('expected one joined operation'); + expect(sole.operation_id).toBe(INITIAL_PROMPT_ID); + + // And no child id was copied into the perf record. + expect('prompt_ids' in sole).toBe(false); + expect('turn_ids' in sole).toBe(false); + }); + + it('does not group an unrelated session under the same operation_id', () => { + const unrelatedPromptId = 'sess-2#agentic-loop#dead-beef'; + expect(joinKeyFromPromptId(unrelatedPromptId)).not.toBe(INITIAL_PROMPT_ID); + }); +}); diff --git a/packages/telemetry/src/perf/perfReader.streaming.behavior.test.ts b/packages/telemetry/src/perf/perfReader.streaming.behavior.test.ts new file mode 100644 index 0000000000..d3f047cc14 --- /dev/null +++ b/packages/telemetry/src/perf/perfReader.streaming.behavior.test.ts @@ -0,0 +1,345 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Behavioral tests for the genuinely streaming perf JSONL reader (P04A + * correction B). + * + * The streaming API must emit parsed records / classification outcomes + * incrementally — it must NOT accumulate the entire file in memory before + * yielding the first entry. P11 must be able to use this API for a 24/7 file + * that never closes. + * + * All tests use real files and the package-private readable-stream seam — + * no source-text assertions, no mocks. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { Readable } from 'node:stream'; +import { streamPerfRecords, type PerfStreamEntry } from './perfRecords.js'; +// The controlled-readable seam is package-private (not exported from +// package.json); same-package tests import it directly from the internal module. +import { streamPerfFromReadable } from './perfRecordsStream.js'; + +// --------------------------------------------------------------------------- +// Temp-file helper +// --------------------------------------------------------------------------- + +function makeTempFile(content: string): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'perf-stream-')); + const filePath = path.join(dir, 'perf.jsonl'); + fs.writeFileSync(filePath, content, 'utf8'); + return filePath; +} + +function cleanupFile(filePath: string): void { + try { + fs.rmSync(path.dirname(filePath), { recursive: true, force: true }); + } catch { + // best-effort + } +} + +// --------------------------------------------------------------------------- +// Valid record factory +// --------------------------------------------------------------------------- + +function operationLine(overrides: Record = {}): string { + const record = { + schema_version: 1, + record_type: 'operation', + ts: '2026-08-08T12:00:00.000Z', + session_id: 'sess-abc', + operation_id: 'sess-abc#agentic-loop#f7e2', + runtime_id: 'rt-main', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'sha256:project-hash', + llxprt_version: '0.11.0', + git_sha: 'abc1234', + runtime: 'bun-1.3.14', + platform: 'darwin-arm64', + provider: 'openai', + model: 'gpt-4o', + context_tokens: 1000, + output_tokens: 500, + terminal_cols: 120, + terminal_rows: 40, + render_mode: 'incremental', + concurrent_instances: 1, + status: 'completed', + client_prepare_ms: 5, + stream_handler_ms: 10, + ink_render_ms: 20, + ink_render_count: 3, + stdout_bytes: 4096, + stdout_write_calls: 3, + stdout_write_sync_ms: 2, + client_finalize_ms: 1, + provider_attempts: 1, + provider_attempt_sum_ms: 800, + provider_union_ms: 800, + tool_calls: 2, + tool_call_sum_ms: 300, + tool_union_ms: 280, + agent_activity_union_ms: 1000, + operation_elapsed_ms: 1200, + approval_wait_ms: 0, + unclassified_elapsed_ms: 100, + session_operation_index: 1, + uptime_ms: 50000, + ...overrides, + }; + return JSON.stringify(record); +} + +// --------------------------------------------------------------------------- +// streamPerfRecords — real file iteration +// --------------------------------------------------------------------------- + +describe('streamPerfRecords — real file', () => { + let filePath: string; + + beforeEach(() => { + filePath = makeTempFile( + [operationLine(), operationLine(), operationLine()].join('\n') + '\n', + ); + }); + + afterEach(() => cleanupFile(filePath)); + + it('yields one ok entry per valid record', async () => { + const entries: PerfStreamEntry[] = []; + for await (const entry of streamPerfRecords(filePath)) { + entries.push(entry); + } + expect(entries).toHaveLength(3); + for (const entry of entries) { + expect(entry.kind).toBe('ok'); + } + }); + + it('yields classification outcomes for non-ok lines', async () => { + const tmpFile = makeTempFile( + operationLine() + + '\n' + + '{not valid json\n' + + JSON.stringify({ ...JSON.parse(operationLine()), schema_version: 99 }) + + '\n' + + operationLine() + + '\n', + ); + try { + const entries: PerfStreamEntry[] = []; + for await (const entry of streamPerfRecords(tmpFile)) { + entries.push(entry); + } + const kinds = entries.map((e) => e.kind); + expect(kinds).toEqual(['ok', 'malformed', 'future_version', 'ok']); + } finally { + cleanupFile(tmpFile); + } + }); + + it('handles an empty file', async () => { + const empty = makeTempFile(''); + try { + const entries: PerfStreamEntry[] = []; + for await (const entry of streamPerfRecords(empty)) { + entries.push(entry); + } + expect(entries).toHaveLength(0); + } finally { + cleanupFile(empty); + } + }); + + it('yields a whitespace-only final line without a trailing newline as blank', async () => { + const tmpFile = makeTempFile(' '); + try { + const entries: PerfStreamEntry[] = []; + for await (const entry of streamPerfRecords(tmpFile)) { + entries.push(entry); + } + + expect(entries).toEqual([{ kind: 'blank' }]); + } finally { + cleanupFile(tmpFile); + } + }); + + it('classifies a truncated final line as truncated', async () => { + const tmpFile = makeTempFile( + operationLine() + '\n' + '{"schema_version":1,"record_type":"operation",', + ); + try { + const entries: PerfStreamEntry[] = []; + for await (const entry of streamPerfRecords(tmpFile)) { + entries.push(entry); + } + expect(entries.map((e) => e.kind)).toEqual(['ok', 'truncated']); + } finally { + cleanupFile(tmpFile); + } + }); +}); + +// Issue #3167 (P04): a final line with NO trailing newline keeps its CONTENT +// classification. Only a final line that is not valid JSON is "truncated". +// A final line that parses as JSON keeps future_version / unversioned / +// malformed. Each file below has one complete valid record, then the final +// line under test with no trailing newline. +describe('streamPerfRecords — final line without trailing newline classification (issue #3167)', () => { + it('yields future_version for a final future-version JSON line without a newline', async () => { + const future = JSON.stringify({ + ...JSON.parse(operationLine()), + schema_version: 99, + }); + const tmpFile = makeTempFile(operationLine() + '\n' + future); + try { + const entries: PerfStreamEntry[] = []; + for await (const entry of streamPerfRecords(tmpFile)) { + entries.push(entry); + } + expect(entries.map((e) => e.kind)).toEqual(['ok', 'future_version']); + } finally { + cleanupFile(tmpFile); + } + }); + + it('yields unversioned for a final unversioned JSON line without a newline', async () => { + const unversioned = JSON.stringify({ + some_legacy_field: 1, + ts: '2026-01-01T00:00:00.000Z', + }); + const tmpFile = makeTempFile(operationLine() + '\n' + unversioned); + try { + const entries: PerfStreamEntry[] = []; + for await (const entry of streamPerfRecords(tmpFile)) { + entries.push(entry); + } + expect(entries.map((e) => e.kind)).toEqual(['ok', 'unversioned']); + } finally { + cleanupFile(tmpFile); + } + }); + + it('yields malformed for a final valid-JSON current-version record missing fields without a newline', async () => { + const malformed = JSON.stringify({ + schema_version: 1, + record_type: 'operation', + }); + const tmpFile = makeTempFile(operationLine() + '\n' + malformed); + try { + const entries: PerfStreamEntry[] = []; + for await (const entry of streamPerfRecords(tmpFile)) { + entries.push(entry); + } + expect(entries.map((e) => e.kind)).toEqual(['ok', 'malformed']); + } finally { + cleanupFile(tmpFile); + } + }); + + it('yields truncated for a final invalid/partial JSON line without a newline', async () => { + const partial = '{"schema_version":1,"record_type":"operation",'; + const tmpFile = makeTempFile(operationLine() + '\n' + partial); + try { + const entries: PerfStreamEntry[] = []; + for await (const entry of streamPerfRecords(tmpFile)) { + entries.push(entry); + } + expect(entries.map((e) => e.kind)).toEqual(['ok', 'truncated']); + } finally { + cleanupFile(tmpFile); + } + }); +}); + +// --------------------------------------------------------------------------- +// streamPerfFromReadable — incremental yield proof +// --------------------------------------------------------------------------- + +describe('streamPerfFromReadable — incremental yield proof', () => { + it('yields the first record before the second chunk is pushed', async () => { + const line1 = operationLine({ session_operation_index: 1 }) + '\n'; + const line2 = operationLine({ session_operation_index: 2 }) + '\n'; + + // A manually-controlled readable. We push chunks one at a time and verify + // the iterator yields from chunk 1 before chunk 2 is even available. + const readable = new Readable({ read() {} }); + + const iter = streamPerfFromReadable(readable); + + // Push first chunk only. + readable.push(Buffer.from(line1)); + + // Pull the first entry — it MUST resolve from chunk 1 alone. + const first = await iter.next(); + expect(first.done).toBe(false); + expect(first.value?.kind).toBe('ok'); + if (first.value?.kind !== 'ok') throw new Error('unreachable'); + expect(first.value.record.record_type).toBe('operation'); + + // NOW push the second chunk and close the stream. + readable.push(Buffer.from(line2)); + readable.push(null); + + const second = await iter.next(); + expect(second.done).toBe(false); + expect(second.value?.kind).toBe('ok'); + + const third = await iter.next(); + expect(third.done).toBe(true); + }); + + it('correctly processes a large file with many records', async () => { + // Create a real large file (many records). + const N = 5000; + const lines: string[] = []; + for (let i = 0; i < N; i++) { + lines.push(operationLine({ session_operation_index: i })); + } + const filePath = makeTempFile(lines.join('\n') + '\n'); + try { + let count = 0; + let firstIndex = -1; + for await (const entry of streamPerfRecords(filePath)) { + if (entry.kind !== 'ok') continue; + count++; + if (count === 1 && entry.record.record_type === 'operation') { + firstIndex = entry.record.session_operation_index; + } + } + expect(count).toBe(N); + expect(firstIndex).toBe(0); + } finally { + cleanupFile(filePath); + } + }); + + it('proves streaming by interleaving pushes and pulls', async () => { + // Push 3 lines one at a time, pulling between each push. + const readable = new Readable({ read() {} }); + const iter = streamPerfFromReadable(readable); + + for (let i = 0; i < 3; i++) { + readable.push( + Buffer.from(operationLine({ session_operation_index: i }) + '\n'), + ); + const result = await iter.next(); + expect(result.done).toBe(false); + expect(result.value?.kind).toBe('ok'); + } + + readable.push(null); + const done = await iter.next(); + expect(done.done).toBe(true); + }); +}); diff --git a/packages/telemetry/src/perf/perfReader.tolerant.behavior.test.ts b/packages/telemetry/src/perf/perfReader.tolerant.behavior.test.ts new file mode 100644 index 0000000000..c80d094d3b --- /dev/null +++ b/packages/telemetry/src/perf/perfReader.tolerant.behavior.test.ts @@ -0,0 +1,428 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Behavioral tests for the tolerant streaming perf JSONL reader (AC-9 reader + * partial). Every case writes a REAL temporary JSONL file and asserts on the + * actual parsed records and self-health counters — no mocks. + * + * The reader must: stream lines (never read the whole file), tolerate + * malformed/truncated final lines with explicit counters, ignore unknown + * fields on known schema versions, skip+count future schema versions without + * coercion, and never throw for malformed external JSONL. It must distinguish + * a malformed complete line from a truncated final line. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { readPerfRecords } from './perfRecords.js'; + +// --------------------------------------------------------------------------- +// Temp-file helper (real files, real fs) +// --------------------------------------------------------------------------- + +function makeTempFile(content: string): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'perf-reader-')); + const filePath = path.join(dir, 'perf.jsonl'); + fs.writeFileSync(filePath, content, 'utf8'); + return filePath; +} + +function cleanupFile(filePath: string): void { + try { + fs.rmSync(path.dirname(filePath), { recursive: true, force: true }); + } catch { + // best-effort + } +} + +// --------------------------------------------------------------------------- +// Valid record factories (plain objects; the real schema validates them) +// --------------------------------------------------------------------------- + +function operationLine(overrides: Record = {}): string { + const record = { + schema_version: 1, + record_type: 'operation', + ts: '2026-08-08T12:00:00.000Z', + session_id: 'sess-abc', + operation_id: 'sess-abc#agentic-loop#f7e2', + runtime_id: 'rt-main', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'sha256:project-hash', + llxprt_version: '0.11.0', + git_sha: 'abc1234', + runtime: 'bun-1.3.14', + platform: 'darwin-arm64', + provider: 'openai', + model: 'gpt-4o', + context_tokens: 1000, + output_tokens: 500, + terminal_cols: 120, + terminal_rows: 40, + render_mode: 'incremental', + concurrent_instances: 1, + status: 'completed', + client_prepare_ms: 5, + stream_handler_ms: 10, + ink_render_ms: 20, + ink_render_count: 3, + stdout_bytes: 4096, + stdout_write_calls: 3, + stdout_write_sync_ms: 2, + client_finalize_ms: 1, + provider_attempts: 1, + provider_attempt_sum_ms: 800, + provider_union_ms: 800, + tool_calls: 2, + tool_call_sum_ms: 300, + tool_union_ms: 280, + agent_activity_union_ms: 1000, + operation_elapsed_ms: 1200, + approval_wait_ms: 0, + unclassified_elapsed_ms: 100, + session_operation_index: 1, + uptime_ms: 50000, + ...overrides, + }; + return JSON.stringify(record); +} + +describe('readPerfRecords — happy path', () => { + let filePath: string; + + beforeEach(() => { + filePath = makeTempFile( + [operationLine(), operationLine(), operationLine()].join('\n') + '\n', + ); + }); + + afterEach(() => cleanupFile(filePath)); + + it('parses every valid operation record in the file', async () => { + const { records, counts } = await readPerfRecords(filePath); + expect(records).toHaveLength(3); + expect(counts.parsed).toBe(3); + for (const record of records) { + expect(record.record_type).toBe('operation'); + } + }); + + it('reports zero for every non-parsed counter on a clean file', async () => { + const { counts } = await readPerfRecords(filePath); + expect(counts.malformed).toBe(0); + expect(counts.futureVersion).toBe(0); + expect(counts.unversioned).toBe(0); + expect(counts.truncated).toBe(0); + expect(counts.blank).toBe(0); + }); +}); + +describe('readPerfRecords — unknown fields ignored (§2)', () => { + let filePath: string; + + beforeEach(() => { + filePath = makeTempFile(operationLine({ future_metric_ms: 42 }) + '\n'); + }); + + afterEach(() => cleanupFile(filePath)); + + it('parses a known-version record carrying an unknown field', async () => { + const { records, counts } = await readPerfRecords(filePath); + expect(records).toHaveLength(1); + expect(counts.parsed).toBe(1); + expect(counts.malformed).toBe(0); + }); +}); + +describe('readPerfRecords — future schema version (skip+count)', () => { + let filePath: string; + + beforeEach(() => { + filePath = makeTempFile( + operationLine() + + '\n' + + JSON.stringify({ + ...JSON.parse(operationLine()), + schema_version: 99, + record_type: 'operation', + }) + + '\n' + + operationLine() + + '\n', + ); + }); + + afterEach(() => cleanupFile(filePath)); + + it('skips and counts the future-version record without coercing it', async () => { + const { records, counts } = await readPerfRecords(filePath); + expect(records).toHaveLength(2); + expect(counts.parsed).toBe(2); + expect(counts.futureVersion).toBe(1); + expect(counts.malformed).toBe(0); + }); +}); + +describe('readPerfRecords — malformed complete line', () => { + let filePath: string; + + beforeEach(() => { + filePath = makeTempFile( + operationLine() + '\n' + '{not valid json\n' + operationLine() + '\n', + ); + }); + + afterEach(() => cleanupFile(filePath)); + + it('counts the malformed line and continues parsing the rest', async () => { + const { records, counts } = await readPerfRecords(filePath); + expect(records).toHaveLength(2); + expect(counts.parsed).toBe(2); + expect(counts.malformed).toBe(1); + expect(counts.truncated).toBe(0); + }); +}); + +describe('readPerfRecords — unversioned legacy record', () => { + let filePath: string; + + beforeEach(() => { + filePath = makeTempFile( + operationLine() + + '\n' + + JSON.stringify({ + some_legacy_field: 1, + ts: '2026-01-01T00:00:00.000Z', + }) + + '\n', + ); + }); + + afterEach(() => cleanupFile(filePath)); + + it('counts the unversioned record rather than fake-normalizing it', async () => { + const { records, counts } = await readPerfRecords(filePath); + expect(records).toHaveLength(1); + expect(counts.parsed).toBe(1); + expect(counts.unversioned).toBe(1); + expect(counts.malformed).toBe(0); + }); +}); + +describe('readPerfRecords — truncated final line (SIGKILL mid-append)', () => { + it('counts a partial-JSON final line as truncated', async () => { + // A complete valid line, then a partial JSON line with NO trailing newline. + const filePath = makeTempFile( + operationLine() + '\n' + '{"schema_version":1,"record_type":"operation",', + ); + try { + const { records, counts } = await readPerfRecords(filePath); + expect(records).toHaveLength(1); + expect(counts.parsed).toBe(1); + expect(counts.truncated).toBe(1); + expect(counts.malformed).toBe(0); + } finally { + cleanupFile(filePath); + } + }); + + it('parses a complete final record that simply lacks a trailing newline', async () => { + const filePath = makeTempFile(operationLine()); + try { + const { records, counts } = await readPerfRecords(filePath); + expect(records).toHaveLength(1); + expect(counts.parsed).toBe(1); + expect(counts.truncated).toBe(0); + } finally { + cleanupFile(filePath); + } + }); +}); + +describe('readPerfRecords — blank lines', () => { + let filePath: string; + + beforeEach(() => { + filePath = makeTempFile( + '\n' + operationLine() + '\n\n\n' + operationLine() + '\n', + ); + }); + + afterEach(() => cleanupFile(filePath)); + + it('skips and counts blank lines without treating them as malformed', async () => { + const { records, counts } = await readPerfRecords(filePath); + expect(records).toHaveLength(2); + expect(counts.parsed).toBe(2); + expect(counts.blank).toBe(3); + expect(counts.malformed).toBe(0); + }); + + it('counts a whitespace-only final line without a trailing newline', async () => { + cleanupFile(filePath); + filePath = makeTempFile(operationLine() + '\n '); + + const { records, counts } = await readPerfRecords(filePath); + + expect(records).toHaveLength(1); + expect(counts.parsed).toBe(1); + expect(counts.blank).toBe(1); + expect(counts.truncated).toBe(0); + }); +}); + +describe('readPerfRecords — mixed real-world fileset (EVIDENCE-AC9)', () => { + it('reports accurate per-category counters on a mixed file', async () => { + const futureRecord = JSON.stringify({ + ...JSON.parse(operationLine()), + schema_version: 5, + }); + // Complete (newline-terminated) lines, then a final truncated line with + // NO trailing newline (simulating SIGKILL mid-append). + const completeLines = [ + operationLine(), + JSON.stringify({ + some_legacy_field: 1, + ts: '2026-01-01T00:00:00.000Z', + }), + '{broken json', + operationLine({ status: 'superseded' }), + futureRecord, + '', + ].join('\n'); + const filePath = makeTempFile( + completeLines + + '\n' + + '{"schema_version":1,"record_type":"operation","ts":"tr', + ); + try { + const { records, counts } = await readPerfRecords(filePath); + // Two clean operation records parsed; the rest are counted by category. + expect(records).toHaveLength(2); + expect(counts.parsed).toBe(2); + expect(counts.unversioned).toBe(1); + expect(counts.malformed).toBe(1); + expect(counts.futureVersion).toBe(1); + expect(counts.blank).toBe(1); + expect(counts.truncated).toBe(1); + // records only ever contains successfully parsed records + for (const record of records) { + expect(record.schema_version).toBe(1); + } + } finally { + cleanupFile(filePath); + } + }); +}); + +describe('readPerfRecords — never throws on malformed external JSONL', () => { + it('does not reject on a file full of garbage', async () => { + const filePath = makeTempFile('garbage\ngarbage2\n{still bad\n123\n'); + try { + const result = await readPerfRecords(filePath); + expect(result.counts.parsed).toBe(0); + // 123 parses as JSON but is not a perf record → malformed + expect(result.counts.malformed).toBe(4); + } finally { + cleanupFile(filePath); + } + }); +}); + +describe('readPerfRecords — empty file', () => { + it('returns no records and zero counters for an empty file', async () => { + const filePath = makeTempFile(''); + try { + const { records, counts } = await readPerfRecords(filePath); + expect(records).toHaveLength(0); + expect(counts.parsed).toBe(0); + expect(counts.truncated).toBe(0); + expect(counts.malformed).toBe(0); + } finally { + cleanupFile(filePath); + } + }); +}); + +// Issue #3167 (P04): a final line with NO trailing newline must be classified +// by its CONTENT, not by the missing newline. Only a final line that is not +// valid JSON is "truncated" (realistic cause: SIGKILL mid-append). A final +// line that parses as JSON keeps its content classification (future_version, +// unversioned, ok, or malformed for a current-version record missing required +// fields). Each file below has one complete valid record, then the final line +// under test with no trailing newline. +describe('readPerfRecords — final line without trailing newline (issue #3167)', () => { + it('classifies a final future-version JSON line as future_version, not truncated', async () => { + const future = JSON.stringify({ + ...JSON.parse(operationLine()), + schema_version: 99, + }); + const filePath = makeTempFile(operationLine() + '\n' + future); + try { + const { records, counts } = await readPerfRecords(filePath); + expect(records).toHaveLength(1); + expect(counts.parsed).toBe(1); + expect(counts.futureVersion).toBe(1); + expect(counts.truncated).toBe(0); + expect(counts.malformed).toBe(0); + } finally { + cleanupFile(filePath); + } + }); + + it('classifies a final unversioned JSON line as unversioned, not truncated', async () => { + const unversioned = JSON.stringify({ + some_legacy_field: 1, + ts: '2026-01-01T00:00:00.000Z', + }); + const filePath = makeTempFile(operationLine() + '\n' + unversioned); + try { + const { records, counts } = await readPerfRecords(filePath); + expect(records).toHaveLength(1); + expect(counts.parsed).toBe(1); + expect(counts.unversioned).toBe(1); + expect(counts.truncated).toBe(0); + expect(counts.malformed).toBe(0); + } finally { + cleanupFile(filePath); + } + }); + + it('classifies a final valid-JSON current-version record missing fields as malformed, not truncated', async () => { + const malformed = JSON.stringify({ + schema_version: 1, + record_type: 'operation', + }); + const filePath = makeTempFile(operationLine() + '\n' + malformed); + try { + const { records, counts } = await readPerfRecords(filePath); + expect(records).toHaveLength(1); + expect(counts.parsed).toBe(1); + expect(counts.malformed).toBe(1); + expect(counts.truncated).toBe(0); + } finally { + cleanupFile(filePath); + } + }); + + it('classifies a final invalid/partial JSON line as truncated', async () => { + const partial = '{"schema_version":1,"record_type":"operation",'; + const filePath = makeTempFile(operationLine() + '\n' + partial); + try { + const { records, counts } = await readPerfRecords(filePath); + expect(records).toHaveLength(1); + expect(counts.parsed).toBe(1); + expect(counts.truncated).toBe(1); + expect(counts.malformed).toBe(0); + } finally { + cleanupFile(filePath); + } + }); +}); diff --git a/packages/telemetry/src/perf/perfRecordSize.bench.ts b/packages/telemetry/src/perf/perfRecordSize.bench.ts new file mode 100644 index 0000000000..6293a32f22 --- /dev/null +++ b/packages/telemetry/src/perf/perfRecordSize.bench.ts @@ -0,0 +1,121 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Record-size benchmark for the actual v1 perf schema (D5). + * + * Serializes a representative `operation` record (INCLUDING the optional + * memory columns) and a `memory_sample` record from the real schema, + * validates both, and reports the byte size of a single JSONL line. P08 uses + * this output to derive retention constants (max-bytes / max-files / + * maintenance-interval / diagnostic-rate-limit). + * + * Run: `bun packages/telemetry/src/perf/perfRecordSize.bench.ts` + * + * This is a measurement script, not a test file; it exits non-zero if either + * record fails to validate. + */ + +import { + PERF_SCHEMA_VERSION, + PerfRecordSchema, + type PerfOperationRecord, + type PerfMemorySampleRecord, +} from './perfRecords.js'; + +// Representative operation record WITH the optional memory columns, so the +// measured line is the largest single-record shape a writer produces. +const operationWithMemory: PerfOperationRecord = { + schema_version: PERF_SCHEMA_VERSION, + record_type: 'operation', + ts: '2026-08-08T12:00:00.000Z', + session_id: 'sess-abc1234', + operation_id: 'sess-abc1234#agentic-loop#f7e2-9a8b-7c6d-5e4f', + runtime_id: 'rt-550e8400-e29b-41d4-a716-446655440000', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'sha256:1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b', + llxprt_version: '0.11.0', + git_sha: 'abc1234', + runtime: 'bun-1.3.14', + platform: 'darwin-arm64', + provider: 'openai', + model: 'gpt-4o', + context_tokens: 48213, + output_tokens: 1842, + terminal_cols: 120, + terminal_rows: 40, + render_mode: 'incremental', + concurrent_instances: 2, + status: 'completed', + client_prepare_ms: 3, + stream_handler_ms: 47, + ink_render_ms: 128, + ink_render_count: 312, + stdout_bytes: 2845621, + stdout_write_calls: 312, + stdout_write_sync_ms: 91, + client_finalize_ms: 1, + provider_attempts: 1, + provider_attempt_sum_ms: 8421, + provider_union_ms: 8421, + tool_calls: 7, + tool_call_sum_ms: 14302, + tool_union_ms: 12884, + agent_activity_union_ms: 19840, + operation_elapsed_ms: 31204, + approval_wait_ms: 4210, + unclassified_elapsed_ms: 611, + rss_bytes: 312_547_840, + heap_used_bytes: 184_322_560, + external_bytes: 47_001_088, + array_buffers_bytes: 2_490_368, + session_operation_index: 14, + uptime_ms: 1_842_317, +}; + +const memorySample: PerfMemorySampleRecord = { + schema_version: PERF_SCHEMA_VERSION, + record_type: 'memory_sample', + ts: '2026-08-08T12:01:00.000Z', + rss_bytes: 318_004_224, + heap_used_bytes: 186_122_240, + external_bytes: 47_312_896, + array_buffers_bytes: 2_501_888, + uptime_ms: 1_902_317, + ms_since_last_operation: 124_013, +}; + +function assertValid(label: string, record: unknown): void { + const result = PerfRecordSchema.safeParse(record); + if (!result.success) { + process.stderr.write(`${label} FAILED schema validation:\n`); + for (const issue of result.error.issues) { + process.stderr.write(` ${issue.path.join('.')}: ${issue.message}\n`); + } + process.exit(1); + } +} + +assertValid('operation (with memory)', operationWithMemory); +assertValid('memory_sample', memorySample); + +const operationLine = JSON.stringify(operationWithMemory) + '\n'; +const memorySampleLine = JSON.stringify(memorySample) + '\n'; + +const operationBytes = Buffer.byteLength(operationLine, 'utf8'); +const memorySampleBytes = Buffer.byteLength(memorySampleLine, 'utf8'); + +process.stdout.write('perf record-size benchmark (D5) — actual v1 schema\n'); +process.stdout.write( + ` operation (with memory) JSONL line : ${operationBytes} bytes (${operationLine.length} chars)\n`, +); +process.stdout.write( + ` memory_sample JSONL line : ${memorySampleBytes} bytes (${memorySampleLine.length} chars)\n`, +); +process.stdout.write( + ` combined per operation pair : ${operationBytes + memorySampleBytes} bytes\n`, +); diff --git a/packages/telemetry/src/perf/perfRecords.behavior.test.ts b/packages/telemetry/src/perf/perfRecords.behavior.test.ts new file mode 100644 index 0000000000..48a36b319d --- /dev/null +++ b/packages/telemetry/src/perf/perfRecords.behavior.test.ts @@ -0,0 +1,510 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Behavioral tests for the v1 perf record schema, operation-id derivation, + * and the tolerant per-line classifier/reader (AC-1 schema half, AC-3 + * derivation, AC-9 reader classification). + * + * These tests exercise the real Zod schema and the real classify/parse + * functions — no mocks. A round-trip asserts against output produced by the + * actual schema, never a hand-authored fixture shape. + */ + +import { describe, it, expect } from 'bun:test'; +import { + PERF_SCHEMA_VERSION, + PERF_RECORD_TYPE_OPERATION, + PERF_RECORD_TYPE_MEMORY_SAMPLE, + PERF_TERMINAL_STATUSES, + PerfOperationRecordSchema, + PerfMemorySampleRecordSchema, + PerfRecordSchema, + deriveOperationId, + joinKeyFromPromptId, + parsePerfRecord, + classifyPerfLine, +} from './perfRecords.js'; + +// --------------------------------------------------------------------------- +// Record builders (test data only — produce plain objects validated by the +// real schema under test) +// --------------------------------------------------------------------------- + +function operationRecord( + overrides: Record = {}, +): Record { + return { + schema_version: 1, + record_type: 'operation', + ts: '2026-08-08T12:00:00.000Z', + session_id: 'sess-abc', + operation_id: 'sess-abc#agentic-loop#f7e2', + runtime_id: 'rt-main', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'sha256:project-hash', + llxprt_version: '0.11.0', + git_sha: 'abc1234', + runtime: 'bun-1.3.14', + platform: 'darwin-arm64', + provider: 'openai', + model: 'gpt-4o', + context_tokens: 1000, + output_tokens: 500, + terminal_cols: 120, + terminal_rows: 40, + render_mode: 'incremental', + concurrent_instances: 1, + status: 'completed', + client_prepare_ms: 5, + stream_handler_ms: 10, + ink_render_ms: 20, + ink_render_count: 3, + stdout_bytes: 4096, + stdout_write_calls: 3, + stdout_write_sync_ms: 2, + client_finalize_ms: 1, + provider_attempts: 1, + provider_attempt_sum_ms: 800, + provider_union_ms: 800, + tool_calls: 2, + tool_call_sum_ms: 300, + tool_union_ms: 280, + agent_activity_union_ms: 1000, + operation_elapsed_ms: 1200, + approval_wait_ms: 0, + unclassified_elapsed_ms: 100, + session_operation_index: 1, + uptime_ms: 50000, + ...overrides, + }; +} + +function operationRecordWithMemory( + overrides: Record = {}, +): Record { + return operationRecord({ + rss_bytes: 120_000_000, + heap_used_bytes: 60_000_000, + external_bytes: 25_000_000, + array_buffers_bytes: 1_500_000, + ...overrides, + }); +} + +function memorySampleRecord( + overrides: Record = {}, +): Record { + return { + schema_version: 1, + record_type: 'memory_sample', + ts: '2026-08-08T12:01:00.000Z', + rss_bytes: 121_000_000, + heap_used_bytes: 61_000_000, + external_bytes: 25_500_000, + array_buffers_bytes: 1_550_000, + uptime_ms: 60000, + ms_since_last_operation: 30000, + ...overrides, + }; +} + +// --------------------------------------------------------------------------- +// Schema version + record-type constants +// --------------------------------------------------------------------------- + +describe('PERF_SCHEMA_VERSION and record-type constants', () => { + it('is version 1', () => { + expect(PERF_SCHEMA_VERSION).toBe(1); + }); + + it('declares the operation discriminator string', () => { + expect(PERF_RECORD_TYPE_OPERATION).toBe('operation'); + }); + + it('declares the memory_sample discriminator string', () => { + expect(PERF_RECORD_TYPE_MEMORY_SAMPLE).toBe('memory_sample'); + }); +}); + +// --------------------------------------------------------------------------- +// Operation record schema (AC-1 schema half) +// --------------------------------------------------------------------------- + +describe('PerfOperationRecordSchema (AC-1 schema half)', () => { + it('validates a complete operation record without memory fields', () => { + const result = PerfOperationRecordSchema.safeParse(operationRecord()); + expect(result.success).toBe(true); + }); + + it('validates an operation record with the optional memory columns', () => { + const result = PerfOperationRecordSchema.safeParse( + operationRecordWithMemory(), + ); + expect(result.success).toBe(true); + }); + + it('round-trips every field of a full (with-memory) operation record', () => { + const source = operationRecordWithMemory({ + status: 'cancelled_during_tool', + unclassified_elapsed_ms: 246, + external_bytes: 99, + }); + const result = PerfOperationRecordSchema.safeParse(source); + expect(result.success).toBe(true); + if (!result.success) throw new Error('expected parse success'); + expect(result.data).toMatchObject({ + record_type: 'operation', + schema_version: 1, + status: 'cancelled_during_tool', + unclassified_elapsed_ms: 246, + external_bytes: 99, + operation_id: 'sess-abc#agentic-loop#f7e2', + runtime_id: 'rt-main', + parent_runtime_id: null, + subagent_name: null, + concurrent_instances: 1, + render_mode: 'incremental', + }); + }); + + it('omits memory fields when they are absent (not zero-filled)', () => { + const result = PerfOperationRecordSchema.safeParse(operationRecord()); + expect(result.success).toBe(true); + if (!result.success) throw new Error('expected parse success'); + expect('rss_bytes' in result.data).toBe(false); + expect('heap_used_bytes' in result.data).toBe(false); + expect('external_bytes' in result.data).toBe(false); + expect('array_buffers_bytes' in result.data).toBe(false); + }); + + it('strips unknown fields (a field addition is not a version bump — §2)', () => { + const result = PerfOperationRecordSchema.safeParse( + operationRecord({ future_metric_ms: 42, another_new_field: 'x' }), + ); + expect(result.success).toBe(true); + if (!result.success) throw new Error('expected parse success'); + expect('future_metric_ms' in result.data).toBe(false); + expect('another_new_field' in result.data).toBe(false); + }); + + it('rejects an empty operation_id at the schema boundary', () => { + const result = PerfOperationRecordSchema.safeParse( + operationRecord({ operation_id: '' }), + ); + expect(result.success).toBe(false); + }); + + it('rejects an empty session_id at the schema boundary', () => { + const result = PerfOperationRecordSchema.safeParse( + operationRecord({ session_id: '' }), + ); + expect(result.success).toBe(false); + }); + + it('rejects an empty provider string', () => { + const result = PerfOperationRecordSchema.safeParse( + operationRecord({ provider: '' }), + ); + expect(result.success).toBe(false); + }); + + it('accepts a non-null subagent_name for subagent records', () => { + const result = PerfOperationRecordSchema.safeParse( + operationRecord({ + runtime_id: 'rt-sub', + parent_runtime_id: 'rt-main', + subagent_name: 'researcher', + }), + ); + expect(result.success).toBe(true); + }); +}); + +// --------------------------------------------------------------------------- +// Seven terminal statuses (AC-4, spec §1.3) +// --------------------------------------------------------------------------- + +describe('terminal statuses (AC-4)', () => { + it('declares exactly the seven terminal values including superseded', () => { + expect(PERF_TERMINAL_STATUSES).toEqual([ + 'completed', + 'error', + 'cancelled_before_send', + 'cancelled_during_api', + 'cancelled_during_tool', + 'cancelled_during_approval', + 'superseded', + ]); + }); + + for (const status of PERF_TERMINAL_STATUSES) { + it(`accepts status "${status}"`, () => { + const result = PerfOperationRecordSchema.safeParse( + operationRecord({ status }), + ); + expect(result.success).toBe(true); + }); + } + + it('rejects an unknown status value', () => { + const result = PerfOperationRecordSchema.safeParse( + operationRecord({ status: 'paused' }), + ); + expect(result.success).toBe(false); + }); +}); + +// --------------------------------------------------------------------------- +// D1: no child-id arrays / true-count / cap fields on the perf record +// --------------------------------------------------------------------------- + +describe('D1 — no child-id arrays on the perf record', () => { + it('strips prompt_ids/turn_ids/totals if present (they are not schema fields)', () => { + const parsed = parsePerfRecord( + operationRecord({ + prompt_ids: ['child-1', 'child-2'], + turn_ids: ['turn-1'], + prompt_ids_total: 2, + turn_ids_total: 1, + }), + ); + expect(parsed).not.toBeNull(); + if (parsed === null) throw new Error('expected a parsed record'); + expect('prompt_ids' in parsed).toBe(false); + expect('turn_ids' in parsed).toBe(false); + expect('prompt_ids_total' in parsed).toBe(false); + expect('turn_ids_total' in parsed).toBe(false); + }); +}); + +// --------------------------------------------------------------------------- +// Memory sample record schema (§7.2) +// --------------------------------------------------------------------------- + +describe('PerfMemorySampleRecordSchema (§7.2)', () => { + it('validates a complete memory_sample record', () => { + const result = PerfMemorySampleRecordSchema.safeParse(memorySampleRecord()); + expect(result.success).toBe(true); + }); + + it('round-trips the memory values and idle marker', () => { + const result = PerfMemorySampleRecordSchema.safeParse( + memorySampleRecord({ + ms_since_last_operation: 120000, + uptime_ms: 180000, + }), + ); + expect(result.success).toBe(true); + if (!result.success) throw new Error('expected parse success'); + expect(result.data).toMatchObject({ + record_type: 'memory_sample', + schema_version: 1, + ms_since_last_operation: 120000, + uptime_ms: 180000, + }); + }); + + it('requires the four memory values', () => { + const result = PerfMemorySampleRecordSchema.safeParse({ + schema_version: 1, + record_type: 'memory_sample', + ts: '2026-08-08T12:01:00.000Z', + uptime_ms: 60000, + ms_since_last_operation: 30000, + }); + expect(result.success).toBe(false); + }); +}); + +// --------------------------------------------------------------------------- +// Union schema +// --------------------------------------------------------------------------- + +describe('PerfRecordSchema discriminated union', () => { + it('accepts an operation record', () => { + expect(PerfRecordSchema.safeParse(operationRecord()).success).toBe(true); + }); + + it('accepts a memory_sample record', () => { + expect(PerfRecordSchema.safeParse(memorySampleRecord()).success).toBe(true); + }); + + it('rejects an unknown record_type', () => { + expect( + PerfRecordSchema.safeParse({ + ...operationRecord(), + record_type: 'unknown_kind', + }).success, + ).toBe(false); + }); +}); + +// --------------------------------------------------------------------------- +// deriveOperationId / joinKeyFromPromptId (AC-3, settled §3/§9) +// --------------------------------------------------------------------------- + +describe('deriveOperationId (AC-3 — split rule)', () => { + it('leaves an initial prompt id byte-identical (no continuation marker)', () => { + const initial = 'sess-1#agentic-loop#f7e2-aaaa'; + expect(deriveOperationId(initial)).toBe(initial); + }); + + it('strips a single continuation marker to the prefix', () => { + expect( + deriveOperationId('sess-1#agentic-loop#f7e2-aaaa#continuation#1'), + ).toBe('sess-1#agentic-loop#f7e2-aaaa'); + }); + + it('strips continuation #2 to the same prefix as #1', () => { + expect( + deriveOperationId('sess-1#agentic-loop#f7e2-aaaa#continuation#2'), + ).toBe('sess-1#agentic-loop#f7e2-aaaa'); + }); + + it('strips a multi-digit continuation marker', () => { + expect( + deriveOperationId('sess-1#agentic-loop#f7e2-aaaa#continuation#12'), + ).toBe('sess-1#agentic-loop#f7e2-aaaa'); + }); + + it('strips a zero continuation (any occurrence begins the suffix)', () => { + expect( + deriveOperationId('sess-1#agentic-loop#f7e2-aaaa#continuation#0'), + ).toBe('sess-1#agentic-loop#f7e2-aaaa'); + }); + + it('strips a non-numeric continuation suffix (any occurrence begins the suffix)', () => { + expect( + deriveOperationId('sess-1#agentic-loop#f7e2-aaaa#continuation#abc'), + ).toBe('sess-1#agentic-loop#f7e2-aaaa'); + }); + + it('strips a negative continuation suffix (any occurrence begins the suffix)', () => { + expect( + deriveOperationId('sess-1#agentic-loop#f7e2-aaaa#continuation#-1'), + ).toBe('sess-1#agentic-loop#f7e2-aaaa'); + }); + + it('takes the first segment even when the marker is not terminal', () => { + expect(deriveOperationId('sess-1#continuation#1#more')).toBe('sess-1'); + }); + + it('takes the first segment for a trailing-empty marker', () => { + expect(deriveOperationId('prefix#continuation#')).toBe('prefix'); + }); + + it('preserves a CLI-fallback id (8-hash, no continuation marker)', () => { + const fallback = 'test-session########0'; + expect(deriveOperationId(fallback)).toBe(fallback); + }); +}); + +describe('joinKeyFromPromptId (AC-3 read-time join, D1)', () => { + it('derives the same key as deriveOperationId for an initial id', () => { + const initial = 'sess-1#agentic-loop#f7e2-aaaa'; + expect(joinKeyFromPromptId(initial)).toBe(initial); + expect(joinKeyFromPromptId(initial)).toBe(deriveOperationId(initial)); + }); + + it('derives the same key as deriveOperationId for continuations', () => { + const base = 'sess-1#agentic-loop#f7e2-aaaa'; + for (const n of [1, 2, 3, 10, 99]) { + expect(joinKeyFromPromptId(`${base}#continuation#${n}`)).toBe(base); + expect(joinKeyFromPromptId(`${base}#continuation#${n}`)).toBe( + deriveOperationId(`${base}#continuation#${n}`), + ); + } + }); +}); + +// --------------------------------------------------------------------------- +// parsePerfRecord + classifyPerfLine (AC-9 reader classification) +// --------------------------------------------------------------------------- + +describe('parsePerfRecord (AC-9 partial)', () => { + it('returns a parsed operation record for valid input', () => { + const parsed = parsePerfRecord(operationRecord()); + expect(parsed).not.toBeNull(); + if (parsed === null) throw new Error('expected a record'); + expect(parsed.record_type).toBe('operation'); + }); + + it('returns a parsed memory_sample record for valid input', () => { + const parsed = parsePerfRecord(memorySampleRecord()); + expect(parsed).not.toBeNull(); + if (parsed === null) throw new Error('expected a record'); + expect(parsed.record_type).toBe('memory_sample'); + }); + + it('returns null for a future schema version (skip+count, never coerce)', () => { + expect( + parsePerfRecord({ ...operationRecord(), schema_version: 99 }), + ).toBeNull(); + }); + + it('returns null for an unversioned/legacy record (no version, no type)', () => { + expect( + parsePerfRecord({ foo: 'bar', ts: '2026-01-01T00:00:00.000Z' }), + ).toBeNull(); + }); + + it('returns null for a record missing required fields', () => { + expect( + parsePerfRecord({ schema_version: 1, record_type: 'operation' }), + ).toBeNull(); + }); + + it('returns null for null/undefined/string/number input without throwing', () => { + expect(() => parsePerfRecord(null)).not.toThrow(); + expect(parsePerfRecord(null)).toBeNull(); + expect(parsePerfRecord(undefined)).toBeNull(); + expect(parsePerfRecord('not-an-object')).toBeNull(); + expect(parsePerfRecord(42)).toBeNull(); + }); + + it('does not throw on an array', () => { + expect(() => parsePerfRecord([1, 2, 3])).not.toThrow(); + expect(parsePerfRecord([1, 2, 3])).toBeNull(); + }); +}); + +describe('classifyPerfLine (AC-9 reader classification)', () => { + it('classifies a valid operation record as ok', () => { + expect(classifyPerfLine(operationRecord()).kind).toBe('ok'); + }); + + it('classifies a valid memory_sample record as ok', () => { + expect(classifyPerfLine(memorySampleRecord()).kind).toBe('ok'); + }); + + it('classifies a future-version record as future_version with the version', () => { + const c = classifyPerfLine({ ...operationRecord(), schema_version: 7 }); + expect(c).toMatchObject({ kind: 'future_version', schemaVersion: 7 }); + }); + + it('classifies an unversioned record (no version AND no type) as unversioned', () => { + expect( + classifyPerfLine({ some_legacy_field: 1, ts: '2026-01-01T00:00:00.000Z' }) + .kind, + ).toBe('unversioned'); + }); + + it('classifies a record missing required fields as malformed', () => { + expect( + classifyPerfLine({ schema_version: 1, record_type: 'operation' }).kind, + ).toBe('malformed'); + }); + + it('classifies non-object input (including arrays) as malformed', () => { + expect(classifyPerfLine('hello').kind).toBe('malformed'); + expect(classifyPerfLine(null).kind).toBe('malformed'); + expect(classifyPerfLine(42).kind).toBe('malformed'); + // Arrays are malformed, NOT unversioned/v0 — isStringRecord excludes them. + expect(classifyPerfLine([1, 2, 3]).kind).toBe('malformed'); + }); +}); diff --git a/packages/telemetry/src/perf/perfRecords.ts b/packages/telemetry/src/perf/perfRecords.ts new file mode 100644 index 0000000000..62c2abb638 --- /dev/null +++ b/packages/telemetry/src/perf/perfRecords.ts @@ -0,0 +1,415 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Versioned record schema for the client-side performance telemetry JSONL log + * (issue #3167). + * + * Single Zod declaration: the writer and the tolerant reader both derive their + * types from the schemas exported here (dev-docs/RULES.md mandates + * schema-first with Zod). + * + * Compatibility rules (spec §2): + * - Readers MUST ignore unknown fields (a field addition is NOT a version + * bump). Zod's default object mode strips unknown keys, so parsing succeeds. + * - A bump means a field changed meaning or was removed. A reader encountering + * a version above {@link PERF_SCHEMA_VERSION} MUST skip and count the record, + * never coerce it. + * + * Decision D1: the v1 record carries NO child `prompt_ids`/`turn_ids` arrays + * and NO true-count/cap fields. `operation_id` (derived from the top-level + * prompt-id prefix) is the sole join key; the report performs the exact join + * at read time. + */ + +import { z } from 'zod'; +import { createReadStream } from 'node:fs'; + +export const PERF_SCHEMA_VERSION = 1; + +export const PERF_RECORD_TYPE_OPERATION = 'operation'; +export const PERF_RECORD_TYPE_MEMORY_SAMPLE = 'memory_sample'; + +/** + * The seven terminal operation statuses (spec §1.3), including `superseded`, + * which is load-bearing because the ownership release in `useSubmitQuery` is + * guarded by `isCurrentTurn`, so a superseded operation never reaches it. + */ +export const PERF_TERMINAL_STATUSES = [ + 'completed', + 'error', + 'cancelled_before_send', + 'cancelled_during_api', + 'cancelled_during_tool', + 'cancelled_during_approval', + 'superseded', +] as const; + +export type PerfTerminalStatus = (typeof PERF_TERMINAL_STATUSES)[number]; + +// A non-empty string. Identity/build/comparison-dimension strings must not be +// empty — an empty identity string is rejected at the schema boundary. +const nonEmptyString = z.string().min(1); + +// ISO 8601 timestamp with timezone (Z or offset). Rejects bare local times. +const isoTimestamp = z.string().min(1).datetime({ offset: true }); + +// Finite, non-negative number. Used for durations, bytes, memory, uptime, +// and sample ages. +const finiteNonNeg = z.number().finite().nonnegative(); + +// Finite number that may be negative (the honest residual). +const finiteSigned = z.number().finite(); + +// Non-negative integer. Used for counts, tokens, and indices. +const nonNegInt = z.number().int().nonnegative(); + +// Positive integer (>= 1). Used for concurrent_instances. +const posInt = z.number().int().min(1); + +// --------------------------------------------------------------------------- +// Operation record (record_type: "operation") +// --------------------------------------------------------------------------- + +export const PerfOperationRecordSchema = z.object({ + // --- envelope --- + // schema_version accepts both the current version (1) and the normalized + // v0 form. A v0 record is an unversioned legacy object that fully matches + // the operation payload shape; classifyPerfLine normalizes it at read time + // (spec §2, following the tokenUsageRecords pattern). The writer always + // emits PERF_SCHEMA_VERSION (1); v0 is only reachable through normalization. + schema_version: z.union([z.literal(0), z.literal(PERF_SCHEMA_VERSION)]), + record_type: z.literal(PERF_RECORD_TYPE_OPERATION), + ts: isoTimestamp, + + // --- identity (reuses #3130's key names verbatim) --- + session_id: nonEmptyString, + operation_id: nonEmptyString, + runtime_id: nonEmptyString, + parent_runtime_id: nonEmptyString.nullable(), + subagent_name: nonEmptyString.nullable(), + project_hash: nonEmptyString, + + // --- build identity (the x-axis) --- + llxprt_version: nonEmptyString, + git_sha: nonEmptyString, + runtime: nonEmptyString, + platform: nonEmptyString, + + // --- comparison dimensions (compare like with like, never pooled) --- + provider: nonEmptyString, + model: nonEmptyString, + context_tokens: nonNegInt, + output_tokens: nonNegInt, + // Geometry: non-negative integers (unknown terminal geometry is zero — P12). + // concurrent_instances remains a positive integer (minimum 1 — D3). + terminal_cols: nonNegInt, + terminal_rows: nonNegInt, + render_mode: nonEmptyString, + concurrent_instances: posInt, + + // --- terminal status --- + status: z.enum(PERF_TERMINAL_STATUSES), + + // --- client work: directly measured, additive among themselves --- + client_prepare_ms: finiteNonNeg, + stream_handler_ms: finiteNonNeg, + ink_render_ms: finiteNonNeg, + ink_render_count: nonNegInt, + stdout_bytes: finiteNonNeg, + stdout_write_calls: nonNegInt, + stdout_write_sync_ms: finiteNonNeg, + client_finalize_ms: finiteNonNeg, + + // --- provider/tool work: overlapping, NOT additive with client phases --- + provider_attempts: nonNegInt, + provider_attempt_sum_ms: finiteNonNeg, + provider_union_ms: finiteNonNeg, + tool_calls: nonNegInt, + tool_call_sum_ms: finiteNonNeg, + tool_union_ms: finiteNonNeg, + agent_activity_union_ms: finiteNonNeg, + + // --- elapsed --- + operation_elapsed_ms: finiteNonNeg, + approval_wait_ms: finiteNonNeg, + unclassified_elapsed_ms: finiteSigned, + + // --- memory (OPTIONAL: present iff memory enabled; omitted, never zero) --- + rss_bytes: finiteNonNeg.optional(), + heap_used_bytes: finiteNonNeg.optional(), + external_bytes: finiteNonNeg.optional(), + array_buffers_bytes: finiteNonNeg.optional(), + + // --- always present --- + session_operation_index: nonNegInt, + uptime_ms: finiteNonNeg, +}); + +export type PerfOperationRecord = z.infer; + +// --------------------------------------------------------------------------- +// Memory sample record (record_type: "memory_sample") +// --------------------------------------------------------------------------- + +export const PerfMemorySampleRecordSchema = z.object({ + schema_version: z.literal(PERF_SCHEMA_VERSION), + record_type: z.literal(PERF_RECORD_TYPE_MEMORY_SAMPLE), + ts: isoTimestamp, + rss_bytes: finiteNonNeg, + heap_used_bytes: finiteNonNeg, + external_bytes: finiteNonNeg, + array_buffers_bytes: finiteNonNeg, + uptime_ms: finiteNonNeg, + ms_since_last_operation: finiteNonNeg, +}); + +export type PerfMemorySampleRecord = z.infer< + typeof PerfMemorySampleRecordSchema +>; + +// --------------------------------------------------------------------------- +// Full record union (discriminated on record_type) +// --------------------------------------------------------------------------- + +export const PerfRecordSchema = z.discriminatedUnion('record_type', [ + PerfOperationRecordSchema, + PerfMemorySampleRecordSchema, +]); + +export type PerfRecord = z.infer; + +// --------------------------------------------------------------------------- +// operation_id derivation (settled §3 / §9 — derived, NOT minted+propagated) +// --------------------------------------------------------------------------- + +/** + * Derives the operation join key from a prompt id by taking the first segment + * before any `#continuation#` marker. + * + * `AgenticLoop.generateContinuationPromptId()` returns + * `${initialPromptId}#continuation#${n}` for continuations, so taking the first + * `split('#continuation#')` segment recovers the initial id, which is the + * operation's sole join key. An initial id (no marker) is returned byte-identical. + * Any occurrence of the marker — terminal or not, numeric or not — begins the + * continuation suffix. A CLI-fallback id without the marker is preserved. + */ +export function deriveOperationId(promptId: string): string { + return promptId.split('#continuation#')[0]; +} + +/** + * Read-time join key derived from a token-usage / session-recording row's + * `prompt_id`. Same derivation as {@link deriveOperationId}: continuations + * collapse to their shared initial-prompt-id prefix so N continuation rows + * join to the SINGLE perf operation (D1) without any child id on the perf + * record. + */ +export function joinKeyFromPromptId(promptId: string): string { + return promptId.split('#continuation#')[0]; +} + +// --------------------------------------------------------------------------- +// Tolerant per-line classification (external JSONL input — defensive parsing) +// --------------------------------------------------------------------------- + +/** + * Per-line classification of a parsed JSON object from a perf JSONL file. + * + * The reader uses this richer result (rather than a bare `PerfRecord | null`) + * so self-health counters can distinguish future-version skips, unversioned + * legacy rows, and genuinely malformed records. + */ +export type PerfLineClassification = + | { readonly kind: 'ok'; readonly record: PerfRecord } + | { readonly kind: 'future_version'; readonly schemaVersion: number } + | { readonly kind: 'unversioned' } + | { readonly kind: 'malformed' }; + +function isStringRecord(value: unknown): value is Record { + if (typeof value !== 'object' || value === null) return false; + // Arrays are not string records — they must be classified as `malformed` + // (not `unversioned`), so `Array.isArray` is excluded here. + return !Array.isArray(value); +} + +/** + * Classifies a parsed JSON object (NOT a raw text line) from a perf JSONL file. + * + * - A record with NO `schema_version` and NO `record_type` that FULLY matches + * the operation payload shape is normalized to a v0 operation record + * (`{ schema_version: 0, record_type: 'operation' }`) — following the + * tokenUsageRecords pattern (spec §2). An incomplete or arbitrary + * unversioned object that does not validate against the operation schema is + * `unversioned` (counted, never fake-normalized). + * - A record whose numeric `schema_version` exceeds {@link PERF_SCHEMA_VERSION} + * is `future_version` (skip+count, never coerce). + * - Otherwise it is validated against {@link PerfRecordSchema}; unknown fields + * are ignored (stripped). Success → `ok`; failure → `malformed`. + * - Never throws. + */ +export function classifyPerfLine(line: unknown): PerfLineClassification { + if (!isStringRecord(line)) { + return { kind: 'malformed' }; + } + + const hasSchemaVersion = 'schema_version' in line; + const hasRecordType = 'record_type' in line; + + if (!hasSchemaVersion && !hasRecordType) { + // v0 normalization: an unversioned object that fully matches the + // operation payload shape normalizes to a v0 operation record. An + // incomplete/arbitrary object that fails validation remains unversioned. + const normalized = { + ...line, + schema_version: 0, + record_type: PERF_RECORD_TYPE_OPERATION, + }; + const v0Result = PerfRecordSchema.safeParse(normalized); + if (v0Result.success) { + return { kind: 'ok', record: v0Result.data }; + } + return { kind: 'unversioned' }; + } + + const version = line.schema_version; + if (typeof version === 'number' && version > PERF_SCHEMA_VERSION) { + return { kind: 'future_version', schemaVersion: version }; + } + + const result = PerfRecordSchema.safeParse(line); + if (result.success) { + return { kind: 'ok', record: result.data }; + } + return { kind: 'malformed' }; +} + +/** + * Tolerant per-line reader. Returns the parsed record or `null` (never throws). + * Equivalent to {@link classifyPerfLine} for callers that only need the record. + */ +export function parsePerfRecord(line: unknown): PerfRecord | null { + const classification = classifyPerfLine(line); + return classification.kind === 'ok' ? classification.record : null; +} + +// --------------------------------------------------------------------------- +// Streaming JSONL reader (external files — never reads the whole file) +// --------------------------------------------------------------------------- + +/** + * Incremental classification of a single line yielded by the streaming reader. + * + * Superset of {@link PerfLineClassification}: adds `blank` (whitespace-only + * line) and `truncated` (final unterminated line that did not parse, e.g. + * SIGKILL mid-append). + */ +export type PerfStreamEntry = + | { readonly kind: 'ok'; readonly record: PerfRecord } + | { readonly kind: 'future_version'; readonly schemaVersion: number } + | { readonly kind: 'unversioned' } + | { readonly kind: 'malformed' } + | { readonly kind: 'blank' } + | { readonly kind: 'truncated' }; + +export interface PerfReaderCounts { + /** Lines that parsed to a valid current-version perf record. */ + readonly parsed: number; + /** Complete (newline-terminated) lines that failed to parse. */ + readonly malformed: number; + /** Records whose schema_version is above the known version (skipped). */ + readonly futureVersion: number; + /** Records with no schema_version and no record_type (legacy/unknown). */ + readonly unversioned: number; + /** The final unterminated line that did not parse (SIGKILL mid-append). */ + readonly truncated: number; + /** Blank / whitespace-only lines. */ + readonly blank: number; +} + +export interface PerfReaderResult { + readonly records: readonly PerfRecord[]; + readonly counts: PerfReaderCounts; +} + +/** + * Streams perf JSONL entries from a file path, yielding classification outcomes + * incrementally WITHOUT reading the whole file into memory. + * + * P11 uses this to process a 24/7 file that never closes. + * + * The line-splitting + classification engine lives in the package-private + * module `./perfRecordsStream.js` (not exported from package.json). It is + * loaded with a dynamic import so the static dependency graph stays + * one-directional — `perfRecordsStream` imports `classifyPerfLine` from this + * module, and this module never statically imports it back, avoiding an import + * cycle. + * + * Genuine I/O failures (missing file, permission denied) propagate as a + * rejection; those are not line-content problems. + */ +export async function* streamPerfRecords( + filePath: string, +): AsyncGenerator { + const { streamPerfFromReadable } = await import('./perfRecordsStream.js'); + yield* streamPerfFromReadable(createReadStream(filePath)); +} + +/** + * Bounded convenience collector: streams the file with + * {@link streamPerfRecords} and accumulates the results into records + counts. + * P11 should prefer the streaming API for 24/7 files; this collector is kept + * for tests and bounded batch reads. + * + * Genuine I/O failures (missing file, permission denied) propagate as a + * rejection; those are not line-content problems. + */ +export async function readPerfRecords( + filePath: string, +): Promise { + const records: PerfRecord[] = []; + let parsed = 0; + let malformed = 0; + let futureVersion = 0; + let unversioned = 0; + let truncated = 0; + let blank = 0; + + for await (const entry of streamPerfRecords(filePath)) { + switch (entry.kind) { + case 'ok': + records.push(entry.record); + parsed++; + break; + case 'malformed': + malformed++; + break; + case 'future_version': + futureVersion++; + break; + case 'unversioned': + unversioned++; + break; + case 'truncated': + truncated++; + break; + case 'blank': + blank++; + break; + default: { + const _exhaustive: never = entry; + throw new Error( + `Internal invariant violation: unhandled PerfStreamEntry kind: ${JSON.stringify(_exhaustive)}`, + ); + } + } + } + + return { + records, + counts: { parsed, malformed, futureVersion, unversioned, truncated, blank }, + }; +} diff --git a/packages/telemetry/src/perf/perfRecords.v0.behavior.test.ts b/packages/telemetry/src/perf/perfRecords.v0.behavior.test.ts new file mode 100644 index 0000000000..40906c5171 --- /dev/null +++ b/packages/telemetry/src/perf/perfRecords.v0.behavior.test.ts @@ -0,0 +1,369 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Behavioral tests proving tolerant v0 / unversioned operation normalization + * (issue #3167 review finding D, spec §2). + * + * Grounded in the existing tokenUsageRecords normalization pattern (without + * importing packages/agents): a missing schema_version + record_type object + * that FULLY matches the operation payload shape normalizes to internal + * operation form (schema_version 0). Arbitrary or incomplete unversioned + * objects remain counted as unversioned. + * + * Preserves: valid current v1, unknown-field tolerance, future-version + * skip/count, and final-line semantics (only unterminated JSON parse failure + * is truncated). + */ + +import { describe, it, expect } from 'bun:test'; +import { promises as fs } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { + classifyPerfLine, + parsePerfRecord, + readPerfRecords, + streamPerfRecords, +} from './perfRecords.js'; + +// --------------------------------------------------------------------------- +// Record builders +// --------------------------------------------------------------------------- + +function operationFields( + overrides: Record = {}, +): Record { + return { + ts: '2026-08-08T12:00:00.000Z', + session_id: 'sess-abc', + operation_id: 'sess-abc#agentic-loop#f7e2', + runtime_id: 'rt-main', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'sha256:project-hash', + llxprt_version: '0.11.0', + git_sha: 'abc1234', + runtime: 'bun-1.3.14', + platform: 'darwin-arm64', + provider: 'openai', + model: 'gpt-4o', + context_tokens: 1000, + output_tokens: 500, + terminal_cols: 120, + terminal_rows: 40, + render_mode: 'incremental', + concurrent_instances: 1, + status: 'completed', + client_prepare_ms: 5, + stream_handler_ms: 10, + ink_render_ms: 20, + ink_render_count: 3, + stdout_bytes: 4096, + stdout_write_calls: 3, + stdout_write_sync_ms: 2, + client_finalize_ms: 1, + provider_attempts: 1, + provider_attempt_sum_ms: 800, + provider_union_ms: 800, + tool_calls: 2, + tool_call_sum_ms: 300, + tool_union_ms: 280, + agent_activity_union_ms: 1000, + operation_elapsed_ms: 1200, + approval_wait_ms: 0, + unclassified_elapsed_ms: 100, + session_operation_index: 1, + uptime_ms: 50000, + ...overrides, + }; +} + +/** A valid v1 operation record (has schema_version + record_type). */ +function v1Operation( + overrides: Record = {}, +): Record { + return { + schema_version: 1, + record_type: 'operation', + ...operationFields(overrides), + }; +} + +/** An unversioned operation payload: all operation fields, NO version/type. */ +function unversionedOperation( + overrides: Record = {}, +): Record { + return operationFields(overrides); +} + +// --------------------------------------------------------------------------- +// classifyPerfLine: v0 normalization +// --------------------------------------------------------------------------- + +describe('classifyPerfLine v0 normalization (finding D)', () => { + it('normalizes an unversioned object that fully matches the operation shape to ok', () => { + const c = classifyPerfLine(unversionedOperation()); + expect(c.kind).toBe('ok'); + }); + + it('normalized v0 record carries schema_version 0 and record_type operation', () => { + const c = classifyPerfLine(unversionedOperation()); + expect(c.kind).toBe('ok'); + if (c.kind !== 'ok') throw new Error('expected ok'); + expect(c.record.schema_version).toBe(0); + expect(c.record.record_type).toBe('operation'); + }); + + it('normalized v0 record preserves all operation fields', () => { + const c = classifyPerfLine( + unversionedOperation({ provider: 'anthropic', status: 'error' }), + ); + expect(c.kind).toBe('ok'); + if (c.kind !== 'ok') throw new Error('expected ok'); + expect(c.record.record_type).toBe('operation'); + if (c.record.record_type !== 'operation') throw new Error('op'); + expect(c.record.provider).toBe('anthropic'); + expect(c.record.status).toBe('error'); + }); + + it('strips unknown fields from a normalized v0 record (§2 tolerance)', () => { + const c = classifyPerfLine( + unversionedOperation({ future_field: 42, extra: 'x' }), + ); + expect(c.kind).toBe('ok'); + if (c.kind !== 'ok') throw new Error('expected ok'); + expect('future_field' in c.record).toBe(false); + expect('extra' in c.record).toBe(false); + }); + + it('classifies an incomplete unversioned object (missing fields) as unversioned', () => { + // Missing required operation fields → does not fully match → unversioned. + expect(classifyPerfLine({ foo: 'bar' }).kind).toBe('unversioned'); + expect(classifyPerfLine({ session_id: 'x', provider: 'y' }).kind).toBe( + 'unversioned', + ); + }); + + it('classifies an arbitrary unversioned object as unversioned', () => { + expect( + classifyPerfLine({ random: 'data', ts: '2026-01-01T00:00:00.000Z' }).kind, + ).toBe('unversioned'); + }); + + it('classifies non-object input as malformed (not unversioned)', () => { + expect(classifyPerfLine('hello').kind).toBe('malformed'); + expect(classifyPerfLine(null).kind).toBe('malformed'); + expect(classifyPerfLine(42).kind).toBe('malformed'); + }); +}); + +// --------------------------------------------------------------------------- +// parsePerfRecord: v0 normalization +// --------------------------------------------------------------------------- + +describe('parsePerfRecord v0 normalization (finding D)', () => { + it('returns a parsed record for a normalized v0 operation', () => { + const parsed = parsePerfRecord(unversionedOperation()); + expect(parsed).not.toBeNull(); + if (parsed === null) throw new Error('expected a record'); + expect(parsed.record_type).toBe('operation'); + expect(parsed.schema_version).toBe(0); + }); + + it('returns null for an incomplete unversioned object', () => { + expect(parsePerfRecord({ foo: 'bar' })).toBeNull(); + }); +}); + +// --------------------------------------------------------------------------- +// Preservation: v1, future-version, tolerance +// --------------------------------------------------------------------------- + +describe('v0 normalization preserves existing behavior (finding D)', () => { + it('valid current v1 operation classifies as ok with schema_version 1', () => { + const c = classifyPerfLine(v1Operation()); + expect(c.kind).toBe('ok'); + if (c.kind !== 'ok') throw new Error('expected ok'); + expect(c.record.schema_version).toBe(1); + }); + + it('future-version record is still future_version (skip+count)', () => { + const c = classifyPerfLine({ ...v1Operation(), schema_version: 99 }); + expect(c).toMatchObject({ kind: 'future_version', schemaVersion: 99 }); + }); + + it('v1 record with unknown fields still classifies as ok (stripped)', () => { + const c = classifyPerfLine(v1Operation({ new_metric: 7 })); + expect(c.kind).toBe('ok'); + if (c.kind !== 'ok') throw new Error('expected ok'); + expect('new_metric' in c.record).toBe(false); + }); + + it('a record with record_type but missing schema_version is malformed', () => { + // record_type present, schema_version absent → NOT the v0 normalization + // path (which requires BOTH missing). Falls through to normal validation + // which requires schema_version → malformed. + expect( + classifyPerfLine({ record_type: 'operation', ...operationFields() }).kind, + ).toBe('malformed'); + }); + + it('a record with schema_version 0 but no record_type is malformed', () => { + // schema_version present, record_type absent → NOT the v0 normalization + // path. Falls through to normal validation which requires record_type → + // malformed. + expect( + classifyPerfLine({ schema_version: 0, ...operationFields() }).kind, + ).toBe('malformed'); + }); +}); + +// --------------------------------------------------------------------------- +// Real-file + streaming tests +// --------------------------------------------------------------------------- + +async function makeTempDir(): Promise { + const dir = join( + tmpdir(), + `perf-v0-${Date.now()}-${Math.random().toString(36).slice(2)}`, + ); + await fs.mkdir(dir, { recursive: true }); + return dir; +} + +describe('v0 normalization real-file + streaming (finding D)', () => { + let dir: string; + + async function setup(): Promise { + dir = await makeTempDir(); + return dir; + } + + async function cleanup(): Promise { + if (dir) { + await fs.rm(dir, { recursive: true, force: true }); + } + } + + it('readPerfRecords parses a file with v0 + v1 + unversioned records', async () => { + const d = await setup(); + try { + const filePath = join(d, 'perf-20260808-uuid.jsonl'); + const lines = [ + JSON.stringify(v1Operation({ operation_id: 'op-1' })), + JSON.stringify(unversionedOperation({ operation_id: 'op-2' })), + JSON.stringify({ random: 'unversioned-incomplete' }), + ]; + await fs.writeFile(filePath, lines.join('\n') + '\n', 'utf8'); + + const { records, counts } = await readPerfRecords(filePath); + + expect(records).toHaveLength(2); + expect(records[0].schema_version).toBe(1); + expect(records[1].schema_version).toBe(0); + expect(counts.parsed).toBe(2); + expect(counts.unversioned).toBe(1); + } finally { + await cleanup(); + } + }); + + it('streamPerfRecords yields ok for a normalized v0 line', async () => { + const d = await setup(); + try { + const filePath = join(d, 'perf-20260808-uuid.jsonl'); + await fs.writeFile( + filePath, + JSON.stringify(unversionedOperation()) + '\n', + 'utf8', + ); + + const entries = []; + for await (const entry of streamPerfRecords(filePath)) { + entries.push(entry); + } + + expect(entries).toHaveLength(1); + expect(entries[0].kind).toBe('ok'); + if (entries[0].kind !== 'ok') throw new Error('expected ok'); + expect(entries[0].record.schema_version).toBe(0); + } finally { + await cleanup(); + } + }); + + it('final-line semantics: an unterminated v0 JSON line is still ok (content retained)', async () => { + const d = await setup(); + try { + const filePath = join(d, 'perf-20260808-uuid.jsonl'); + // A valid v0 JSON object WITHOUT a trailing newline — this is a + // complete JSON value, just unterminated. It must parse to ok, NOT + // truncated (only JSON.parse failure on the final line is truncated). + await fs.writeFile( + filePath, + JSON.stringify(unversionedOperation()), + 'utf8', + ); + + const entries = []; + for await (const entry of streamPerfRecords(filePath)) { + entries.push(entry); + } + + expect(entries).toHaveLength(1); + expect(entries[0].kind).toBe('ok'); + } finally { + await cleanup(); + } + }); + + it('final-line semantics: an unterminated future-version line retains future_version classification', async () => { + const d = await setup(); + try { + const filePath = join(d, 'perf-20260808-uuid.jsonl'); + const futureRec = JSON.stringify({ + ...v1Operation(), + schema_version: 5, + }); + await fs.writeFile(filePath, futureRec, 'utf8'); // no trailing newline + + const entries = []; + for await (const entry of streamPerfRecords(filePath)) { + entries.push(entry); + } + + expect(entries).toHaveLength(1); + expect(entries[0].kind).toBe('future_version'); + } finally { + await cleanup(); + } + }); + + it('final-line semantics: only a genuinely broken final line is truncated', async () => { + const d = await setup(); + try { + const filePath = join(d, 'perf-20260808-uuid.jsonl'); + // A complete valid line + an unterminated broken JSON fragment. + await fs.writeFile( + filePath, + JSON.stringify(v1Operation()) + '\n{"schema_version":1,"record_t', + 'utf8', + ); + + const entries = []; + for await (const entry of streamPerfRecords(filePath)) { + entries.push(entry); + } + + expect(entries).toHaveLength(2); + expect(entries[0].kind).toBe('ok'); + expect(entries[1].kind).toBe('truncated'); + } finally { + await cleanup(); + } + }); +}); diff --git a/packages/telemetry/src/perf/perfRecordsStream.ts b/packages/telemetry/src/perf/perfRecordsStream.ts new file mode 100644 index 0000000000..e311305a07 --- /dev/null +++ b/packages/telemetry/src/perf/perfRecordsStream.ts @@ -0,0 +1,109 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Package-private streaming JSONL reader internals (issue #3167, P04). + * + * This module is NOT listed in {@link ../../../../package.json} (only + * `./perf/perfRecords.js` is exported for the perf reader), so + * {@link streamPerfFromReadable} is unreachable by package consumers. The + * public {@link ./perfRecords.js} reaches it to implement `streamPerfRecords`, + * and same-package behavior tests import it directly to prove incremental + * yield against a controlled readable. + * + * It depends on the public classifier `classifyPerfLine` from + * {@link ./perfRecords.js}. To keep the static dependency graph + * one-directional (perfRecordsStream -> perfRecords) and avoid an import + * cycle, `perfRecords.ts` loads this module with a dynamic import rather than a + * static one. + */ + +import { StringDecoder } from 'node:string_decoder'; + +import { classifyPerfLine, type PerfStreamEntry } from './perfRecords.js'; + +function toBuffer(chunk: Buffer | string): Buffer { + return Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk); +} + +/** + * Classifies a complete or final text line into a {@link PerfStreamEntry}. + * + * Contract (issue #3167): `truncated` is reserved for a final nonblank line + * that is NOT valid JSON (realistic cause: a SIGKILL mid-append). A final line + * that parses as JSON keeps its content classification — a syntactically valid + * future-version or unversioned value without a trailing newline is still + * `future_version` or `unversioned`, and a current-version record that parses + * but misses required fields is still `malformed`. Only a JSON.parse failure on + * the final line is `truncated`. A complete (newline-terminated) non-JSON line + * is `malformed`. Blank/whitespace-only lines are `blank`. Never throws. + */ +function classifyTextLineToStreamEntry( + text: string, + isFinal: boolean, +): PerfStreamEntry { + if (text.trim() === '') { + return { kind: 'blank' }; + } + let value: unknown; + try { + value = JSON.parse(text); + } catch { + // Only a JSON-parse failure on the final unterminated line is truncated. + return isFinal ? { kind: 'truncated' } : { kind: 'malformed' }; + } + const classification = classifyPerfLine(value); + switch (classification.kind) { + case 'ok': + return { kind: 'ok', record: classification.record }; + case 'future_version': + return { + kind: 'future_version', + schemaVersion: classification.schemaVersion, + }; + case 'unversioned': + return { kind: 'unversioned' }; + case 'malformed': + return { kind: 'malformed' }; + default: { + const _exhaustive: never = classification; + return _exhaustive; + } + } +} + +/** + * Streams perf JSONL entries from any readable stream, yielding classification + * outcomes incrementally WITHOUT accumulating the entire stream. + * + * This is the package-private seam: tests inject a controlled readable to + * prove the iterator yields before the entire stream is consumed. The public + * `streamPerfRecords` (in {@link ./perfRecords.js}) wraps this to process a + * 24/7 file that never closes. + * + * Genuine I/O failures propagate as a rejection; those are not line-content + * problems. + */ +export async function* streamPerfFromReadable( + readable: NodeJS.ReadableStream, +): AsyncGenerator { + const decoder = new StringDecoder('utf8'); + let leftover = ''; + + for await (const chunk of readable) { + const data = leftover + decoder.write(toBuffer(chunk)); + const parts = data.split('\n'); + leftover = parts.pop() ?? ''; + for (const line of parts) { + yield classifyTextLineToStreamEntry(line, false); + } + } + leftover += decoder.end(); + + if (leftover !== '') { + yield classifyTextLineToStreamEntry(leftover, true); + } +} diff --git a/packages/telemetry/src/perf/perfReport.behavior.test.ts b/packages/telemetry/src/perf/perfReport.behavior.test.ts new file mode 100644 index 0000000000..81790fb476 --- /dev/null +++ b/packages/telemetry/src/perf/perfReport.behavior.test.ts @@ -0,0 +1,965 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import { promises as fs } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { + buildReport, + formatReport, + joinTokenRowsByOperation, +} from './perfReport.js'; +import type { PerfOperationRecord } from './perfRecords.js'; + +// --------------------------------------------------------------------------- +// Fixtures +// --------------------------------------------------------------------------- + +function makeOperation( + overrides: Partial = {}, +): PerfOperationRecord { + return { + schema_version: 1, + record_type: 'operation', + ts: '2026-01-01T00:00:00.000Z', + session_id: 'sess-1', + operation_id: 'op-1', + runtime_id: 'rt-1', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'hash-1', + llxprt_version: '0.10.0', + git_sha: 'abc1234', + runtime: 'cli', + platform: 'darwin', + provider: 'test-provider', + model: 'test-model', + context_tokens: 1000, + output_tokens: 500, + terminal_cols: 80, + terminal_rows: 24, + render_mode: 'ink', + concurrent_instances: 1, + status: 'completed', + client_prepare_ms: 10, + stream_handler_ms: 100, + ink_render_ms: 5, + ink_render_count: 3, + stdout_bytes: 1024, + stdout_write_calls: 5, + stdout_write_sync_ms: 2, + client_finalize_ms: 8, + provider_attempts: 1, + provider_attempt_sum_ms: 200, + provider_union_ms: 200, + tool_calls: 2, + tool_call_sum_ms: 50, + tool_union_ms: 50, + agent_activity_union_ms: 250, + operation_elapsed_ms: 1000, + approval_wait_ms: 0, + unclassified_elapsed_ms: 0, + session_operation_index: 0, + uptime_ms: 5000, + ...overrides, + }; +} + +async function makeTempDir(): Promise { + const dir = join( + tmpdir(), + `perf-report-${Date.now()}-${Math.random().toString(36).slice(2)}`, + ); + await fs.mkdir(dir, { recursive: true }); + return dir; +} + +async function writeJsonl( + dir: string, + name: string, + lines: string[], +): Promise { + const content = lines.join('\n'); + await fs.writeFile( + join(dir, name), + content + (content.endsWith('\n') ? '' : '\n'), + 'utf8', + ); +} + +// --------------------------------------------------------------------------- +// Tests +// --------------------------------------------------------------------------- + +describe('PerfReport (P11, AC-9, D7)', () => { + let dir: string; + + beforeEach(async () => { + dir = await makeTempDir(); + }); + + afterEach(async () => { + await fs.rm(dir, { recursive: true, force: true }); + }); + + // --- Grouping and dimensions --- + + it('groups by build identity within exact dimensions', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.10.0', + git_sha: 'aaa', + provider: 'p1', + model: 'm1', + render_mode: 'ink', + terminal_cols: 80, + terminal_rows: 24, + operation_elapsed_ms: 1000, + }), + ), + JSON.stringify( + makeOperation({ + llxprt_version: '0.10.0', + git_sha: 'aaa', + provider: 'p1', + model: 'm1', + render_mode: 'ink', + terminal_cols: 80, + terminal_rows: 24, + operation_elapsed_ms: 2000, + }), + ), + JSON.stringify( + makeOperation({ + llxprt_version: '0.11.0', + git_sha: 'bbb', + provider: 'p1', + model: 'm1', + render_mode: 'ink', + terminal_cols: 80, + terminal_rows: 24, + operation_elapsed_ms: 3000, + }), + ), + ]); + + const report = await buildReport(dir); + + expect(report.groups).toHaveLength(2); + // Same dimensions, different builds → 2 groups + const g1 = report.groups.find((g) => g.build.git_sha === 'aaa'); + const g2 = report.groups.find((g) => g.build.git_sha === 'bbb'); + expect(g1).toBeDefined(); + expect(g2).toBeDefined(); + expect(g1!.sampleCount).toBe(2); + expect(g2!.sampleCount).toBe(1); + }); + + it('never pools groups with different dimensions', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify( + makeOperation({ + provider: 'p1', + model: 'm1', + render_mode: 'ink', + terminal_cols: 80, + terminal_rows: 24, + }), + ), + JSON.stringify( + makeOperation({ + provider: 'p2', // different provider + model: 'm1', + render_mode: 'ink', + terminal_cols: 80, + terminal_rows: 24, + }), + ), + JSON.stringify( + makeOperation({ + provider: 'p1', + model: 'm2', // different model + render_mode: 'ink', + terminal_cols: 80, + terminal_rows: 24, + }), + ), + ]); + + const report = await buildReport(dir); + + expect(report.groups).toHaveLength(3); + }); + + // --- p50 --- + + it('computes p50 (median) for recorded metrics', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify( + makeOperation({ + operation_elapsed_ms: 100, + session_operation_index: 0, + }), + ), + JSON.stringify( + makeOperation({ + operation_elapsed_ms: 200, + session_operation_index: 1, + }), + ), + JSON.stringify( + makeOperation({ + operation_elapsed_ms: 300, + session_operation_index: 2, + }), + ), + ]); + + const report = await buildReport(dir); + const group = report.groups[0]; + + // p50 of [100, 200, 300] = 200 (odd count → middle) + expect(group.p50.operation_elapsed_ms).toBe(200); + expect(group.p50.client_prepare_ms).toBe(10); + }); + + // --- Contamination --- + + it('counts contaminated samples (concurrent_instances >= 2, NOT contended)', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify( + makeOperation({ concurrent_instances: 1, operation_id: 'op-1' }), + ), + JSON.stringify( + makeOperation({ concurrent_instances: 2, operation_id: 'op-2' }), + ), + JSON.stringify( + makeOperation({ concurrent_instances: 3, operation_id: 'op-3' }), + ), + ]); + + const report = await buildReport(dir); + const group = report.groups[0]; + + expect(group.sampleCount).toBe(3); + expect(group.contaminatedSampleCount).toBe(2); + }); + + // --- Terminal status counts --- + + it('counts terminal statuses', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify( + makeOperation({ status: 'completed', operation_id: 'op-1' }), + ), + JSON.stringify( + makeOperation({ status: 'completed', operation_id: 'op-2' }), + ), + JSON.stringify(makeOperation({ status: 'error', operation_id: 'op-3' })), + JSON.stringify( + makeOperation({ + status: 'cancelled_during_tool', + operation_id: 'op-4', + }), + ), + ]); + + const report = await buildReport(dir); + const group = report.groups[0]; + + expect(group.terminalStatusCounts.completed).toBe(2); + expect(group.terminalStatusCounts.error).toBe(1); + expect(group.terminalStatusCounts.cancelled_during_tool).toBe(1); + expect(group.terminalStatusCounts.cancelled_before_send).toBe(0); + }); + + // --- No baseline → no delta --- + + it('without baseline: no delta properties or labels', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation({ operation_elapsed_ms: 1000 })), + ]); + + const report = await buildReport(dir); + + expect(report.baseline).toBeNull(); + for (const group of report.groups) { + expect(group.baselineComparison).toBeUndefined(); + expect(group.isBaseline).toBe(false); + } + + // Formatter: no delta text + const text = formatReport(report); + expect(text).not.toContain('delta'); + expect(text).not.toContain('Baseline:'); + }); + + // --- Baseline by version --- + + it('with baseline by exact version: matched-dimension deltas', async () => { + await writeJsonl(dir, 'perf-20260101-base.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.10.0', + git_sha: 'aaa', + operation_elapsed_ms: 1000, + }), + ), + ]); + await writeJsonl(dir, 'perf-20260102-new.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.11.0', + git_sha: 'bbb', + operation_elapsed_ms: 1500, + }), + ), + ]); + + const report = await buildReport(dir, '0.10.0'); + + expect(report.baseline).not.toBeNull(); + expect(report.baseline!.found).toBe(true); + expect(report.baseline!.value).toBe('0.10.0'); + + const newGroup = report.groups.find( + (g) => g.build.llxprt_version === '0.11.0', + ); + expect(newGroup).toBeDefined(); + expect(newGroup!.isBaseline).toBe(false); + expect(newGroup!.baselineComparison).toBeDefined(); + expect(newGroup!.baselineComparison!.matched).toBe(true); + // Delta: 1500 - 1000 = +500 (50%) + const delta = newGroup!.baselineComparison!.deltas!['operation_elapsed_ms']; + expect(delta.absolute).toBe(500); + expect(delta.percent).toBe(50); + }); + + // --- Baseline by sha --- + + it('with baseline by exact sha: matched-dimension deltas', async () => { + await writeJsonl(dir, 'perf-20260101-base.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.10.0', + git_sha: 'aaa111', + operation_elapsed_ms: 800, + }), + ), + ]); + await writeJsonl(dir, 'perf-20260102-new.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.11.0', + git_sha: 'bbb222', + operation_elapsed_ms: 400, + }), + ), + ]); + + const report = await buildReport(dir, 'aaa111'); + + expect(report.baseline!.found).toBe(true); + const newGroup = report.groups.find((g) => g.build.git_sha === 'bbb222'); + expect(newGroup!.baselineComparison!.matched).toBe(true); + // Delta: 400 - 800 = -400 (-50%) + const delta = newGroup!.baselineComparison!.deltas!['operation_elapsed_ms']; + expect(delta.absolute).toBe(-400); + expect(delta.percent).toBe(-50); + }); + + // --- Unmatched groups (different dimensions) --- + + it('unmatched baseline groups (different dimensions) are explicitly unmatched', async () => { + await writeJsonl(dir, 'perf-20260101-base.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.10.0', + git_sha: 'aaa', + provider: 'p1', + model: 'm1', + operation_elapsed_ms: 1000, + }), + ), + ]); + await writeJsonl(dir, 'perf-20260102-new.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.11.0', + git_sha: 'bbb', + provider: 'p2', // different dimensions → unmatched + model: 'm1', + operation_elapsed_ms: 1500, + }), + ), + ]); + + const report = await buildReport(dir, '0.10.0'); + + const newGroup = report.groups.find( + (g) => g.build.llxprt_version === '0.11.0', + ); + expect(newGroup!.baselineComparison).toBeDefined(); + expect(newGroup!.baselineComparison!.matched).toBe(false); + expect(newGroup!.baselineComparison!.deltas).toBeUndefined(); + }); + + // --- Selector matches no records (baseline not found) --- + + it('baseline not found is represented explicitly', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.10.0', + git_sha: 'aaa', + }), + ), + ]); + + const report = await buildReport(dir, '9.9.9'); + + expect(report.baseline).not.toBeNull(); + expect(report.baseline!.found).toBe(false); + // All groups are non-baseline, none matched + for (const group of report.groups) { + expect(group.isBaseline).toBe(false); + expect(group.baselineComparison).toBeDefined(); + expect(group.baselineComparison!.matched).toBe(false); + } + }); + + // --- No operation records → explicit representation --- + + it('empty directory produces zero groups and counts', async () => { + const report = await buildReport(dir); + + expect(report.groups).toHaveLength(0); + expect(report.counts.parsed).toBe(0); + expect(report.counts.files).toBe(0); + + const text = formatReport(report); + expect(text).toContain('No operation records found.'); + }); + + // --- Percent delta avoids division by zero --- + + it('percent delta is null when baseline p50 is zero', async () => { + await writeJsonl(dir, 'perf-20260101-base.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.10.0', + git_sha: 'aaa', + tool_calls: 0, + }), + ), + ]); + await writeJsonl(dir, 'perf-20260102-new.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.11.0', + git_sha: 'bbb', + tool_calls: 5, + }), + ), + ]); + + const report = await buildReport(dir, '0.10.0'); + const newGroup = report.groups.find((g) => g.build.git_sha === 'bbb'); + const delta = newGroup!.baselineComparison!.deltas!['tool_calls']; + + expect(delta.absolute).toBe(5); + expect(delta.percent).toBeNull(); // base was 0 → avoid div by zero + }); + + // --- Per-file memory slopes --- + + it('computes per-file memory slopes from P10 functions', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify( + makeOperation({ + operation_id: 'op-1', + session_operation_index: 0, + rss_bytes: 50_000_000, + heap_used_bytes: 20_000_000, + external_bytes: 5_000_000, + array_buffers_bytes: 1_000_000, + }), + ), + JSON.stringify( + makeOperation({ + operation_id: 'op-2', + session_operation_index: 1, + rss_bytes: 60_000_000, + heap_used_bytes: 25_000_000, + external_bytes: 5_000_000, + array_buffers_bytes: 1_000_000, + }), + ), + ]); + + const report = await buildReport(dir); + const group = report.groups[0]; + + expect(group.memorySlopes).toHaveLength(1); + const slopes = group.memorySlopes[0]; + expect(slopes.sourceFile).toBe('perf-20260101-run1.jsonl'); + expect(slopes.runUuid).toBe('run1'); + // Slope should be positive (growing memory) + expect(slopes.perOperation.rss_bytes_per_operation).not.toBeNull(); + expect(slopes.perOperation.rss_bytes_per_operation! > 0).toBe(true); + }); + + // --- Formatter stability --- + + it('formatter produces stable deterministic output', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation({ operation_elapsed_ms: 1000 })), + ]); + + const report1 = await buildReport(dir); + const report2 = await buildReport(dir); + + expect(formatReport(report1)).toBe(formatReport(report2)); + }); + + it('formatter includes counts and self-health', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + 'bad json line', + ]); + + const report = await buildReport(dir); + const text = formatReport(report); + + expect(text).toContain('Files scanned:'); + expect(text).toContain('Records:'); + expect(text).toContain('Self-health:'); + expect(text).toContain('skipped:'); + expect(text).toContain('malformed=1'); + }); + + // --- D1 read-time join --- + + it('D1: joinTokenRowsByOperation collapses continuation rows to one operation', () => { + const rows = [ + { promptId: 'abc-123', actualPromptTokens: 1000 }, + { promptId: 'abc-123#continuation#1', actualPromptTokens: 2000 }, + { promptId: 'abc-123#continuation#2', actualPromptTokens: 3000 }, + { promptId: 'def-456', actualPromptTokens: 5000 }, + ]; + + const joined = joinTokenRowsByOperation(rows); + + expect(joined.size).toBe(2); + expect(joined.get('abc-123')).toHaveLength(3); + expect(joined.get('def-456')).toHaveLength(1); + }); + + it('D1: initial prompt id (no continuation marker) is returned unchanged', () => { + const rows = [{ promptId: 'simple-id', actualPromptTokens: 100 }]; + const joined = joinTokenRowsByOperation(rows); + expect(joined.get('simple-id')).toHaveLength(1); + }); + + // --- Mixed version fileset --- + + it('handles mixed multi-version fileset with correct counts', async () => { + await writeJsonl(dir, 'perf-20260101-mixed.jsonl', [ + JSON.stringify(makeOperation({ operation_id: 'valid-1' })), + JSON.stringify({ + ...makeOperation(), + schema_version: 999, + }), + JSON.stringify({ random: 'unversioned' }), + 'totally broken json', + ]); + // Truncated final line + await fs.appendFile( + join(dir, 'perf-20260101-mixed.jsonl'), + '{"schema_version":1,"record_type":"operation"', + 'utf8', + ); + + const report = await buildReport(dir); + + expect(report.counts.parsed).toBe(1); + expect(report.counts.futureVersion).toBe(1); + expect(report.counts.unversioned).toBe(1); + expect(report.counts.malformed).toBe(1); + expect(report.counts.truncated).toBe(1); + expect(report.selfHealth.skipped).toBe(3); // malformed + future + unversioned + expect(report.selfHealth.truncated).toBe(1); + expect(report.selfHealth.lastWriteErrorCode).toBeUndefined(); + expect(report.selfHealth.evictionCount).toBeUndefined(); + }); + + // --- P11: baseline pools ALL matching baseline rows per dimension --- + + it('baseline pools all matching same-version/different-sha rows per dimension (not last-build overwrite)', async () => { + // Two baseline builds sharing version "0.10.0" but different git_shas, + // same dimensions. Pooled p50 must be computed over ALL baseline rows. + // group aaa: [100, 200] -> individual p50 = 100 + // group bbb: [300, 400] -> individual p50 = 300 + // pooled: [100, 200, 300, 400] -> p50 = 200 + await writeJsonl(dir, 'perf-20260101-aaa.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.10.0', + git_sha: 'aaa', + operation_id: 'a1', + operation_elapsed_ms: 100, + }), + ), + JSON.stringify( + makeOperation({ + llxprt_version: '0.10.0', + git_sha: 'aaa', + operation_id: 'a2', + operation_elapsed_ms: 200, + }), + ), + ]); + await writeJsonl(dir, 'perf-20260102-bbb.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.10.0', + git_sha: 'bbb', + operation_id: 'b1', + operation_elapsed_ms: 300, + }), + ), + JSON.stringify( + makeOperation({ + llxprt_version: '0.10.0', + git_sha: 'bbb', + operation_id: 'b2', + operation_elapsed_ms: 400, + }), + ), + ]); + // Non-baseline group compared against the pooled baseline. + await writeJsonl(dir, 'perf-20260103-ccc.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.11.0', + git_sha: 'ccc', + operation_elapsed_ms: 500, + }), + ), + ]); + + const report = await buildReport(dir, '0.10.0'); + + // Baseline build groups are preserved in output. + const baseAaa = report.groups.find((g) => g.build.git_sha === 'aaa'); + const baseBbb = report.groups.find((g) => g.build.git_sha === 'bbb'); + expect(baseAaa).toBeDefined(); + expect(baseBbb).toBeDefined(); + expect(baseAaa!.isBaseline).toBe(true); + expect(baseBbb!.isBaseline).toBe(true); + + const newGroup = report.groups.find((g) => g.build.git_sha === 'ccc'); + expect(newGroup).toBeDefined(); + expect(newGroup!.baselineComparison!.matched).toBe(true); + const delta = newGroup!.baselineComparison!.deltas!['operation_elapsed_ms']; + // Pooled baseline p50 = 200, so delta = 500 - 200 = 300. + // The previous overwrite behavior compared against bbb's p50 (300), + // yielding 200 — this assertion fails that behavior. + expect(delta.absolute).toBe(300); + }); + + it('baseline by exact sha pools all matching rows across builds', async () => { + // Same git_sha appears in two version groups (same dimensions). Both are + // baseline; the pooled p50 is over all four rows. + await writeJsonl(dir, 'perf-20260101-a.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.10.0', + git_sha: 'sha1', + operation_elapsed_ms: 100, + }), + ), + JSON.stringify( + makeOperation({ + llxprt_version: '0.10.0', + git_sha: 'sha1', + operation_elapsed_ms: 200, + }), + ), + ]); + await writeJsonl(dir, 'perf-20260102-b.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.10.1', + git_sha: 'sha1', + operation_elapsed_ms: 300, + }), + ), + JSON.stringify( + makeOperation({ + llxprt_version: '0.10.1', + git_sha: 'sha1', + operation_elapsed_ms: 400, + }), + ), + ]); + await writeJsonl(dir, 'perf-20260103-c.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.11.0', + git_sha: 'sha2', + operation_elapsed_ms: 500, + }), + ), + ]); + + const report = await buildReport(dir, 'sha1'); + const newGroup = report.groups.find((g) => g.build.git_sha === 'sha2'); + const delta = newGroup!.baselineComparison!.deltas!['operation_elapsed_ms']; + // Pooled over [100,200,300,400] -> p50 200; delta = 500 - 200 = 300. + expect(delta.absolute).toBe(300); + }); + + // --- P11: self-health skipped includes blank lines, excludes truncated --- + + it('self-health skipped includes blank lines (not double-counting truncated)', async () => { + await writeJsonl(dir, 'perf-20260101-mixed.jsonl', [ + JSON.stringify(makeOperation({ operation_id: 'valid-1' })), + '', // blank + ' ', // blank (whitespace-only) + 'not valid json', // malformed + ]); + + const report = await buildReport(dir); + + // skipped = malformed(1) + future(0) + unversioned(0) + blank(2) = 3 + expect(report.counts.blank).toBe(2); + expect(report.counts.malformed).toBe(1); + expect(report.counts.truncated).toBe(0); + expect(report.selfHealth.skipped).toBe(3); + expect(report.selfHealth.truncated).toBe(0); + }); + + // --- P11: formatter surfaces all four memory slopes for both axes --- + + it('formatter surfaces all four memory slopes for per-op and per-min', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify( + makeOperation({ + operation_id: 'op-1', + session_operation_index: 0, + rss_bytes: 50_000_000, + heap_used_bytes: 20_000_000, + external_bytes: 5_000_000, + array_buffers_bytes: 1_000_000, + }), + ), + JSON.stringify( + makeOperation({ + operation_id: 'op-2', + session_operation_index: 1, + rss_bytes: 60_000_000, + heap_used_bytes: 25_000_000, + external_bytes: 6_000_000, + array_buffers_bytes: 2_000_000, + }), + ), + JSON.stringify({ + schema_version: 1, + record_type: 'memory_sample', + ts: '2026-01-01T00:00:00.000Z', + rss_bytes: 50_000_000, + heap_used_bytes: 20_000_000, + external_bytes: 5_000_000, + array_buffers_bytes: 1_000_000, + uptime_ms: 0, + ms_since_last_operation: 0, + }), + JSON.stringify({ + schema_version: 1, + record_type: 'memory_sample', + ts: '2026-01-01T00:01:00.000Z', + rss_bytes: 60_000_000, + heap_used_bytes: 25_000_000, + external_bytes: 6_000_000, + array_buffers_bytes: 2_000_000, + uptime_ms: 60_000, + ms_since_last_operation: 30_000, + }), + ]); + + const report = await buildReport(dir); + const text = formatReport(report); + + // per-op: all four slopes present + expect(text).toContain('per-op:'); + expect(text).toContain('rss='); + expect(text).toContain('heap='); + expect(text).toContain('external='); + expect(text).toContain('array_buffers='); + // per-min: all four slopes present. Both per-op and per-min lines carry + // external/array_buffers, so each appears at least twice in the slopes + // section (once per axis). + const externalCount = (text.match(/external=/g) ?? []).length; + const arrayBuffersCount = (text.match(/array_buffers=/g) ?? []).length; + expect(externalCount).toBeGreaterThanOrEqual(2); + expect(arrayBuffersCount).toBeGreaterThanOrEqual(2); + expect(text).toContain('per-min:'); + }); + + // --- P11: pooled baseline p50 uses aggregated continuation token totals --- + + it('pooled baseline p50 uses the same aggregated continuation token totals as report groups (D1 join)', async () => { + // Baseline perf record carries persisted context_tokens: 1000. + // Token-usage rows for the baseline operation have an initial send (500) + // and one continuation (700), aggregated to 1200. + await writeJsonl(dir, 'perf-20260101-base.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.10.0', + git_sha: 'aaa', + operation_id: 'base-op', + context_tokens: 1000, + output_tokens: 200, + }), + ), + ]); + + // Token usage directory with continuation rows. + const tokenDir = await makeTempDir(); + try { + await writeJsonl(tokenDir, 'tokens.jsonl', [ + JSON.stringify({ + prompt_id: 'base-op', + actual_prompt_tokens: 500, + output_tokens: 100, + }), + JSON.stringify({ + prompt_id: 'base-op#continuation#1', + actual_prompt_tokens: 700, + output_tokens: 150, + }), + ]); + + const report = await buildReport(dir, undefined, undefined, tokenDir); + + const baselineGroup = report.groups.find( + (g) => g.build.git_sha === 'aaa', + ); + expect(baselineGroup).toBeDefined(); + // The aggregated join total (500 + 700 = 1200) replaces the persisted + // perf total (1000). + expect(baselineGroup!.p50.context_tokens).toBe(1200); + expect(baselineGroup!.p50.output_tokens).toBe(250); + } finally { + await fs.rm(tokenDir, { recursive: true, force: true }); + } + }); + + it('persisted baseline tokens differ from initial+continuation rows and delta proves joined value was used', async () => { + // Baseline: persisted context_tokens: 1000 (but the real joined total is + // 500 + 700 = 1200). Current: context_tokens: 1500. + // + // If the persisted baseline (1000) were used: delta = 1500 - 1000 = 500. + // If the joined baseline (1200) is used: delta = 1500 - 1200 = 300. + // Asserting 300 proves the joined value was used. + await writeJsonl(dir, 'perf-20260101-base.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.10.0', + git_sha: 'aaa', + operation_id: 'base-op', + context_tokens: 1000, + output_tokens: 200, + }), + ), + ]); + await writeJsonl(dir, 'perf-20260102-new.jsonl', [ + JSON.stringify( + makeOperation({ + llxprt_version: '0.11.0', + git_sha: 'bbb', + operation_id: 'new-op', + context_tokens: 1500, + output_tokens: 300, + }), + ), + ]); + + const tokenDir = await makeTempDir(); + try { + await writeJsonl(tokenDir, 'tokens.jsonl', [ + JSON.stringify({ + prompt_id: 'base-op', + actual_prompt_tokens: 500, + }), + JSON.stringify({ + prompt_id: 'base-op#continuation#1', + actual_prompt_tokens: 700, + }), + ]); + + const report = await buildReport(dir, '0.10.0', undefined, tokenDir); + + // Baseline p50 must use the joined total (1200), not persisted (1000). + const baselineGroup = report.groups.find( + (g) => g.build.git_sha === 'aaa', + ); + expect(baselineGroup!.p50.context_tokens).toBe(1200); + + const newGroup = report.groups.find((g) => g.build.git_sha === 'bbb'); + expect(newGroup!.baselineComparison!.matched).toBe(true); + const delta = newGroup!.baselineComparison!.deltas!['context_tokens']; + // delta = 1500 - 1200 = 300 (joined baseline), NOT 1500 - 1000 = 500. + expect(delta.absolute).toBe(300); + } finally { + await fs.rm(tokenDir, { recursive: true, force: true }); + } + }); + + it('preserves lower-nearest-rank p50 for even sample counts', async () => { + // With an even count of [100, 200], the lower nearest-rank p50 is 100 + // (sorted[0]), not the average (150). + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify( + makeOperation({ + operation_elapsed_ms: 100, + operation_id: 'op-a', + }), + ), + JSON.stringify( + makeOperation({ + operation_elapsed_ms: 200, + operation_id: 'op-b', + }), + ), + ]); + + const report = await buildReport(dir); + expect(report.groups[0].p50.operation_elapsed_ms).toBe(100); + + // With four values [100, 200, 300, 400], lower nearest-rank is 200. + await writeJsonl(dir, 'perf-20260102-run2.jsonl', [ + JSON.stringify( + makeOperation({ + operation_elapsed_ms: 300, + operation_id: 'op-c', + }), + ), + JSON.stringify( + makeOperation({ + operation_elapsed_ms: 400, + operation_id: 'op-d', + }), + ), + ]); + + const report2 = await buildReport(dir); + expect(report2.groups[0].p50.operation_elapsed_ms).toBe(200); + }); +}); diff --git a/packages/telemetry/src/perf/perfReport.tokenJoin.behavior.test.ts b/packages/telemetry/src/perf/perfReport.tokenJoin.behavior.test.ts new file mode 100644 index 0000000000..e32f244359 --- /dev/null +++ b/packages/telemetry/src/perf/perfReport.tokenJoin.behavior.test.ts @@ -0,0 +1,374 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Real-directory behavioral evidence for the production read-time continuation + * join (D1, AC-3). + * + * One initial send plus multiple continuations carry distinct `prompt_id` + * values in the telemetry-owned token-usage JSONL. The report derives each + * row's operation id at read time and joins ALL continuation rows onto the + * SINGLE matched perf operation, replacing the persisted perf token totals + * with the SUMMED joined actual_prompt_tokens / output_tokens. Unmatched + * operations retain their persisted perf token totals. Malformed / lifecycle + * rows in the token-usage directory are tolerated (counted, never fatal). + * + * No mocks — real files in two real directories (perf + token-usage). + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import { promises as fs } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { buildReport, assembleReport } from './perfReport.js'; +import { aggregateTokenUsageByOperation } from './perfReport.js'; +import { consumeTokenUsageDirectory } from './tokenUsageReader.js'; +import type { PerfOperationRecord } from './perfRecords.js'; + +const INITIAL_PROMPT_ID = 'sess-1#agentic-loop#aaaa'; + +function makeOperation( + overrides: Partial = {}, +): PerfOperationRecord { + return { + schema_version: 1, + record_type: 'operation', + ts: '2026-01-01T00:00:00.000Z', + session_id: 'sess-1', + operation_id: INITIAL_PROMPT_ID, + runtime_id: 'rt-1', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'hash-1', + llxprt_version: '0.10.0', + git_sha: 'abc1234', + runtime: 'cli', + platform: 'darwin', + provider: 'test-provider', + model: 'test-model', + context_tokens: 1000, + output_tokens: 500, + terminal_cols: 80, + terminal_rows: 24, + render_mode: 'ink', + concurrent_instances: 1, + status: 'completed', + client_prepare_ms: 10, + stream_handler_ms: 100, + ink_render_ms: 5, + ink_render_count: 3, + stdout_bytes: 1024, + stdout_write_calls: 5, + stdout_write_sync_ms: 2, + client_finalize_ms: 8, + provider_attempts: 1, + provider_attempt_sum_ms: 200, + provider_union_ms: 200, + tool_calls: 2, + tool_call_sum_ms: 50, + tool_union_ms: 50, + agent_activity_union_ms: 250, + operation_elapsed_ms: 1000, + approval_wait_ms: 0, + unclassified_elapsed_ms: 0, + session_operation_index: 0, + uptime_ms: 5000, + ...overrides, + }; +} + +async function makeTempDir(prefix: string): Promise { + const dir = join( + tmpdir(), + `${prefix}-${Date.now()}-${Math.random().toString(36).slice(2)}`, + ); + await fs.mkdir(dir, { recursive: true }); + return dir; +} + +async function writeJsonl( + dir: string, + name: string, + lines: string[], +): Promise { + const content = lines.join('\n'); + await fs.writeFile( + join(dir, name), + content + (content.endsWith('\n') ? '' : '\n'), + 'utf8', + ); +} + +function tokenLine(overrides: Record = {}): string { + return JSON.stringify({ + prompt_id: INITIAL_PROMPT_ID, + actual_prompt_tokens: 1000, + output_tokens: 100, + ...overrides, + }); +} + +describe('buildReport — read-time continuation join (D1, AC-3)', () => { + let perfDir: string; + let tokenDir: string; + + beforeEach(async () => { + perfDir = await makeTempDir('perf-join'); + tokenDir = await makeTempDir('token-join'); + }); + + afterEach(async () => { + await Promise.all([ + fs.rm(perfDir, { recursive: true, force: true }), + fs.rm(tokenDir, { recursive: true, force: true }), + ]); + }); + + it('one initial + multiple continuations join to exactly one operation with summed tokens', async () => { + // Perf: a single operation whose operation_id is the initial prompt id. + await writeJsonl(perfDir, 'perf-20260101-run1.jsonl', [ + JSON.stringify( + makeOperation({ + operation_id: INITIAL_PROMPT_ID, + context_tokens: 1000, + output_tokens: 500, + }), + ), + ]); + + // Token usage: initial + 2 continuations. + await writeJsonl(tokenDir, 'usage.jsonl', [ + tokenLine({ + prompt_id: INITIAL_PROMPT_ID, + actual_prompt_tokens: 1000, + output_tokens: 100, + }), + tokenLine({ + prompt_id: `${INITIAL_PROMPT_ID}#continuation#1`, + actual_prompt_tokens: 2000, + output_tokens: 200, + }), + tokenLine({ + prompt_id: `${INITIAL_PROMPT_ID}#continuation#2`, + actual_prompt_tokens: 3000, + output_tokens: 300, + }), + ]); + + const report = await buildReport(perfDir, undefined, undefined, tokenDir); + + // Exactly one operation group with one sample. + const ops = report.groups.flatMap((g) => g.sampleCount); + expect(ops).toEqual([1]); + + const group = report.groups[0]; + // Joined actual_prompt_tokens = 1000 + 2000 + 3000 = 6000 (replaces + // persisted context_tokens 1000). + expect(group.p50['context_tokens']).toBe(6000); + // Joined output_tokens = 100 + 200 + 300 = 600 (replaces persisted 500). + expect(group.p50['output_tokens']).toBe(600); + // Non-token metrics are untouched. + expect(group.p50['operation_elapsed_ms']).toBe(1000); + }); + + it('unmatched operations retain their persisted perf token totals', async () => { + await writeJsonl(perfDir, 'perf-20260101-run1.jsonl', [ + JSON.stringify( + makeOperation({ + operation_id: 'op-unmatched', + context_tokens: 9999, + output_tokens: 8888, + }), + ), + ]); + + // Token usage references a DIFFERENT operation id — no match. + await writeJsonl(tokenDir, 'usage.jsonl', [ + tokenLine({ + prompt_id: 'sess-other#agentic-loop#zzzz', + actual_prompt_tokens: 1111, + output_tokens: 2222, + }), + ]); + + const report = await buildReport(perfDir, undefined, undefined, tokenDir); + + const group = report.groups[0]; + // Persisted totals retained (no join match). + expect(group.p50['context_tokens']).toBe(9999); + expect(group.p50['output_tokens']).toBe(8888); + }); + + it('tolerates malformed and lifecycle rows in the token-usage directory without failing', async () => { + await writeJsonl(perfDir, 'perf-20260101-run1.jsonl', [ + JSON.stringify( + makeOperation({ + operation_id: INITIAL_PROMPT_ID, + context_tokens: 0, + output_tokens: 0, + }), + ), + ]); + + await writeJsonl(tokenDir, 'usage.jsonl', [ + tokenLine({ + prompt_id: INITIAL_PROMPT_ID, + actual_prompt_tokens: 400, + output_tokens: 40, + }), + tokenLine({ + prompt_id: `${INITIAL_PROMPT_ID}#continuation#1`, + actual_prompt_tokens: 600, + output_tokens: 60, + }), + JSON.stringify({ record_type: 'compression', before: 10, after: 5 }), + `this is not json`, + ``, + tokenLine({ + prompt_id: `${INITIAL_PROMPT_ID}#continuation#2`, + actual_prompt_tokens: 1000, + output_tokens: 100, + }), + ]); + + const report = await buildReport(perfDir, undefined, undefined, tokenDir); + + const group = report.groups[0]; + // 400 + 600 + 1000 = 2000, 40 + 60 + 100 = 200 — lifecycle/malformed/blank + // rows ignored, not fatal. + expect(group.p50['context_tokens']).toBe(2000); + expect(group.p50['output_tokens']).toBe(200); + }); + + it('does not mutate the input token rows', async () => { + await writeJsonl(perfDir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation({ operation_id: INITIAL_PROMPT_ID })), + ]); + await writeJsonl(tokenDir, 'usage.jsonl', [ + tokenLine({ prompt_id: INITIAL_PROMPT_ID, actual_prompt_tokens: 1000 }), + tokenLine({ + prompt_id: `${INITIAL_PROMPT_ID}#continuation#1`, + actual_prompt_tokens: 2000, + }), + ]); + + // Read the rows, snapshot the prompt ids, build the report, re-read. + const before = await consumeTokenUsageDirectory(tokenDir); + const beforeIds = before.rows.map((r) => r.promptId); + + await buildReport(perfDir, undefined, undefined, tokenDir); + + const after = await consumeTokenUsageDirectory(tokenDir); + expect(after.rows.map((r) => r.promptId)).toEqual(beforeIds); + expect(after.counts.turns).toBe(2); + }); + + it('omitting tokenUsageDir keeps persisted perf token totals (backward compatible)', async () => { + await writeJsonl(perfDir, 'perf-20260101-run1.jsonl', [ + JSON.stringify( + makeOperation({ + operation_id: INITIAL_PROMPT_ID, + context_tokens: 1234, + output_tokens: 567, + }), + ), + ]); + + const report = await buildReport(perfDir); + const group = report.groups[0]; + expect(group.p50['context_tokens']).toBe(1234); + expect(group.p50['output_tokens']).toBe(567); + }); +}); + +describe('assembleReport — direct join composition (D1)', () => { + it('groups provided token rows by deriveOperationId and matches to perf operation_id', async () => { + const tokenDir = await makeTempDir('token-asm'); + try { + await writeJsonl(tokenDir, 'usage.jsonl', [ + tokenLine({ + prompt_id: 'op-A', + actual_prompt_tokens: 10, + output_tokens: 1, + }), + tokenLine({ + prompt_id: 'op-A#continuation#1', + actual_prompt_tokens: 20, + output_tokens: 2, + }), + tokenLine({ + prompt_id: 'op-A#continuation#2', + actual_prompt_tokens: 30, + output_tokens: undefined, + }), + ]); + + const { rows } = await consumeTokenUsageDirectory(tokenDir); + const op = makeOperation({ + operation_id: 'op-A', + context_tokens: 999, + output_tokens: 999, + }); + + const report = assembleReport( + [ + { + op, + sourceFile: 'perf-20260101-x.jsonl', + runUuid: 'run-x', + }, + ], + new Map(), + { + files: 1, + bytes: 0, + parsed: 1, + malformed: 0, + futureVersion: 0, + unversioned: 0, + truncated: 0, + blank: 0, + }, + undefined, + undefined, + aggregateTokenUsageByOperation(rows), + ); + + // 10 + 20 + 30 = 60; output on first two only (third omitted) → 1 + 2 = 3. + expect(report.groups[0].p50['context_tokens']).toBe(60); + expect(report.groups[0].p50['output_tokens']).toBe(3); + } finally { + await fs.rm(tokenDir, { recursive: true, force: true }); + } + }); + + it('an empty token-usage directory yields no join (persisted totals retained)', async () => { + const tokenDir = await makeTempDir('token-empty'); + let perfDir = ''; + try { + const op = makeOperation({ + operation_id: 'op-B', + context_tokens: 42, + output_tokens: 7, + }); + perfDir = await writePerfOp(op); + const report = await buildReport(perfDir, undefined, undefined, tokenDir); + expect(report.groups[0].p50['context_tokens']).toBe(42); + expect(report.groups[0].p50['output_tokens']).toBe(7); + } finally { + if (perfDir !== '') { + await fs.rm(perfDir, { recursive: true, force: true }); + } + await fs.rm(tokenDir, { recursive: true, force: true }); + } + }); + + async function writePerfOp(op: PerfOperationRecord): Promise { + const p = await makeTempDir('perf-asm'); + await writeJsonl(p, 'perf-20260101-x.jsonl', [JSON.stringify(op)]); + return p; + } +}); diff --git a/packages/telemetry/src/perf/perfReport.ts b/packages/telemetry/src/perf/perfReport.ts new file mode 100644 index 0000000000..0f766f4183 --- /dev/null +++ b/packages/telemetry/src/perf/perfReport.ts @@ -0,0 +1,937 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Longitudinal perf report data model + stable human formatter (P11, + * REQ-3167-9, D7). + * + * Groups operations by build identity (`llxprt_version` + `git_sha`) within + * exact comparison dimensions (provider, model, render_mode, terminal_cols, + * terminal_rows). Computes sample count, contaminated sample count + * (`concurrent_instances >= 2`, NOT contended), p50 for meaningful timing / + * counter / token metrics, terminal status counts, and per-file memory slopes + * from P10. + * + * Without `--baseline`: prints grouped matched-dimension p50 / sample / + * self-health, NO delta. + * + * With `--baseline `: each non-baseline group is compared ONLY + * against baseline rows sharing identical dimensions; unmatched groups are + * reported as unmatched, NEVER pooled. p50 deltas are absolute and/or percent, + * grounded in finite math. + * + * D1 read-time join: token-usage / session rows carrying a `prompt_id` are + * joined to a perf operation by deriving the operation_id at read time via + * `promptId.split('#continuation#')[0]`. Since no persisted token-usage source + * exists in the perf JSONL format, a typed row consumer API is exposed for + * P12 / report integration rather than inventing a file format. + */ + +import type { + PerfOperationRecord, + PerfMemorySampleRecord, + PerfTerminalStatus, +} from './perfRecords.js'; +import { joinKeyFromPromptId } from './perfRecords.js'; +import type { PerfConsumerCounts } from './perfConsumer.js'; +import { + derivePerOperationMemorySlope, + derivePerMinuteMemorySlope, +} from './perfSlopeBridge.js'; +import type { + PerOperationMemorySlope, + PerMinuteMemorySlope, +} from './perfSlopeBridge.js'; + +// =========================================================================== +// D1 read-time join API (typed row consumer for P12 / report integration) +// =========================================================================== + +/** + * A token-usage / session-recording turn row that carries a `prompt_id`. These + * rows are streamed from the telemetry-owned token-usage JSONL directory by + * {@link consumeTokenUsageDirectory}; they are NOT persisted in the perf JSONL + * format. This type is the read-time join input for the report. + * + * `promptId` is the initial prompt id or a continuation + * (`${initial}#continuation#${n}`). The join key is derived by taking the + * first segment before `#continuation#`. `actualPromptTokens` is the per-send + * prompt/context token count; `outputTokens` is optional (omitted when the + * provider did not report it). + */ +export interface PerfTokenUsageRow { + readonly promptId: string; + readonly actualPromptTokens: number; + readonly outputTokens?: number; +} + +/** + * Groups token-usage rows by derived operation id. N continuation rows + * (sharing the same prefix before `#continuation#`) join to one operation + * without persisted child ids (D1). The original rows are never mutated. + * + * Returns a Map: operationId → token rows for that operation. + */ +export function joinTokenRowsByOperation( + tokenRows: readonly PerfTokenUsageRow[], +): ReadonlyMap { + const map = new Map(); + for (const row of tokenRows) { + const operationId = joinKeyFromPromptId(row.promptId); + const list = map.get(operationId); + if (list !== undefined) { + list.push(row); + } else { + map.set(operationId, [row]); + } + } + return map; +} + +/** + * Aggregated token usage for a single operation id — the SUMMED + * `actual_prompt_tokens` (context) and `output_tokens` across all continuation + * rows that join to that operation. This is the O(operation IDs) aggregation + * that production consumes incrementally instead of retaining all raw rows. + */ +export interface AggregatedTokenUsage { + readonly contextTokens: number; + readonly outputTokens: number; +} + +/** + * Aggregates token-usage rows by derived operation id into the compact + * {@link AggregatedTokenUsage} form. Provided as a bounded helper for tests + * that have in-memory rows; production uses {@link streamAndAggregateTokenUsage} + * to stream files and aggregate incrementally with O(operation IDs) memory. + */ +export function aggregateTokenUsageByOperation( + tokenRows: readonly PerfTokenUsageRow[], +): ReadonlyMap { + const byOp = new Map(); + for (const row of tokenRows) { + const operationId = joinKeyFromPromptId(row.promptId); + const existing = byOp.get(operationId); + if (existing !== undefined) { + byOp.set(operationId, { + contextTokens: existing.contextTokens + row.actualPromptTokens, + outputTokens: existing.outputTokens + (row.outputTokens ?? 0), + }); + } else { + byOp.set(operationId, { + contextTokens: row.actualPromptTokens, + outputTokens: row.outputTokens ?? 0, + }); + } + } + return byOp; +} + +/** + * Streams token-usage files from a directory one file at a time and aggregates + * by derived operation id incrementally, WITHOUT retaining all raw rows. + * O(operation IDs) memory, not O(total rows). Files are visited in sorted + * order; each file is streamed line-by-line. A missing directory yields an + * empty aggregation. Other genuine filesystem errors propagate. + */ +async function streamAndAggregateTokenUsage( + tokenUsageDir: string, +): Promise> { + const { streamTokenUsageDirectory } = await import('./tokenUsageReader.js'); + const byOp = new Map(); + for await (const { entry } of streamTokenUsageDirectory(tokenUsageDir)) { + if (entry.kind !== 'turn') continue; + const operationId = joinKeyFromPromptId(entry.row.promptId); + const existing = byOp.get(operationId); + if (existing !== undefined) { + byOp.set(operationId, { + contextTokens: existing.contextTokens + entry.row.actualPromptTokens, + outputTokens: existing.outputTokens + (entry.row.outputTokens ?? 0), + }); + } else { + byOp.set(operationId, { + contextTokens: entry.row.actualPromptTokens, + outputTokens: entry.row.outputTokens ?? 0, + }); + } + } + return byOp; +} + +// =========================================================================== +// Data model +// =========================================================================== + +/** The exact comparison dimensions — never pooled. */ +export interface ReportDimensions { + readonly provider: string; + readonly model: string; + readonly render_mode: string; + readonly terminal_cols: number; + readonly terminal_rows: number; +} + +/** Build identity (the x-axis). */ +export interface ReportBuildIdentity { + readonly llxprt_version: string; + readonly git_sha: string; +} + +/** Per-file memory slopes computed from P10 functions (per run/file, not pooled). */ +export interface ReportFileMemorySlopes { + readonly sourceFile: string; + readonly runUuid: string; + readonly perOperation: PerOperationMemorySlope; + readonly perMinute: PerMinuteMemorySlope; +} + +/** + * A group of operations sharing the same build identity AND dimensions. + */ +export interface ReportGroup { + readonly dimensions: ReportDimensions; + readonly build: ReportBuildIdentity; + readonly sampleCount: number; + /** Operations with `concurrent_instances >= 2` (contamination, NOT contended). */ + readonly contaminatedSampleCount: number; + /** p50 for each meaningful metric, or null if no samples. */ + readonly p50: Readonly>; + readonly terminalStatusCounts: Readonly>; + readonly memorySlopes: readonly ReportFileMemorySlopes[]; +} + +/** + * Baseline comparison result for a non-baseline group. + */ +export interface BaselineComparison { + readonly matched: boolean; + /** The p50 deltas vs the baseline group (only present when matched). */ + readonly deltas?: Readonly< + Record< + string, + { readonly absolute: number; readonly percent: number | null } + > + >; +} + +/** + * A report group optionally annotated with its baseline comparison. + */ +export interface ReportGroupWithBaseline extends ReportGroup { + readonly isBaseline: boolean; + readonly baselineComparison?: BaselineComparison; +} + +/** + * Self-health surfaced in the report. + * + * Reader health (`skipped`, `truncated`) always derives from consumer + * counts. Process-local health (`lastWriteErrorCode`, `evictionCount`) is + * modelled as a three-state value: + * - `undefined` — process-local health is unavailable (not wired by the + * caller; e.g. batch reads or default-off CLI). This is NOT a false fact. + * - `null` (lastWriteErrorCode) — known: the last write succeeded. + * - `0` (evictionCount) — known: zero evictions occurred. + * Distinguishing `undefined` from `null`/`0` prevents the report from + * claiming "no write errors" or "zero evictions" when those facts were + * simply never supplied. + */ +export interface ReportSelfHealth { + readonly skipped: number; + readonly truncated: number; + readonly lastWriteErrorCode: string | null | undefined; + readonly evictionCount: number | undefined; +} + +/** The full report result. */ +export interface ReportResult { + readonly groups: readonly ReportGroupWithBaseline[]; + readonly counts: PerfConsumerCounts; + readonly selfHealth: ReportSelfHealth; + readonly baseline: { + readonly value: string; + readonly found: boolean; + } | null; +} + +// =========================================================================== +// Metric keys (meaningful recorded timing / counter / token metrics) +// =========================================================================== + +/** The p50-eligible metric keys from the operation record. */ +const P50_METRIC_KEYS = [ + 'operation_elapsed_ms', + 'client_prepare_ms', + 'stream_handler_ms', + 'ink_render_ms', + 'client_finalize_ms', + 'stdout_write_sync_ms', + 'provider_attempt_sum_ms', + 'provider_union_ms', + 'tool_call_sum_ms', + 'tool_union_ms', + 'agent_activity_union_ms', + 'approval_wait_ms', + 'unclassified_elapsed_ms', + 'ink_render_count', + 'stdout_write_calls', + 'provider_attempts', + 'tool_calls', + 'context_tokens', + 'output_tokens', + 'stdout_bytes', +] as const; + +export type P50MetricKey = (typeof P50_METRIC_KEYS)[number]; + +// =========================================================================== +// Internal grouping types +// =========================================================================== + +interface GroupedOperation { + readonly op: PerfOperationRecord; + readonly sourceFile: string; + readonly runUuid: string; +} + +interface ReportGroupData { + readonly dims: ReportDimensions; + readonly build: ReportBuildIdentity; + readonly ops: GroupedOperation[]; +} + +// =========================================================================== +// Helpers +// =========================================================================== + +function p50(values: readonly number[]): number | null { + if (values.length === 0) return null; + const sorted = [...values].sort((a, b) => a - b); + const mid = Math.floor(sorted.length / 2); + if (sorted.length % 2 === 1) return sorted[mid]; + return sorted[mid - 1]; +} + +function dimensionKey(d: ReportDimensions): string { + return [ + d.provider, + d.model, + d.render_mode, + d.terminal_cols, + d.terminal_rows, + ].join('|'); +} + +function buildKey(b: ReportBuildIdentity): string { + return `${b.llxprt_version}@${b.git_sha}`; +} + +function dimsFromOperation(op: PerfOperationRecord): ReportDimensions { + return { + provider: op.provider, + model: op.model, + render_mode: op.render_mode, + terminal_cols: op.terminal_cols, + terminal_rows: op.terminal_rows, + }; +} + +function buildFromOperation(op: PerfOperationRecord): ReportBuildIdentity { + return { llxprt_version: op.llxprt_version, git_sha: op.git_sha }; +} + +function extractMetric(op: PerfOperationRecord, key: P50MetricKey): number { + return op[key]; +} + +function isBaselineBuild( + build: ReportBuildIdentity, + baseline: string, +): boolean { + return build.llxprt_version === baseline || build.git_sha === baseline; +} + +const TERMINAL_STATUSES: readonly PerfTerminalStatus[] = [ + 'completed', + 'error', + 'cancelled_before_send', + 'cancelled_during_api', + 'cancelled_during_tool', + 'cancelled_during_approval', + 'superseded', +]; + +// =========================================================================== +// Report builder +// =========================================================================== + +/** + * Builds a longitudinal report from all `perf-*.jsonl` files in a directory. + * + * Groups operations by build identity within exact dimensions. Computes p50, + * contamination, terminal status counts, and per-file memory slopes. + * + * Without baseline: groups + p50 / sample / self-health, NO delta. + * With baseline: each non-baseline group is compared to the baseline group + * with matching dimensions; unmatched groups reported as unmatched. + * + * Self-health (`lastWriteErrorCode`, `evictionCount`): when not provided they + * remain `undefined` (unavailable), NOT `null`/`0` (known-no-error/zero). P12 + * wires the live sink/retention state; the CLI command injects it when the + * active perf runtime is available. When unavailable the report formats them + * honestly as "unavailable" rather than claiming zero errors/evictions. + * + * `tokenUsageDir` (optional) is a telemetry-owned token-usage JSONL directory. + * When provided, its turn rows are STREAMED file-by-file and aggregated by + * derived operation id incrementally (O(operation IDs) memory, never retaining + * all raw rows) so that one initial send plus its N continuations contribute + * SUMMED `actual_prompt_tokens`/`output_tokens` to the single matched perf + * operation (D1 continuation join). Unmatched operations retain their + * persisted perf token totals. The join never imports packages/agents and + * never mutates the token rows. + */ +export async function buildReport( + dir: string, + baseline?: string, + selfHealth?: Partial, + tokenUsageDir?: string, +): Promise { + const { consumePerfDirectory } = await import('./perfConsumer.js'); + const { entries, counts } = await consumePerfDirectory(dir); + + const groupedOps: GroupedOperation[] = []; + const memorySamplesByFile = new Map(); + + for (const ce of entries) { + if (ce.entry.kind !== 'ok') continue; + if (ce.entry.record.record_type === 'operation') { + groupedOps.push({ + op: ce.entry.record, + sourceFile: ce.sourceFile, + runUuid: ce.runUuid, + }); + } else { + const list = memorySamplesByFile.get(ce.sourceFile); + if (list !== undefined) { + list.push(ce.entry.record); + } else { + memorySamplesByFile.set(ce.sourceFile, [ce.entry.record]); + } + } + } + + // Stream and aggregate token rows by operation id incrementally — never + // retain all raw rows for the entire directory. + let aggregatedTokens: ReadonlyMap | undefined; + if (tokenUsageDir !== undefined) { + aggregatedTokens = await streamAndAggregateTokenUsage(tokenUsageDir); + } + + return assembleReport( + groupedOps, + memorySamplesByFile, + counts, + baseline, + selfHealth, + aggregatedTokens, + ); +} + +/** + * Groups operations by (dimensions, build) into a map keyed by the composite key. + */ +function groupOperationsByDimension( + groupedOps: readonly GroupedOperation[], +): Map { + const groupMap = new Map(); + for (const gop of groupedOps) { + const dims = dimsFromOperation(gop.op); + const build = buildFromOperation(gop.op); + const key = `${dimensionKey(dims)}::${buildKey(build)}`; + let g = groupMap.get(key); + if (g === undefined) { + g = { dims, build, ops: [] }; + groupMap.set(key, g); + } + g.ops.push(gop); + } + return groupMap; +} + +/** + * Resolves a token metric for one operation from the aggregated join map, or + * `undefined` when there is no join match (caller falls back to the persisted + * perf total). Extracted so {@link computeP50Values} stays shallow. + */ +function joinedTokenValue( + op: PerfOperationRecord, + metricKey: P50MetricKey, + aggregatedTokens: ReadonlyMap, +): number | undefined { + const joined = aggregatedTokens.get(op.operation_id); + if (joined === undefined) return undefined; + if (metricKey === 'context_tokens') return joined.contextTokens; + if (metricKey === 'output_tokens') return joined.outputTokens; + return undefined; +} + +/** + * Computes p50 values for all P50 metric keys from the given operation records. + * + * When `aggregatedTokens` is provided, matched operations (operation_id present + * in the join map) use the SUMMED joined token metrics (actual_prompt_tokens → + * context_tokens, output_tokens → output_tokens) instead of the persisted perf + * token totals. This is the read-time continuation join (D1): N continuation + * rows collapse onto the single perf operation. Unmatched operations retain + * their persisted perf token totals. The join never adds to persisted totals + * (it replaces them for matched operations), so continuations are never + * double-counted. + */ +function computeP50Values( + opsRecords: readonly PerfOperationRecord[], + aggregatedTokens?: ReadonlyMap, +): Record { + const p50Values: Record = {}; + for (const metricKey of P50_METRIC_KEYS) { + const values = opsRecords.map((o) => { + if (aggregatedTokens !== undefined) { + const joined = joinedTokenValue(o, metricKey, aggregatedTokens); + if (joined !== undefined) return joined; + } + return extractMetric(o, metricKey); + }); + p50Values[metricKey] = p50(values); + } + return p50Values; +} + +/** + * Counts terminal statuses for each of the seven statuses. + */ +function countTerminalStatuses( + opsRecords: readonly PerfOperationRecord[], +): Record { + const terminalStatusCounts = {} as Record; + for (const status of TERMINAL_STATUSES) { + terminalStatusCounts[status] = opsRecords.filter( + (o) => o.status === status, + ).length; + } + return terminalStatusCounts; +} + +/** + * Computes per-file memory slopes (per run/file, never pooled across files). + * + * The run UUID always comes from an actual operation in the file (never an + * invented `unknown` fallback): the consumer guarantees a non-null run UUID + * for every perf JSONL file, and `sourceFile` is derived from the same op set. + */ +function computeMemorySlopes( + ops: readonly GroupedOperation[], + memorySamplesByFile: ReadonlyMap, +): ReportFileMemorySlopes[] { + const byFile = new Map< + string, + { readonly runUuid: string; readonly fileOps: PerfOperationRecord[] } + >(); + for (const gop of ops) { + let entry = byFile.get(gop.sourceFile); + if (entry === undefined) { + entry = { runUuid: gop.runUuid, fileOps: [] }; + byFile.set(gop.sourceFile, entry); + } + entry.fileOps.push(gop.op); + } + + const memorySlopes: ReportFileMemorySlopes[] = []; + for (const [sourceFile, { runUuid, fileOps }] of byFile) { + const fileSamples = memorySamplesByFile.get(sourceFile) ?? []; + memorySlopes.push({ + sourceFile, + runUuid, + perOperation: derivePerOperationMemorySlope(fileOps), + perMinute: derivePerMinuteMemorySlope(fileSamples), + }); + } + return memorySlopes; +} + +/** + * Builds a single `ReportGroupWithBaseline` from grouped operation data. + */ +function buildGroup( + g: ReportGroupData, + memorySamplesByFile: ReadonlyMap, + baseline: string | undefined, + aggregatedTokens?: ReadonlyMap, +): ReportGroupWithBaseline { + const opsRecords = g.ops.map((o) => o.op); + const group: ReportGroup = { + dimensions: g.dims, + build: g.build, + sampleCount: opsRecords.length, + contaminatedSampleCount: opsRecords.filter( + (o) => o.concurrent_instances >= 2, + ).length, + p50: computeP50Values(opsRecords, aggregatedTokens), + terminalStatusCounts: countTerminalStatuses(opsRecords), + memorySlopes: computeMemorySlopes(g.ops, memorySamplesByFile), + }; + return { + ...group, + isBaseline: baseline !== undefined && isBaselineBuild(g.build, baseline), + }; +} + +/** + * Computes pooled baseline p50 lookups by dimension key. + * + * When `--baseline` is an exact version (or sha) matching multiple git_sha + * builds with the same dimensions, ALL matching baseline operation rows for + * those dimensions are pooled and the p50 is computed over every row — not + * just whichever baseline build group was last in a Map (the previous + * overwrite bug). Baseline build groups are preserved as separate output + * groups; only the comparison baseline is pooled. + */ +function buildPooledBaselineByDims( + baselineOps: readonly GroupedOperation[], + aggregatedTokens?: ReadonlyMap, +): ReadonlyMap>> { + const byDims = new Map(); + for (const gop of baselineOps) { + const dk = dimensionKey(dimsFromOperation(gop.op)); + const list = byDims.get(dk); + if (list !== undefined) { + list.push(gop.op); + } else { + byDims.set(dk, [gop.op]); + } + } + + const result = new Map>>(); + for (const [dk, ops] of byDims) { + result.set(dk, computeP50Values(ops, aggregatedTokens)); + } + return result; +} + +/** + * Computes the baseline comparison for a non-baseline group. Returns + * `matched: false` when no baseline shares the group's dimensions; otherwise + * computes per-metric absolute and percent deltas. + */ +function compareAgainstBaseline( + group: ReportGroupWithBaseline, + baselineP50: Readonly>, +): BaselineComparison { + const deltas: Record< + string, + { readonly absolute: number; readonly percent: number | null } + > = {}; + for (const metricKey of P50_METRIC_KEYS) { + const current = group.p50[metricKey]; + const base = baselineP50[metricKey]; + if (current !== null && base !== null) { + const absolute = current - base; + const percent = base !== 0 ? (absolute / base) * 100 : null; + deltas[metricKey] = { absolute, percent }; + } + } + return { matched: true, deltas }; +} + +/** + * Applies baseline comparison to all non-baseline groups using the pooled + * baseline p50 map (computed over ALL matching baseline rows per dimension). + */ +function applyBaseline( + groups: readonly ReportGroupWithBaseline[], + baseline: string, + pooledBaselineByDims: ReadonlyMap< + string, + Readonly> + >, +): { + readonly groups: readonly ReportGroupWithBaseline[]; + readonly baselineInfo: { readonly value: string; readonly found: boolean }; +} { + const compared = groups.map((group) => { + if (group.isBaseline) return group; + const baselineP50 = pooledBaselineByDims.get( + dimensionKey(group.dimensions), + ); + if (baselineP50 === undefined) { + return { ...group, baselineComparison: { matched: false } }; + } + return { + ...group, + baselineComparison: compareAgainstBaseline(group, baselineP50), + }; + }); + + return { + groups: compared, + baselineInfo: { + value: baseline, + found: pooledBaselineByDims.size > 0, + }, + }; +} + +/** + * Resolves self-health with fallback defaults from consumer counts. + * + * `skipped` includes every non-parsed skipped line EXCEPT truncated (which + * remains separately surfaced): malformed + future + unversioned + blank. + * Truncated is never double-counted here. + * + * Process-local health (`lastWriteErrorCode`, `evictionCount`) is NOT + * defaulted to a false fact: when the caller does not supply them they + * remain `undefined` (unavailable), not `null`/`0` (known-no-error/zero). + */ +function resolveSelfHealth( + counts: PerfConsumerCounts, + selfHealth?: Partial, +): ReportSelfHealth { + return { + skipped: + selfHealth?.skipped ?? + counts.malformed + + counts.futureVersion + + counts.unversioned + + counts.blank, + truncated: selfHealth?.truncated ?? counts.truncated, + lastWriteErrorCode: selfHealth?.lastWriteErrorCode, + evictionCount: selfHealth?.evictionCount, + }; +} + +/** + * Assembles a report from pre-grouped data. Exposed for testing and P12 + * integration where entries may come from a non-filesystem source. + * + * `aggregatedTokens` (optional) is the pre-aggregated read-time join input: + * per-operation-id SUMMED `actual_prompt_tokens`/`output_tokens` aggregated + * incrementally from the token-usage JSONL directory. For every operation + * present in the join map, the report's token metrics use the SUMMED values + * instead of the persisted perf totals (D1 continuation join). Unmatched + * operations retain their persisted perf token totals. + */ +export function assembleReport( + groupedOps: readonly GroupedOperation[], + memorySamplesByFile: ReadonlyMap, + counts: PerfConsumerCounts, + baseline: string | undefined, + selfHealth?: Partial, + aggregatedTokens?: ReadonlyMap, +): ReportResult { + const groupMap = groupOperationsByDimension(groupedOps); + + const groups: ReportGroupWithBaseline[] = []; + for (const g of groupMap.values()) { + groups.push(buildGroup(g, memorySamplesByFile, baseline, aggregatedTokens)); + } + + // Sort groups deterministically. + groups.sort((a, b) => { + const dk = dimensionKey(a.dimensions).localeCompare( + dimensionKey(b.dimensions), + ); + if (dk !== 0) return dk; + return buildKey(a.build).localeCompare(buildKey(b.build)); + }); + + // Apply baseline comparison if requested. + let baselineResult: ReportResult['baseline'] = null; + let finalGroups: readonly ReportGroupWithBaseline[] = groups; + + if (baseline !== undefined) { + const baselineOps = groupedOps.filter((gop) => + isBaselineBuild(buildFromOperation(gop.op), baseline), + ); + const pooledBaselineByDims = buildPooledBaselineByDims( + baselineOps, + aggregatedTokens, + ); + const result = applyBaseline(groups, baseline, pooledBaselineByDims); + finalGroups = result.groups; + baselineResult = result.baselineInfo; + } + + return { + groups: finalGroups, + counts, + selfHealth: resolveSelfHealth(counts, selfHealth), + baseline: baselineResult, + }; +} + +// =========================================================================== +// Stable human formatter +// =========================================================================== + +/** + * Formats a report result into a stable, human-readable string. + * + * The output is deterministic: groups are listed in sorted order, metrics are + * listed in a fixed order, and no delta appears when no baseline was provided. + */ +export function formatReport(report: ReportResult): string { + const lines: string[] = []; + + lines.push('Perf Report'); + lines.push('==========='); + lines.push(''); + + const c = report.counts; + lines.push(`Files scanned: ${c.files} (${formatBytes(c.bytes)})`); + lines.push( + `Records: parsed=${c.parsed} malformed=${c.malformed} future=${c.futureVersion} unversioned=${c.unversioned} truncated=${c.truncated} blank=${c.blank}`, + ); + lines.push(''); + + const sh = report.selfHealth; + lines.push('Self-health:'); + lines.push(` skipped: ${sh.skipped}`); + lines.push(` truncated: ${sh.truncated}`); + lines.push( + ` last write error: ${formatWriteErrorCode(sh.lastWriteErrorCode)}`, + ); + lines.push(` evictions: ${sh.evictionCount ?? 'unavailable'}`); + lines.push(''); + + if (report.baseline !== null) { + if (report.baseline.found) { + lines.push(`Baseline: ${report.baseline.value} (matched)`); + } else { + lines.push(`Baseline: ${report.baseline.value} (NO MATCH FOUND)`); + } + lines.push(''); + } + + if (report.groups.length === 0) { + lines.push('No operation records found.'); + return lines.join('\n'); + } + + for (const group of report.groups) { + lines.push(formatGroup(group)); + lines.push(''); + } + + return lines.join('\n').trimEnd(); +} + +function formatGroup(group: ReportGroupWithBaseline): string { + const lines: string[] = []; + const d = group.dimensions; + const b = group.build; + lines.push( + `[${b.llxprt_version}@${b.git_sha}] provider=${d.provider} model=${d.model} render=${d.render_mode} cols=${d.terminal_cols} rows=${d.terminal_rows}`, + ); + lines.push( + ` samples: ${group.sampleCount} contaminated: ${group.contaminatedSampleCount}`, + ); + + const statusParts: string[] = []; + for (const [status, count] of Object.entries(group.terminalStatusCounts)) { + if (count > 0) { + statusParts.push(`${status}=${count}`); + } + } + if (statusParts.length > 0) { + lines.push(` status: ${statusParts.join(' ')}`); + } + + const p50Lines: string[] = []; + for (const metricKey of P50_METRIC_KEYS) { + const val = group.p50[metricKey]; + if (val === null) continue; + p50Lines.push(formatMetricLine(metricKey, val, group)); + } + if (p50Lines.length > 0) { + lines.push(' p50:'); + lines.push(...p50Lines); + } + + if (group.baselineComparison && !group.baselineComparison.matched) { + lines.push(' WARNING: UNMATCHED (no baseline with same dimensions)'); + } + + if (group.memorySlopes.length > 0) { + lines.push(' memory slopes (per file):'); + for (const ms of group.memorySlopes) { + lines.push(` ${ms.sourceFile} (run ${ms.runUuid}):`); + const ops = ms.perOperation; + const mins = ms.perMinute; + lines.push( + ` per-op: rss=${formatNullable(ops.rss_bytes_per_operation)} heap=${formatNullable(ops.heap_used_bytes_per_operation)} external=${formatNullable(ops.external_bytes_per_operation)} array_buffers=${formatNullable(ops.array_buffers_bytes_per_operation)}`, + ); + lines.push( + ` per-min: rss=${formatNullable(mins.rss_bytes_per_minute)} heap=${formatNullable(mins.heap_used_bytes_per_minute)} external=${formatNullable(mins.external_bytes_per_minute)} array_buffers=${formatNullable(mins.array_buffers_bytes_per_minute)}`, + ); + } + } + + return lines.join('\n'); +} + +function formatMetricLine( + metricKey: string, + val: number, + group: ReportGroupWithBaseline, +): string { + let line = ` ${metricKey}: p50=${formatNumber(val)}`; + const bc = group.baselineComparison; + if (bc?.matched === true) { + const delta = bc.deltas?.[metricKey]; + if (delta !== undefined) { + line += formatDelta(delta); + } + } + return line; +} + +function formatPercent(percent: number | null): string { + if (percent === null) return 'n/a%'; + const sign = percent >= 0 ? '+' : ''; + return `${sign}${percent.toFixed(1)}%`; +} + +function formatDelta(delta: { + readonly absolute: number; + readonly percent: number | null; +}): string { + const sign = delta.absolute >= 0 ? '+' : ''; + return ` (delta ${sign}${formatNumber(delta.absolute)} / ${formatPercent(delta.percent)})`; +} + +function formatBytes(bytes: number): string { + if (bytes < 1024) return `${bytes} B`; + if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KiB`; + return `${(bytes / (1024 * 1024)).toFixed(1)} MiB`; +} + +function formatNumber(n: number): string { + if (Number.isInteger(n)) return n.toString(); + return n.toFixed(2); +} + +function formatNullable(n: number | null): string { + return n === null ? 'n/a' : formatNumber(n); +} + +/** + * Formats the process-local write-error code for the self-health surface, + * preserving the three-state distinction: `undefined` (unavailable) → + * 'unavailable'; `null` (known: last write succeeded) → 'none'; a string + * errno → the code itself. + */ +function formatWriteErrorCode(code: string | null | undefined): string { + if (code === undefined) return 'unavailable'; + return code ?? 'none'; +} diff --git a/packages/telemetry/src/perf/perfSchema.boundary.behavior.test.ts b/packages/telemetry/src/perf/perfSchema.boundary.behavior.test.ts new file mode 100644 index 0000000000..68c81cdb3e --- /dev/null +++ b/packages/telemetry/src/perf/perfSchema.boundary.behavior.test.ts @@ -0,0 +1,511 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Behavioral boundary tests for the tightened v1 perf schema (P04A correction A). + * + * The schema must encode accepted value boundaries: + * - ts: ISO 8601 timestamp + * - all durations except unclassified_elapsed_ms: finite and non-negative + * - unclassified_elapsed_ms: finite but may be negative + * - counts/tokens/geometry/index/concurrent_instances: integers with + * appropriate non-negative/minimum constraints + * - byte/memory/uptime/sample ages: finite non-negative + * + * Unknown-field tolerance must be preserved. + */ + +import { describe, it, expect } from 'bun:test'; +import { + PerfOperationRecordSchema, + PerfMemorySampleRecordSchema, +} from './perfRecords.js'; + +function operationRecord( + overrides: Record = {}, +): Record { + return { + schema_version: 1, + record_type: 'operation', + ts: '2026-08-08T12:00:00.000Z', + session_id: 'sess-abc', + operation_id: 'sess-abc#agentic-loop#f7e2', + runtime_id: 'rt-main', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'sha256:project-hash', + llxprt_version: '0.11.0', + git_sha: 'abc1234', + runtime: 'bun-1.3.14', + platform: 'darwin-arm64', + provider: 'openai', + model: 'gpt-4o', + context_tokens: 1000, + output_tokens: 500, + terminal_cols: 120, + terminal_rows: 40, + render_mode: 'incremental', + concurrent_instances: 1, + status: 'completed', + client_prepare_ms: 5, + stream_handler_ms: 10, + ink_render_ms: 20, + ink_render_count: 3, + stdout_bytes: 4096, + stdout_write_calls: 3, + stdout_write_sync_ms: 2, + client_finalize_ms: 1, + provider_attempts: 1, + provider_attempt_sum_ms: 800, + provider_union_ms: 800, + tool_calls: 2, + tool_call_sum_ms: 300, + tool_union_ms: 280, + agent_activity_union_ms: 1000, + operation_elapsed_ms: 1200, + approval_wait_ms: 0, + unclassified_elapsed_ms: 100, + session_operation_index: 1, + uptime_ms: 50000, + ...overrides, + }; +} + +function memorySampleRecord( + overrides: Record = {}, +): Record { + return { + schema_version: 1, + record_type: 'memory_sample', + ts: '2026-08-08T12:01:00.000Z', + rss_bytes: 121_000_000, + heap_used_bytes: 61_000_000, + external_bytes: 25_500_000, + array_buffers_bytes: 1_550_000, + uptime_ms: 60000, + ms_since_last_operation: 30000, + ...overrides, + }; +} + +// --------------------------------------------------------------------------- +// ts — ISO 8601 timestamp +// --------------------------------------------------------------------------- + +describe('schema boundary — ts is ISO 8601', () => { + it('accepts a UTC ISO 8601 timestamp', () => { + expect( + PerfOperationRecordSchema.safeParse( + operationRecord({ ts: '2026-08-08T12:00:00.000Z' }), + ).success, + ).toBe(true); + }); + + it('accepts an offset ISO 8601 timestamp', () => { + expect( + PerfOperationRecordSchema.safeParse( + operationRecord({ ts: '2026-08-08T12:00:00.000+02:00' }), + ).success, + ).toBe(true); + }); + + it('rejects a non-ISO string', () => { + expect( + PerfOperationRecordSchema.safeParse(operationRecord({ ts: 'not-a-date' })) + .success, + ).toBe(false); + }); + + it('rejects a timestamp without timezone', () => { + expect( + PerfOperationRecordSchema.safeParse( + operationRecord({ ts: '2026-08-08T12:00:00' }), + ).success, + ).toBe(false); + }); +}); + +// --------------------------------------------------------------------------- +// Durations — finite and non-negative (except unclassified_elapsed_ms) +// --------------------------------------------------------------------------- + +describe('schema boundary — durations are finite and non-negative', () => { + const nonNegDurations = [ + 'client_prepare_ms', + 'stream_handler_ms', + 'ink_render_ms', + 'stdout_write_sync_ms', + 'client_finalize_ms', + 'provider_attempt_sum_ms', + 'provider_union_ms', + 'tool_call_sum_ms', + 'tool_union_ms', + 'agent_activity_union_ms', + 'operation_elapsed_ms', + 'approval_wait_ms', + ] as const; + + for (const field of nonNegDurations) { + it(`rejects negative ${field}`, () => { + expect( + PerfOperationRecordSchema.safeParse(operationRecord({ [field]: -1 })) + .success, + ).toBe(false); + }); + + it(`rejects Infinity for ${field}`, () => { + expect( + PerfOperationRecordSchema.safeParse( + operationRecord({ [field]: Number.POSITIVE_INFINITY }), + ).success, + ).toBe(false); + }); + + it(`rejects NaN for ${field}`, () => { + expect( + PerfOperationRecordSchema.safeParse( + operationRecord({ [field]: Number.NaN }), + ).success, + ).toBe(false); + }); + + it(`accepts zero for ${field}`, () => { + expect( + PerfOperationRecordSchema.safeParse(operationRecord({ [field]: 0 })) + .success, + ).toBe(true); + }); + } +}); + +// --------------------------------------------------------------------------- +// unclassified_elapsed_ms — finite but may be negative +// --------------------------------------------------------------------------- + +describe('schema boundary — unclassified_elapsed_ms is finite, may be negative', () => { + it('accepts a negative value', () => { + expect( + PerfOperationRecordSchema.safeParse( + operationRecord({ unclassified_elapsed_ms: -500 }), + ).success, + ).toBe(true); + }); + + it('accepts zero', () => { + expect( + PerfOperationRecordSchema.safeParse( + operationRecord({ unclassified_elapsed_ms: 0 }), + ).success, + ).toBe(true); + }); + + it('rejects Infinity', () => { + expect( + PerfOperationRecordSchema.safeParse( + operationRecord({ unclassified_elapsed_ms: Number.POSITIVE_INFINITY }), + ).success, + ).toBe(false); + }); + + it('rejects NaN', () => { + expect( + PerfOperationRecordSchema.safeParse( + operationRecord({ unclassified_elapsed_ms: Number.NaN }), + ).success, + ).toBe(false); + }); +}); + +// --------------------------------------------------------------------------- +// Counts — integers, non-negative +// --------------------------------------------------------------------------- + +describe('schema boundary — counts are non-negative integers', () => { + const countFields = [ + 'ink_render_count', + 'stdout_write_calls', + 'provider_attempts', + 'tool_calls', + ] as const; + + for (const field of countFields) { + it(`rejects negative ${field}`, () => { + expect( + PerfOperationRecordSchema.safeParse(operationRecord({ [field]: -1 })) + .success, + ).toBe(false); + }); + + it(`rejects non-integer ${field}`, () => { + expect( + PerfOperationRecordSchema.safeParse(operationRecord({ [field]: 1.5 })) + .success, + ).toBe(false); + }); + + it(`accepts zero for ${field}`, () => { + expect( + PerfOperationRecordSchema.safeParse(operationRecord({ [field]: 0 })) + .success, + ).toBe(true); + }); + } +}); + +// --------------------------------------------------------------------------- +// Tokens — integers, non-negative +// --------------------------------------------------------------------------- + +describe('schema boundary — tokens are non-negative integers', () => { + for (const field of ['context_tokens', 'output_tokens'] as const) { + it(`rejects negative ${field}`, () => { + expect( + PerfOperationRecordSchema.safeParse(operationRecord({ [field]: -1 })) + .success, + ).toBe(false); + }); + + it(`rejects non-integer ${field}`, () => { + expect( + PerfOperationRecordSchema.safeParse(operationRecord({ [field]: 1.5 })) + .success, + ).toBe(false); + }); + } +}); + +// --------------------------------------------------------------------------- +// Geometry — non-negative integers (unknown terminal geometry is zero — P12) +// --------------------------------------------------------------------------- + +describe('schema boundary — geometry are non-negative integers (unknown is zero)', () => { + for (const field of ['terminal_cols', 'terminal_rows'] as const) { + it(`accepts zero for ${field} (unknown terminal geometry)`, () => { + expect( + PerfOperationRecordSchema.safeParse(operationRecord({ [field]: 0 })) + .success, + ).toBe(true); + }); + + it(`rejects negative for ${field}`, () => { + expect( + PerfOperationRecordSchema.safeParse(operationRecord({ [field]: -1 })) + .success, + ).toBe(false); + }); + + it(`rejects non-integer for ${field}`, () => { + expect( + PerfOperationRecordSchema.safeParse(operationRecord({ [field]: 80.5 })) + .success, + ).toBe(false); + }); + + it(`accepts 1 for ${field}`, () => { + expect( + PerfOperationRecordSchema.safeParse(operationRecord({ [field]: 1 })) + .success, + ).toBe(true); + }); + } +}); + +// --------------------------------------------------------------------------- +// concurrent_instances — integer, minimum 1 +// --------------------------------------------------------------------------- + +describe('schema boundary — concurrent_instances is integer with minimum 1', () => { + it('rejects zero', () => { + expect( + PerfOperationRecordSchema.safeParse( + operationRecord({ concurrent_instances: 0 }), + ).success, + ).toBe(false); + }); + + it('rejects negative', () => { + expect( + PerfOperationRecordSchema.safeParse( + operationRecord({ concurrent_instances: -1 }), + ).success, + ).toBe(false); + }); + + it('rejects non-integer', () => { + expect( + PerfOperationRecordSchema.safeParse( + operationRecord({ concurrent_instances: 1.5 }), + ).success, + ).toBe(false); + }); + + it('accepts 1', () => { + expect( + PerfOperationRecordSchema.safeParse( + operationRecord({ concurrent_instances: 1 }), + ).success, + ).toBe(true); + }); +}); + +// --------------------------------------------------------------------------- +// session_operation_index — integer, non-negative +// --------------------------------------------------------------------------- + +describe('schema boundary — session_operation_index is non-negative integer', () => { + it('rejects negative', () => { + expect( + PerfOperationRecordSchema.safeParse( + operationRecord({ session_operation_index: -1 }), + ).success, + ).toBe(false); + }); + + it('rejects non-integer', () => { + expect( + PerfOperationRecordSchema.safeParse( + operationRecord({ session_operation_index: 1.5 }), + ).success, + ).toBe(false); + }); + + it('accepts zero', () => { + expect( + PerfOperationRecordSchema.safeParse( + operationRecord({ session_operation_index: 0 }), + ).success, + ).toBe(true); + }); +}); + +// --------------------------------------------------------------------------- +// Bytes/memory/uptime — finite, non-negative +// --------------------------------------------------------------------------- + +describe('schema boundary — bytes/memory/uptime are finite non-negative', () => { + const byteFields = [ + 'stdout_bytes', + 'rss_bytes', + 'heap_used_bytes', + 'external_bytes', + 'array_buffers_bytes', + 'uptime_ms', + ] as const; + + for (const field of byteFields) { + const base = + field === 'stdout_bytes' + ? operationRecord() + : operationRecord({ + rss_bytes: 120_000_000, + heap_used_bytes: 60_000_000, + external_bytes: 25_000_000, + array_buffers_bytes: 1_500_000, + }); + + it(`rejects negative ${field}`, () => { + expect( + PerfOperationRecordSchema.safeParse({ + ...base, + [field]: -1, + }).success, + ).toBe(false); + }); + + it(`rejects Infinity for ${field}`, () => { + expect( + PerfOperationRecordSchema.safeParse({ + ...base, + [field]: Number.POSITIVE_INFINITY, + }).success, + ).toBe(false); + }); + + it(`rejects NaN for ${field}`, () => { + expect( + PerfOperationRecordSchema.safeParse({ + ...base, + [field]: Number.NaN, + }).success, + ).toBe(false); + }); + + it(`accepts zero for ${field}`, () => { + expect( + PerfOperationRecordSchema.safeParse({ + ...base, + [field]: 0, + }).success, + ).toBe(true); + }); + } +}); + +// --------------------------------------------------------------------------- +// Memory sample — sample ages finite non-negative +// --------------------------------------------------------------------------- + +describe('schema boundary — memory sample fields finite non-negative', () => { + const sampleFields = [ + 'rss_bytes', + 'heap_used_bytes', + 'external_bytes', + 'array_buffers_bytes', + 'uptime_ms', + 'ms_since_last_operation', + ] as const; + + for (const field of sampleFields) { + it(`rejects negative ${field}`, () => { + expect( + PerfMemorySampleRecordSchema.safeParse( + memorySampleRecord({ [field]: -1 }), + ).success, + ).toBe(false); + }); + + it(`rejects Infinity for ${field}`, () => { + expect( + PerfMemorySampleRecordSchema.safeParse( + memorySampleRecord({ [field]: Number.POSITIVE_INFINITY }), + ).success, + ).toBe(false); + }); + + it(`rejects NaN for ${field}`, () => { + expect( + PerfMemorySampleRecordSchema.safeParse( + memorySampleRecord({ [field]: Number.NaN }), + ).success, + ).toBe(false); + }); + } +}); + +// --------------------------------------------------------------------------- +// Unknown-field tolerance preserved +// --------------------------------------------------------------------------- + +describe('schema boundary — unknown-field tolerance preserved', () => { + it('strips unknown fields on operation records', () => { + const result = PerfOperationRecordSchema.safeParse( + operationRecord({ future_metric_ms: 42, unknown_col: 'x' }), + ); + expect(result.success).toBe(true); + if (!result.success) throw new Error('expected success'); + expect('future_metric_ms' in result.data).toBe(false); + expect('unknown_col' in result.data).toBe(false); + }); + + it('strips unknown fields on memory sample records', () => { + const result = PerfMemorySampleRecordSchema.safeParse( + memorySampleRecord({ extra_field: true }), + ); + expect(result.success).toBe(true); + if (!result.success) throw new Error('expected success'); + expect('extra_field' in result.data).toBe(false); + }); +}); diff --git a/packages/telemetry/src/perf/perfSelfHealth.behavior.test.ts b/packages/telemetry/src/perf/perfSelfHealth.behavior.test.ts new file mode 100644 index 0000000000..c07e4d0d62 --- /dev/null +++ b/packages/telemetry/src/perf/perfSelfHealth.behavior.test.ts @@ -0,0 +1,217 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import { promises as fs } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { PerfSink, FaultInjectingPerfFilesystem } from './PerfSink.js'; +import { + PerfRetention, + FaultInjectingRetentionFilesystem, +} from './retention.js'; +import type { PerfOperationRecord } from './perfRecords.js'; +import { + PERF_SCHEMA_VERSION, + PERF_RECORD_TYPE_OPERATION, +} from './perfRecords.js'; + +function makeOperation( + overrides: Partial = {}, +): PerfOperationRecord { + return { + schema_version: PERF_SCHEMA_VERSION, + record_type: PERF_RECORD_TYPE_OPERATION, + ts: '2026-01-01T00:00:00.000Z', + session_id: 'sess-1', + operation_id: 'op-1', + runtime_id: 'rt-1', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'hash-1', + llxprt_version: '0.10.0', + git_sha: 'abc1234', + runtime: 'cli', + platform: 'darwin', + provider: 'test-provider', + model: 'test-model', + context_tokens: 1000, + output_tokens: 500, + terminal_cols: 80, + terminal_rows: 24, + render_mode: 'ink', + concurrent_instances: 1, + status: 'completed', + client_prepare_ms: 10, + stream_handler_ms: 100, + ink_render_ms: 5, + ink_render_count: 3, + stdout_bytes: 1024, + stdout_write_calls: 5, + stdout_write_sync_ms: 2, + client_finalize_ms: 8, + provider_attempts: 1, + provider_attempt_sum_ms: 200, + provider_union_ms: 200, + tool_calls: 2, + tool_call_sum_ms: 50, + tool_union_ms: 50, + agent_activity_union_ms: 250, + operation_elapsed_ms: 1000, + approval_wait_ms: 0, + unclassified_elapsed_ms: 0, + session_operation_index: 0, + uptime_ms: 5000, + ...overrides, + }; +} + +async function makeTempDir(): Promise { + const dir = join( + tmpdir(), + `perf-health-${Date.now()}-${Math.random().toString(36).slice(2)}`, + ); + await fs.mkdir(dir, { recursive: true }); + return dir; +} + +describe('PerfSink self-health: lastWriteErrorCode (P11)', () => { + let dir: string; + + beforeEach(async () => { + dir = await makeTempDir(); + }); + + afterEach(async () => { + await fs.rm(dir, { recursive: true, force: true }); + }); + + it('lastWriteErrorCode is null on clean sink', () => { + const sink = new PerfSink({ dir, runUuid: 'old' }); + expect(sink.lastWriteErrorCode).toBeNull(); + }); + + it('lastWriteErrorCode surfaces filesystem write errors', async () => { + const sink = new PerfSink({ + dir, + runUuid: 'old', + fs: new FaultInjectingPerfFilesystem({ + failMethod: 'appendFile', + code: 'ENOSPC', + }), + onDiagnostic: () => {}, + }); + + await sink.write(makeOperation()); + + expect(sink.lastWriteErrorCode).toBe('ENOSPC'); + }); + + it('lastWriteErrorCode persists across multiple write errors', async () => { + const sink = new PerfSink({ + dir, + runUuid: 'old', + fs: new FaultInjectingPerfFilesystem({ + failMethod: 'appendFile', + code: 'EACCES', + }), + onDiagnostic: () => {}, + }); + + await sink.write(makeOperation()); + await sink.write(makeOperation({ operation_id: 'op-2' })); + + expect(sink.lastWriteErrorCode).toBe('EACCES'); + }); + + it('does not add records_dropped counter', () => { + const sink = new PerfSink({ dir, runUuid: 'old' }); + + expect('recordsDropped' in sink).toBe(false); + expect('records_dropped' in sink).toBe(false); + }); +}); + +describe('PerfRetention self-health: evictionCount (P11)', () => { + let dir: string; + + beforeEach(async () => { + dir = await makeTempDir(); + }); + + afterEach(async () => { + await fs.rm(dir, { recursive: true, force: true }); + }); + + it('evictionCount is 0 on clean retention', () => { + const retention = new PerfRetention({ + dir, + runUuid: 'old', + }); + expect(retention.evictionCount).toBe(0); + }); + + it('evictionCount increments on successful eviction', async () => { + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-0000000000cc', + maxFiles: 1, + }); + + await retention.start(); + + const oldName = 'perf-20250101-old.jsonl'; + await fs.writeFile(join(dir, oldName), '{"v":1}\n'); + + const oldTime = new Date(Date.now() - 10 * 24 * 60 * 60 * 1000); + await fs.utimes(join(dir, oldName), oldTime, oldTime); + + await retention.maintain(Date.now()); + + expect(retention.evictionCount).toBeGreaterThan(0); + + await retention.dispose(); + }); + + it('evictionCount does not increment on failed unlinks (fail open)', async () => { + const diagnostics: string[] = []; + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-0000000000cd', + fs: new FaultInjectingRetentionFilesystem({ + failMethod: 'unlink', + code: 'EACCES', + }), + maxFiles: 1, + maxBytes: 1, + onDiagnostic: (message) => diagnostics.push(message), + }); + + const oldName = 'perf-20250101-00000000-0000-4000-8000-0000000000ef.jsonl'; + await fs.writeFile(join(dir, oldName), '{"v":1}\n'); + const oldTime = new Date(Date.now() - 10 * 24 * 60 * 60 * 1000); + await fs.utimes(join(dir, oldName), oldTime, oldTime); + + const beforeCount = retention.evictionCount; + await retention.maintain(Date.now()); + + expect(retention.evictionCount).toBe(beforeCount); + expect(diagnostics).toHaveLength(1); + expect(diagnostics[0]).toContain('EACCES'); + expect((await fs.stat(join(dir, oldName))).isFile()).toBe(true); + + await retention.dispose(); + }); + + it('does not add records_dropped counter', () => { + const retention = new PerfRetention({ + dir, + runUuid: 'old', + }); + expect('recordsDropped' in retention).toBe(false); + expect('records_dropped' in retention).toBe(false); + }); +}); diff --git a/packages/telemetry/src/perf/perfSelfHealth.model.behavior.test.ts b/packages/telemetry/src/perf/perfSelfHealth.model.behavior.test.ts new file mode 100644 index 0000000000..16fb2e5fc3 --- /dev/null +++ b/packages/telemetry/src/perf/perfSelfHealth.model.behavior.test.ts @@ -0,0 +1,258 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Behavioral tests proving the live self-health report model distinguishes + * process-local health facts that are UNAVAILABLE (not supplied) from known + * null/zero values (issue #3167 review finding C). + * + * Reader health (skipped/truncated) always derives from consumer counts. + * Process-local health (lastWriteErrorCode/evictionCount) is unavailable + * (undefined) when not supplied, distinguishing: + * - undefined = process-local health not wired (CLI default-off / batch read) + * - null = known: last write succeeded (no error) + * - '' = known: last write failed with this errno code + * - 0 = known: zero evictions + * + * Uses real buildReport/assembleReport/formatReport with real temp files. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import { promises as fs } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { buildReport, assembleReport, formatReport } from './perfReport.js'; +import type { PerfOperationRecord } from './perfRecords.js'; + +function makeOperation( + overrides: Partial = {}, +): PerfOperationRecord { + return { + schema_version: 1, + record_type: 'operation', + ts: '2026-01-01T00:00:00.000Z', + session_id: 'sess-1', + operation_id: 'op-1', + runtime_id: 'rt-1', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'hash-1', + llxprt_version: '0.10.0', + git_sha: 'abc1234', + runtime: 'cli', + platform: 'darwin', + provider: 'test-provider', + model: 'test-model', + context_tokens: 1000, + output_tokens: 500, + terminal_cols: 80, + terminal_rows: 24, + render_mode: 'ink', + concurrent_instances: 1, + status: 'completed', + client_prepare_ms: 10, + stream_handler_ms: 100, + ink_render_ms: 5, + ink_render_count: 3, + stdout_bytes: 1024, + stdout_write_calls: 5, + stdout_write_sync_ms: 2, + client_finalize_ms: 8, + provider_attempts: 1, + provider_attempt_sum_ms: 200, + provider_union_ms: 200, + tool_calls: 2, + tool_call_sum_ms: 50, + tool_union_ms: 50, + agent_activity_union_ms: 250, + operation_elapsed_ms: 1000, + approval_wait_ms: 0, + unclassified_elapsed_ms: 0, + session_operation_index: 0, + uptime_ms: 5000, + ...overrides, + }; +} + +async function makeTempDir(): Promise { + const dir = join( + tmpdir(), + `perf-health-model-${Date.now()}-${Math.random().toString(36).slice(2)}`, + ); + await fs.mkdir(dir, { recursive: true }); + return dir; +} + +async function writeJsonl( + dir: string, + name: string, + lines: string[], +): Promise { + const content = lines.join('\n'); + await fs.writeFile( + join(dir, name), + content + (content.endsWith('\n') ? '' : '\n'), + 'utf8', + ); +} + +describe('Self-health report model (finding C)', () => { + let dir: string; + + beforeEach(async () => { + dir = await makeTempDir(); + }); + + afterEach(async () => { + await fs.rm(dir, { recursive: true, force: true }); + }); + + // --- Process-local health is UNAVAILABLE when not supplied --- + + it('lastWriteErrorCode is undefined (unavailable) when not supplied', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + ]); + const report = await buildReport(dir); + expect(report.selfHealth.lastWriteErrorCode).toBeUndefined(); + }); + + it('evictionCount is undefined (unavailable) when not supplied', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + ]); + const report = await buildReport(dir); + expect(report.selfHealth.evictionCount).toBeUndefined(); + }); + + // --- Known null/zero are distinguished from unavailable --- + + it('supplied null lastWriteErrorCode is preserved as known-no-error', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + ]); + const report = await buildReport(dir, undefined, { + lastWriteErrorCode: null, + }); + expect(report.selfHealth.lastWriteErrorCode).toBeNull(); + expect(report.selfHealth.lastWriteErrorCode).not.toBeUndefined(); + }); + + it('supplied zero evictionCount is preserved as known-zero', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + ]); + const report = await buildReport(dir, undefined, { evictionCount: 0 }); + expect(report.selfHealth.evictionCount).toBe(0); + expect(report.selfHealth.evictionCount).not.toBeUndefined(); + }); + + it('supplied error code and nonzero evictionCount are preserved', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + ]); + const report = await buildReport(dir, undefined, { + lastWriteErrorCode: 'ENOSPC', + evictionCount: 3, + }); + expect(report.selfHealth.lastWriteErrorCode).toBe('ENOSPC'); + expect(report.selfHealth.evictionCount).toBe(3); + }); + + // --- Reader health always derives from consumer counts --- + + it('skipped and truncated always derive from consumer counts', async () => { + await writeJsonl(dir, 'perf-20260101-mixed.jsonl', [ + JSON.stringify(makeOperation()), + 'broken json', + ]); + const report = await buildReport(dir); + expect(report.selfHealth.skipped).toBe(1); // 1 malformed + expect(report.selfHealth.truncated).toBe(0); + }); + + it('supplied skipped/truncated override consumer-derived defaults', async () => { + await writeJsonl(dir, 'perf-20260101-mixed.jsonl', [ + JSON.stringify(makeOperation()), + 'broken json', + ]); + const report = await buildReport(dir, undefined, { + skipped: 99, + truncated: 7, + }); + expect(report.selfHealth.skipped).toBe(99); + expect(report.selfHealth.truncated).toBe(7); + }); + + // --- Formatter reflects the three-state model --- + + it('formatter shows unavailable for last write error when not supplied', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + ]); + const report = await buildReport(dir); + const text = formatReport(report); + expect(text).toContain('last write error: unavailable'); + }); + + it('formatter shows unavailable for evictions when not supplied', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + ]); + const report = await buildReport(dir); + const text = formatReport(report); + expect(text).toContain('evictions: unavailable'); + }); + + it('formatter shows none for known-null last write error', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + ]); + const report = await buildReport(dir, undefined, { + lastWriteErrorCode: null, + }); + const text = formatReport(report); + expect(text).toContain('last write error: none'); + }); + + it('formatter shows the count for known evictionCount', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + ]); + const report = await buildReport(dir, undefined, { evictionCount: 5 }); + const text = formatReport(report); + expect(text).toContain('evictions: 5'); + }); + + it('formatter shows the code for known lastWriteErrorCode', async () => { + await writeJsonl(dir, 'perf-20260101-run1.jsonl', [ + JSON.stringify(makeOperation()), + ]); + const report = await buildReport(dir, undefined, { + lastWriteErrorCode: 'EACCES', + }); + const text = formatReport(report); + expect(text).toContain('last write error: EACCES'); + }); + + // --- assembleReport also reflects the model --- + + it('assembleReport: undefined process-local health when selfHealth omitted', () => { + const counts = { + parsed: 0, + malformed: 0, + futureVersion: 0, + unversioned: 0, + truncated: 0, + blank: 0, + files: 0, + bytes: 0, + }; + const report = assembleReport([], new Map(), counts, undefined); + expect(report.selfHealth.lastWriteErrorCode).toBeUndefined(); + expect(report.selfHealth.evictionCount).toBeUndefined(); + }); +}); diff --git a/packages/telemetry/src/perf/perfSink.failopen.behavior.test.ts b/packages/telemetry/src/perf/perfSink.failopen.behavior.test.ts new file mode 100644 index 0000000000..1019b1c9d4 --- /dev/null +++ b/packages/telemetry/src/perf/perfSink.failopen.behavior.test.ts @@ -0,0 +1,297 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Behavioral tests for PerfSink fail-open + rate-limited diagnostics under + * EACCES/EROFS/ENOSPC (P04B, EVIDENCE-AC8, D6). + * + * The filesystem port injects deterministic errno failures at the append + * boundary. The caller must remain unaffected (no throw escapes to the + * operation path), diagnostics are rate-limited, and a failure in one write + * must not poison later writes forever. + * + * No real-disk fill, no chmod. Narrow package-private fault injection only. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { + PerfSink, + FaultInjectingPerfFilesystem, + type PerfSinkFilesystem, +} from './PerfSink.js'; + +// --------------------------------------------------------------------------- +// Temp-dir helper +// --------------------------------------------------------------------------- + +let dir: string; + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'perf-failopen-')); +}); + +afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); +}); + +// --------------------------------------------------------------------------- +// Valid record factory +// --------------------------------------------------------------------------- + +function operationRecord( + overrides: Record = {}, +): Record { + return { + schema_version: 1, + record_type: 'operation', + ts: '2026-08-08T12:00:00.000Z', + session_id: 'sess-abc', + operation_id: 'sess-abc#agentic-loop#f7e2', + runtime_id: 'rt-main', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'sha256:project-hash', + llxprt_version: '0.11.0', + git_sha: 'abc1234', + runtime: 'bun-1.3.14', + platform: 'darwin-arm64', + provider: 'openai', + model: 'gpt-4o', + context_tokens: 1000, + output_tokens: 500, + terminal_cols: 120, + terminal_rows: 40, + render_mode: 'incremental', + concurrent_instances: 1, + status: 'completed', + client_prepare_ms: 5, + stream_handler_ms: 10, + ink_render_ms: 20, + ink_render_count: 3, + stdout_bytes: 4096, + stdout_write_calls: 3, + stdout_write_sync_ms: 2, + client_finalize_ms: 1, + provider_attempts: 1, + provider_attempt_sum_ms: 800, + provider_union_ms: 800, + tool_calls: 2, + tool_call_sum_ms: 300, + tool_union_ms: 280, + agent_activity_union_ms: 1000, + operation_elapsed_ms: 1200, + approval_wait_ms: 0, + unclassified_elapsed_ms: 100, + session_operation_index: 1, + uptime_ms: 50000, + ...overrides, + }; +} + +// --------------------------------------------------------------------------- +// Caller remains unaffected under EACCES/EROFS/ENOSPC on appendFile +// --------------------------------------------------------------------------- + +describe('PerfSink fail-open — caller unaffected (AC-8, D6)', () => { + for (const code of ['EACCES', 'EROFS', 'ENOSPC'] as const) { + it(`write does not reject on ${code} from appendFile`, async () => { + const faultFs = new FaultInjectingPerfFilesystem({ + failMethod: 'appendFile', + code, + }); + + const sink = new PerfSink({ + dir, + runUuid: `fail-${code}`, + fs: faultFs, + onDiagnostic: () => {}, + }); + + // write should resolve (fail-open), not reject. + await expect(sink.write(operationRecord())).resolves.toBeUndefined(); + await sink.dispose(); + }); + + it(`write does not reject on ${code} from openExclusive`, async () => { + const faultFs = new FaultInjectingPerfFilesystem({ + failMethod: 'openExclusive', + code, + }); + + const sink = new PerfSink({ + dir, + runUuid: `open-${code}`, + fs: faultFs, + onDiagnostic: () => {}, + }); + + await expect(sink.write(operationRecord())).resolves.toBeUndefined(); + await sink.dispose(); + }); + } +}); + +// --------------------------------------------------------------------------- +// Rate-limited diagnostics +// --------------------------------------------------------------------------- + +describe('PerfSink diagnostics are rate-limited (AC-8)', () => { + it('emits at most one diagnostic per rate-limit window', async () => { + const diagnostics: string[] = []; + const faultFs = new FaultInjectingPerfFilesystem({ + failMethod: 'appendFile', + code: 'EACCES', + }); + + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000000', + fs: faultFs, + diagRateLimitMs: 60_000, + onDiagnostic: (msg) => diagnostics.push(msg), + }); + + // Three failing writes within one rate-limit window. + await sink.write(operationRecord({ session_operation_index: 0 })); + await sink.write(operationRecord({ session_operation_index: 1 })); + await sink.write(operationRecord({ session_operation_index: 2 })); + await sink.dispose(); + + expect(diagnostics.length).toBe(1); + }); + + it('a zero-length rate-limit window never suppresses diagnostics', async () => { + const diagnostics: string[] = []; + const faultFs = new FaultInjectingPerfFilesystem({ + failMethod: 'appendFile', + code: 'ENOSPC', + }); + + // Use a zero-length window so every failure emits. + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000001', + fs: faultFs, + diagRateLimitMs: 0, + onDiagnostic: (msg) => diagnostics.push(msg), + }); + + await sink.write(operationRecord({ session_operation_index: 0 })); + await sink.write(operationRecord({ session_operation_index: 1 })); + await sink.dispose(); + + expect(diagnostics.length).toBe(2); + }); +}); + +// --------------------------------------------------------------------------- +// A filesystem failure in one write does not poison later writes +// --------------------------------------------------------------------------- + +describe('PerfSink failure recovery (AC-8)', () => { + it('a failed write does not poison later writes when the filesystem recovers', async () => { + let shouldFail = true; + + const recoveringFs: PerfSinkFilesystem = { + async ensureDir(d: string): Promise { + try { + await fs.promises.access(d); + } catch { + await fs.promises.mkdir(d, { recursive: true, mode: 0o700 }); + } + }, + async openExclusive(p: string, mode: number): Promise { + if (shouldFail) { + throw Object.assign(new Error('EACCES'), { code: 'EACCES' }); + } + const fd = await fs.promises.open(p, 'wx', mode); + await fd.close(); + }, + async appendFile(p: string, data: string, mode: number): Promise { + if (shouldFail) { + throw Object.assign(new Error('EACCES'), { code: 'EACCES' }); + } + await fs.promises.appendFile(p, data, { encoding: 'utf8', mode }); + }, + }; + + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000002', + fs: recoveringFs, + diagRateLimitMs: 0, + onDiagnostic: () => {}, + }); + + // First write fails (filesystem error, fail-open). + await sink.write(operationRecord({ session_operation_index: 0 })); + + // Filesystem recovers. + shouldFail = false; + + // Second write succeeds. + await sink.write(operationRecord({ session_operation_index: 1 })); + await sink.dispose(); + + // A file should exist with one record (the second write). + const files = fs.readdirSync(dir); + expect(files).toHaveLength(1); + const content = fs.readFileSync(path.join(dir, files[0]), 'utf8'); + const lines = content.trim().split('\n'); + expect(lines).toHaveLength(1); + const parsed = JSON.parse(lines[0]); + expect(parsed.session_operation_index).toBe(1); + }); + + it('failed exclusive open does not advance day/file state', async () => { + const diagnostics: string[] = []; + let openAttempts = 0; + + const stateCheckingFs: PerfSinkFilesystem = { + async ensureDir(d: string): Promise { + try { + await fs.promises.access(d); + } catch { + await fs.promises.mkdir(d, { recursive: true, mode: 0o700 }); + } + }, + async openExclusive(p: string, mode: number): Promise { + openAttempts++; + if (openAttempts === 1) { + throw Object.assign(new Error('EACCES'), { code: 'EACCES' }); + } + const fd = await fs.promises.open(p, 'wx', mode); + await fd.close(); + }, + async appendFile(p: string, data: string, mode: number): Promise { + await fs.promises.appendFile(p, data, { encoding: 'utf8', mode }); + }, + }; + + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000003', + fs: stateCheckingFs, + diagRateLimitMs: 0, + onDiagnostic: (msg) => diagnostics.push(msg), + }); + + // First write: openExclusive fails, state must NOT advance. + await sink.write(operationRecord()); + expect(diagnostics.length).toBe(1); + + // Second write: openExclusive succeeds, state advances, append succeeds. + await sink.write(operationRecord()); + await sink.dispose(); + + expect(openAttempts).toBe(2); + const files = fs.readdirSync(dir); + expect(files).toHaveLength(1); + }); +}); diff --git a/packages/telemetry/src/perf/perfSink.retention.behavior.test.ts b/packages/telemetry/src/perf/perfSink.retention.behavior.test.ts new file mode 100644 index 0000000000..7f3c5cf841 --- /dev/null +++ b/packages/telemetry/src/perf/perfSink.retention.behavior.test.ts @@ -0,0 +1,415 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Real-file behavioral tests for PerfSink + PerfRetention integration (P08). + * + * Verifies that maintenance is wired narrowly: + * - Roll boundary triggers maybeMaintain. + * - Disposal drains writes, stops maintenance, and removes the claim. + * - Empty started sink creates ONLY its claim (no perf JSONL). + * - Concurrent appends produce documented overshoot, then next sweep converges. + * + * No mocks. Real files, real filesystem. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { PerfSink } from './PerfSink.js'; +import { PerfRetention } from './retention.js'; + +let dir: string; + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'perf-sink-retention-')); +}); + +afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); +}); + +function operationRecord( + overrides: Record = {}, +): Record { + return { + schema_version: 1, + record_type: 'operation', + ts: '2026-08-08T12:00:00.000Z', + session_id: 'sess-abc', + operation_id: 'sess-abc#agentic-loop#f7e2', + runtime_id: 'rt-main', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'sha256:project-hash', + llxprt_version: '0.11.0', + git_sha: 'abc1234', + runtime: 'bun-1.3.14', + platform: 'darwin-arm64', + provider: 'openai', + model: 'gpt-4o', + context_tokens: 1000, + output_tokens: 500, + terminal_cols: 120, + terminal_rows: 40, + render_mode: 'incremental', + concurrent_instances: 1, + status: 'completed', + client_prepare_ms: 5, + stream_handler_ms: 10, + ink_render_ms: 20, + ink_render_count: 3, + stdout_bytes: 4096, + stdout_write_calls: 3, + stdout_write_sync_ms: 2, + client_finalize_ms: 1, + provider_attempts: 1, + provider_attempt_sum_ms: 800, + provider_union_ms: 800, + tool_calls: 2, + tool_call_sum_ms: 300, + tool_union_ms: 280, + agent_activity_union_ms: 1000, + operation_elapsed_ms: 1200, + approval_wait_ms: 0, + unclassified_elapsed_ms: 100, + session_operation_index: 1, + uptime_ms: 50000, + ...overrides, + }; +} + +// --------------------------------------------------------------------------- +// start() creates only the claim — no perf JSONL +// --------------------------------------------------------------------------- + +describe('PerfSink + retention — start creates only claim (AC-7)', () => { + it('an empty started sink creates only its claim, no perf JSONL', async () => { + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000000', + }); + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000000', + retention, + }); + await sink.start(); + + const files = fs.readdirSync(dir).sort(); + expect(files).toEqual(['00000000-0000-4000-8000-000000000000.claim']); + await sink.dispose(); + }); + + it('start then write creates the claim AND the perf JSONL', async () => { + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000001', + }); + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000001', + retention, + }); + await sink.start(); + await sink.write(operationRecord()); + + const files = fs.readdirSync(dir).sort(); + expect(files).toContain('00000000-0000-4000-8000-000000000001.claim'); + expect(files).toContain( + 'perf-20260808-00000000-0000-4000-8000-000000000001.jsonl', + ); + await sink.dispose(); + }); +}); + +// --------------------------------------------------------------------------- +// Roll boundary triggers maintenance +// --------------------------------------------------------------------------- + +describe('PerfSink + retention — roll boundary triggers maintenance (AC-7)', () => { + it('a midnight roll triggers maybeMaintain which evicts old files', async () => { + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000002', + maxFiles: 2, + maxBytes: 10_000_000, + }); + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000002', + retention, + }); + await sink.start(); + + // Create old files BEFORE the first perf write. The first write's roll + // triggers maybeMaintain (lastMaintenanceMs starts at 0, so the default + // interval always admits the first call), which evicts them. These belong + // to a DIFFERENT prior run (not the owner) so they are eligible. + fs.writeFileSync( + path.join( + dir, + 'perf-20260101-00000000-0000-4000-8000-000000000092.jsonl', + ), + 'data\n', + ); + fs.writeFileSync( + path.join( + dir, + 'perf-20260102-00000000-0000-4000-8000-000000000092.jsonl', + ), + 'data\n', + ); + fs.writeFileSync( + path.join( + dir, + 'perf-20260103-00000000-0000-4000-8000-000000000092.jsonl', + ), + 'data\n', + ); + + // Write a record — the roll boundary triggers maintenance. + await sink.write(operationRecord({ ts: '2026-08-08T12:00:00.000Z' })); + await sink.dispose(); + + const files = fs.readdirSync(dir).sort(); + // Old files should have been evicted (maxFiles: 2, first maybeMaintain + // always runs because lastMaintenanceMs initializes to 0). + expect(files).not.toContain( + 'perf-20260101-00000000-0000-4000-8000-000000000092.jsonl', + ); + expect(files).not.toContain( + 'perf-20260102-00000000-0000-4000-8000-000000000092.jsonl', + ); + expect(files).not.toContain( + 'perf-20260103-00000000-0000-4000-8000-000000000092.jsonl', + ); + }); +}); + +// --------------------------------------------------------------------------- +// Disposal drains writes, stops maintenance, removes claim +// --------------------------------------------------------------------------- + +describe('PerfSink + retention — disposal (AC-7)', () => { + it('dispose drains writes, then stops maintenance and removes claim', async () => { + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000003', + }); + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000003', + retention, + }); + await sink.start(); + + // Queue a write. + await sink.write(operationRecord()); + + // Dispose should drain + remove claim. + await sink.dispose(); + + const files = fs.readdirSync(dir).sort(); + // Claim should be gone; perf file should remain. + expect(files).not.toContain('00000000-0000-4000-8000-000000000003.claim'); + expect(files).toContain( + 'perf-20260808-00000000-0000-4000-8000-000000000003.jsonl', + ); + }); + + it('dispose after start with no writes leaves no files', async () => { + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000004', + }); + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000004', + retention, + }); + await sink.start(); + await sink.dispose(); + + expect(fs.readdirSync(dir)).toHaveLength(0); + }); +}); + +// --------------------------------------------------------------------------- +// Backward compatibility: PerfSink without retention works as before +// --------------------------------------------------------------------------- + +describe('PerfSink without retention — backward compatible', () => { + it('write + dispose work without retention (no start needed)', async () => { + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000005', + }); + await sink.write(operationRecord()); + await sink.dispose(); + + const files = fs.readdirSync(dir); + expect(files).toEqual([ + 'perf-20260808-00000000-0000-4000-8000-000000000005.jsonl', + ]); + // No claim file. + expect(files).not.toContain('00000000-0000-4000-8000-000000000005.claim'); + }); + + it('start is a no-op without retention', async () => { + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000006', + }); + await sink.start(); + expect(fs.readdirSync(dir)).toHaveLength(0); + await sink.dispose(); + }); +}); + +// --------------------------------------------------------------------------- +// Manual sweep convergence after exceeding retention caps +// --------------------------------------------------------------------------- + +describe('PerfSink + retention — manual sweep convergence (AC-7)', () => { + it('a manual sweep converges after sequential setup exceeds the cap', async () => { + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000007', + maxFiles: 2, + maxBytes: 10_000_000, + }); + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000007', + retention, + }); + await sink.start(); + + // Write a record (creates today's file). + await sink.write(operationRecord()); + + // Create many old files to exceed the cap. + for (let i = 0; i < 10; i++) { + const dayKey = sequentialDayKey(i); + fs.writeFileSync(path.join(dir, `perf-${dayKey}-old.jsonl`), 'data\n'); + const oldTime = Date.now() - (10 - i) * 3_600_000; + fs.utimesSync( + path.join(dir, `perf-${dayKey}-old.jsonl`), + new Date(oldTime), + new Date(oldTime), + ); + } + + // Manual sweep converges. + await retention.maintain(Date.now()); + + const jsonlFiles = fs.readdirSync(dir).filter((f) => f.endsWith('.jsonl')); + + // The temporary over-cap state converges after the sweep. + expect(jsonlFiles.length).toBeLessThanOrEqual(2); + + await sink.dispose(); + }); +}); + +// --------------------------------------------------------------------------- +// countNonStaleClaims available via retention +// --------------------------------------------------------------------------- + +describe('PerfSink + retention — countNonStaleClaims', () => { + it('retention.countNonStaleClaims reflects the current claim', async () => { + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000008', + }); + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000008', + retention, + }); + await sink.start(); + + const now = Date.now(); + const count = await retention.countNonStaleClaims(now); + expect(count).toBe(1); // this run's claim + + await sink.dispose(); + }); +}); + +// --------------------------------------------------------------------------- +// Dispose always runs retention cleanup even when writeChain rejects (D8) +// --------------------------------------------------------------------------- + +describe('PerfSink.dispose — always drains writeChain AND retention cleanup', () => { + it('retention claim is removed even when an internal write error rejects the chain', async () => { + // A filesystem port whose appendFile throws a NON-errno error (no .code). + // This is an internal/programming error that rethrows through write(), + // rejecting the writeChain. dispose() must still run retention.dispose() + // (removing the claim and stopping the maintenance timer). + const internalErrorFs: { + ensureDir(d: string): Promise; + openExclusive(p: string, mode: number): Promise; + appendFile(p: string, data: string, mode: number): Promise; + } = { + async ensureDir(d: string): Promise { + try { + await fs.promises.access(d); + } catch { + await fs.promises.mkdir(d, { recursive: true, mode: 0o700 }); + } + }, + async openExclusive(p: string, mode: number): Promise { + const fd = await fs.promises.open(p, 'wx', mode); + await fd.close(); + }, + async appendFile(): Promise { + throw new Error('internal serialization bug'); + }, + }; + + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000009', + }); + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000009', + retention, + fs: internalErrorFs, + onDiagnostic: () => {}, + }); + await sink.start(); + + expect( + fs.existsSync( + path.join(dir, '00000000-0000-4000-8000-000000000009.claim'), + ), + ).toBe(true); + + const writePromise = sink.write(operationRecord()); + await expect(writePromise).rejects.toThrow('internal serialization bug'); + + await expect(sink.dispose()).rejects.toThrow('internal serialization bug'); + + expect( + fs.existsSync( + path.join(dir, '00000000-0000-4000-8000-000000000009.claim'), + ), + ).toBe(false); + }); +}); + +function sequentialDayKey(index: number): string { + const base = new Date(Date.UTC(2025, 0, 1)); + base.setUTCDate(base.getUTCDate() + index); + const year = base.getUTCFullYear(); + const month = String(base.getUTCMonth() + 1).padStart(2, '0'); + const day = String(base.getUTCDate()).padStart(2, '0'); + return `${year}${month}${day}`; +} diff --git a/packages/telemetry/src/perf/perfSink.roundtrip.behavior.test.ts b/packages/telemetry/src/perf/perfSink.roundtrip.behavior.test.ts new file mode 100644 index 0000000000..51e0a1b0c4 --- /dev/null +++ b/packages/telemetry/src/perf/perfSink.roundtrip.behavior.test.ts @@ -0,0 +1,468 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Real-file behavioral tests for PerfSink (P04B, EVIDENCE-AC1). + * + * PerfSink is a constructible, non-singleton writer that uses a serialized + * no-drop promise chain. One exclusive-created 0600 file per run UUID per UTC + * record day: perf-YYYYMMDD-runUuid.jsonl. Empty sink creates no file. Drain + * on dispose. + * + * Tests use REAL files and the REAL reader round-trip — no mocks. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { PerfSink } from './PerfSink.js'; +import { readPerfRecords } from './perfRecords.js'; + +// --------------------------------------------------------------------------- +// Temp-dir helper +// --------------------------------------------------------------------------- + +let dir: string; + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'perf-sink-')); +}); + +afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); +}); + +// --------------------------------------------------------------------------- +// Valid record factory (plain object; PerfSink validates via the real schema) +// --------------------------------------------------------------------------- + +function operationRecord( + overrides: Record = {}, +): Record { + return { + schema_version: 1, + record_type: 'operation', + ts: '2026-08-08T12:00:00.000Z', + session_id: 'sess-abc', + operation_id: 'sess-abc#agentic-loop#f7e2', + runtime_id: 'rt-main', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'sha256:project-hash', + llxprt_version: '0.11.0', + git_sha: 'abc1234', + runtime: 'bun-1.3.14', + platform: 'darwin-arm64', + provider: 'openai', + model: 'gpt-4o', + context_tokens: 1000, + output_tokens: 500, + terminal_cols: 120, + terminal_rows: 40, + render_mode: 'incremental', + concurrent_instances: 1, + status: 'completed', + client_prepare_ms: 5, + stream_handler_ms: 10, + ink_render_ms: 20, + ink_render_count: 3, + stdout_bytes: 4096, + stdout_write_calls: 3, + stdout_write_sync_ms: 2, + client_finalize_ms: 1, + provider_attempts: 1, + provider_attempt_sum_ms: 800, + provider_union_ms: 800, + tool_calls: 2, + tool_call_sum_ms: 300, + tool_union_ms: 280, + agent_activity_union_ms: 1000, + operation_elapsed_ms: 1200, + approval_wait_ms: 0, + unclassified_elapsed_ms: 100, + session_operation_index: 1, + uptime_ms: 50000, + ...overrides, + }; +} + +// --------------------------------------------------------------------------- +// Round-trip: writer → filesystem → real reader (EVIDENCE-AC1) +// --------------------------------------------------------------------------- + +describe('PerfSink round-trip (AC-1)', () => { + it('writes a record that round-trips through the real reader with exact field values', async () => { + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000000', + }); + + const source = operationRecord({ + status: 'superseded', + unclassified_elapsed_ms: -42, + tool_union_ms: 280, + external_bytes: 99, + concurrent_instances: 3, + session_operation_index: 7, + }); + + await sink.write(source); + await sink.dispose(); + + const files = fs.readdirSync(dir); + expect(files).toEqual([ + 'perf-20260808-00000000-0000-4000-8000-000000000000.jsonl', + ]); + + const { records, counts } = await readPerfRecords(path.join(dir, files[0])); + expect(counts.parsed).toBe(1); + expect(records).toHaveLength(1); + + const [parsed] = records; + expect(parsed.record_type).toBe('operation'); + if (parsed.record_type !== 'operation') return; + expect(parsed).toMatchObject({ + schema_version: 1, + record_type: 'operation', + ts: '2026-08-08T12:00:00.000Z', + operation_id: 'sess-abc#agentic-loop#f7e2', + status: 'superseded', + unclassified_elapsed_ms: -42, + tool_union_ms: 280, + external_bytes: 99, + concurrent_instances: 3, + session_operation_index: 7, + }); + // D1: no child-id arrays + expect('prompt_ids' in parsed).toBe(false); + expect('turn_ids' in parsed).toBe(false); + }); + + it('writes memory fields when present and round-trips them', async () => { + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000001', + }); + + await sink.write( + operationRecord({ + rss_bytes: 200_000_000, + heap_used_bytes: 100_000_000, + external_bytes: 30_000_000, + array_buffers_bytes: 2_000_000, + }), + ); + await sink.dispose(); + + const files = fs.readdirSync(dir); + expect(files).toHaveLength(1); + + const { records } = await readPerfRecords(path.join(dir, files[0])); + expect(records).toHaveLength(1); + const [parsed] = records; + expect(parsed.record_type).toBe('operation'); + if (parsed.record_type !== 'operation') return; + expect(parsed.rss_bytes).toBe(200_000_000); + expect(parsed.heap_used_bytes).toBe(100_000_000); + }); +}); + +// --------------------------------------------------------------------------- +// Concurrent writes → ordered, untorn lines +// --------------------------------------------------------------------------- + +describe('PerfSink concurrency (AC-1)', () => { + it('N concurrent writes produce N ordered untorn lines', async () => { + const N = 100; + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000002', + }); + + const records = Array.from({ length: N }, (_, i) => + operationRecord({ session_operation_index: i }), + ); + + // Fire all writes without awaiting individually first. + const promises = records.map((r) => sink.write(r)); + await Promise.all(promises); + await sink.dispose(); + + const files = fs.readdirSync(dir); + expect(files).toHaveLength(1); + + const { records: read, counts } = await readPerfRecords( + path.join(dir, files[0]), + ); + expect(counts.parsed).toBe(N); + expect(counts.malformed).toBe(0); + expect(read).toHaveLength(N); + + // Verify order is preserved and no torn lines. + for (let i = 0; i < N; i++) { + const rec = read[i]; + expect(rec.record_type).toBe('operation'); + expect( + (rec as { session_operation_index: number }).session_operation_index, + ).toBe(i); + } + }); +}); + +// --------------------------------------------------------------------------- +// Distinct run UUIDs → distinct files (exclusive create) +// --------------------------------------------------------------------------- + +describe('PerfSink exclusive create (AC-1)', () => { + it('distinct run UUID sinks do not share files', async () => { + const sink1 = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000003', + }); + const sink2 = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000004', + }); + + await sink1.write(operationRecord()); + await sink2.write(operationRecord()); + await sink1.dispose(); + await sink2.dispose(); + + const files = fs.readdirSync(dir).sort(); + expect(files).toEqual([ + 'perf-20260808-00000000-0000-4000-8000-000000000003.jsonl', + 'perf-20260808-00000000-0000-4000-8000-000000000004.jsonl', + ]); + + // Each file has exactly one record. + for (const f of files) { + const { counts } = await readPerfRecords(path.join(dir, f)); + expect(counts.parsed).toBe(1); + } + }); + + it('creates the file with 0600 permissions', async () => { + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000005', + }); + await sink.write(operationRecord()); + await sink.dispose(); + + const files = fs.readdirSync(dir); + const stat = fs.statSync(path.join(dir, files[0])); + // Mask to permission bits only. + const mode = stat.mode & 0o777; + expect(mode).toBe(0o600); + }); +}); + +// --------------------------------------------------------------------------- +// UTC midnight roll +// --------------------------------------------------------------------------- + +describe('PerfSink UTC midnight roll (AC-1)', () => { + it('a record crossing UTC midnight rolls to a second file and both parse', async () => { + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000006', + }); + + await sink.write(operationRecord({ ts: '2026-08-08T23:59:59.999Z' })); + await sink.write(operationRecord({ ts: '2026-08-09T00:00:00.000Z' })); + await sink.dispose(); + + const files = fs.readdirSync(dir).sort(); + expect(files).toEqual([ + 'perf-20260808-00000000-0000-4000-8000-000000000006.jsonl', + 'perf-20260809-00000000-0000-4000-8000-000000000006.jsonl', + ]); + + const result1 = await readPerfRecords(path.join(dir, files[0])); + expect(result1.counts.parsed).toBe(1); + + const result2 = await readPerfRecords(path.join(dir, files[1])); + expect(result2.counts.parsed).toBe(1); + }); + + it('does not roll when consecutive records share the same day', async () => { + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000007', + }); + + await sink.write(operationRecord({ ts: '2026-08-08T01:00:00.000Z' })); + await sink.write(operationRecord({ ts: '2026-08-08T23:00:00.000Z' })); + await sink.dispose(); + + const files = fs.readdirSync(dir); + expect(files).toEqual([ + 'perf-20260808-00000000-0000-4000-8000-000000000007.jsonl', + ]); + }); + + it('a record whose UTC day moves backwards re-adopts the earlier-day file without losing records', async () => { + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-0000000000bd', + }); + + // day 1 → day 2 (forward roll) → day 1 again (backward roll). + await sink.write( + operationRecord({ + ts: '2026-08-08T12:00:00.000Z', + operation_id: 'sess-abc#agentic-loop#d1a', + }), + ); + await sink.write( + operationRecord({ + ts: '2026-08-09T12:00:00.000Z', + operation_id: 'sess-abc#agentic-loop#d2', + }), + ); + await sink.write( + operationRecord({ + ts: '2026-08-08T13:00:00.000Z', + operation_id: 'sess-abc#agentic-loop#d1b', + }), + ); + await sink.dispose(); + + const files = fs.readdirSync(dir).sort(); + expect(files).toEqual([ + 'perf-20260808-00000000-0000-4000-8000-0000000000bd.jsonl', + 'perf-20260809-00000000-0000-4000-8000-0000000000bd.jsonl', + ]); + + // Both day-1 records survive — the backward-day record was not lost. + const day1 = await readPerfRecords(path.join(dir, files[0])); + expect(day1.counts.parsed).toBe(2); + const day2 = await readPerfRecords(path.join(dir, files[1])); + expect(day2.counts.parsed).toBe(1); + }); +}); + +// --------------------------------------------------------------------------- +// Empty sink creates no file +// --------------------------------------------------------------------------- + +describe('PerfSink empty (AC-1)', () => { + it('an empty sink creates no file', async () => { + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000008', + }); + await sink.dispose(); + + const files = fs.readdirSync(dir); + expect(files).toHaveLength(0); + }); + + it('dispose on an empty sink does not throw', async () => { + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-000000000009', + }); + await expect(sink.dispose()).resolves.toBeUndefined(); + }); +}); + +// --------------------------------------------------------------------------- +// Dispose drains accepted writes +// --------------------------------------------------------------------------- + +describe('PerfSink dispose drains (AC-1)', () => { + it('dispose drains all accepted writes before returning', async () => { + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-00000000000a', + }); + + const N = 50; + const promises = Array.from({ length: N }, (_, i) => + sink.write(operationRecord({ session_operation_index: i })), + ); + + // Dispose without awaiting individual writes. + await sink.dispose(); + + // All writes should have been drained. + await Promise.all(promises); + + const files = fs.readdirSync(dir); + expect(files).toHaveLength(1); + const { counts } = await readPerfRecords(path.join(dir, files[0])); + expect(counts.parsed).toBe(N); + }); + + it('write after dispose is a no-op (no new file created)', async () => { + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-00000000000b', + }); + await sink.write(operationRecord()); + await sink.dispose(); + + // Write after dispose should not create new content. + await sink.write(operationRecord({ ts: '2026-08-09T00:00:00.000Z' })); + + const files = fs.readdirSync(dir); + expect(files).toEqual([ + 'perf-20260808-00000000-0000-4000-8000-00000000000b.jsonl', + ]); + const { counts } = await readPerfRecords(path.join(dir, files[0])); + expect(counts.parsed).toBe(1); + }); +}); + +// --------------------------------------------------------------------------- +// Fail-fast: invalid records reject at the schema boundary +// --------------------------------------------------------------------------- + +describe('PerfSink fail-fast for invalid records', () => { + it('rejects a record missing required fields', async () => { + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-00000000000c', + }); + expect(() => + sink.write({ schema_version: 1, record_type: 'operation' }), + ).toThrow('invalid'); + await sink.dispose(); + }); + + it('rejects a record with an invalid field value', async () => { + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-00000000000d', + }); + expect(() => sink.write(operationRecord({ terminal_cols: -1 }))).toThrow( + 'terminal_cols', + ); + await sink.dispose(); + }); + + it('rejects null input', async () => { + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-00000000000e', + }); + expect(() => sink.write(null)).toThrow('Expected'); + await sink.dispose(); + }); + + it('does not create a file when the record is invalid', async () => { + const sink = new PerfSink({ + dir, + runUuid: '00000000-0000-4000-8000-00000000000f', + }); + expect(() => sink.write({ bad: 'record' })).toThrow('invalid'); + await sink.dispose(); + expect(fs.readdirSync(dir)).toHaveLength(0); + }); +}); diff --git a/packages/telemetry/src/perf/perfSlopeBridge.ts b/packages/telemetry/src/perf/perfSlopeBridge.ts new file mode 100644 index 0000000000..66ed08bbd5 --- /dev/null +++ b/packages/telemetry/src/perf/perfSlopeBridge.ts @@ -0,0 +1,150 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Canonical read-time memory slope derivation (P10/P11, AC-10). + * + * This is the SINGLE owner of the generic memory slope algorithm and its + * types. It lives below the CLI layer (in telemetry) so both the longitudinal + * report and the CLI live view (`/perf`) import one shared implementation + * instead of maintaining parallel copies. + * + * Two axes separate legitimate growth (tracks work) from a leak (tracks + * uptime): + * - per-operation: least-squares of each memory column on + * `session_operation_index` using operation records. + * - per-minute: least-squares of each memory column on `uptime_ms` using + * `memory_sample` rows, scaled to bytes/min. + * + * P11 invokes these once per run/file so memory slopes are per run/file, + * never accidentally pooled across process uptimes / session indices. + * + * Robustness: requires >=2 usable points and nonzero x variance; otherwise + * returns null (never NaN/Infinity). Negative slopes are preserved. + */ + +import type { + PerfOperationRecord, + PerfMemorySampleRecord, +} from './perfRecords.js'; + +/** The per-operation slope for each of the four memory values. */ +export interface PerOperationMemorySlope { + readonly rss_bytes_per_operation: number | null; + readonly heap_used_bytes_per_operation: number | null; + readonly external_bytes_per_operation: number | null; + readonly array_buffers_bytes_per_operation: number | null; +} + +/** The per-minute slope for each of the four memory values. */ +export interface PerMinuteMemorySlope { + readonly rss_bytes_per_minute: number | null; + readonly heap_used_bytes_per_minute: number | null; + readonly external_bytes_per_minute: number | null; + readonly array_buffers_bytes_per_minute: number | null; +} + +const MS_PER_MINUTE = 60_000; + +/** + * Ordinary-least-squares slope of y on x. Returns null when fewer than 2 points + * or the x variance is zero. Also returns null for any non-finite result. + */ +function leastSquaresSlope( + points: ReadonlyArray, +): number | null { + if (points.length < 2) return null; + + const n = points.length; + let sumX = 0; + let sumY = 0; + for (const [x, y] of points) { + sumX += x; + sumY += y; + } + const meanX = sumX / n; + const meanY = sumY / n; + + let numerator = 0; + let denominator = 0; + for (const [x, y] of points) { + const dx = x - meanX; + numerator += dx * (y - meanY); + denominator += dx * dx; + } + + if (denominator === 0) return null; + + const slope = numerator / denominator; + if (!Number.isFinite(slope)) return null; + return slope; +} + +function slopeFromOps( + operations: readonly PerfOperationRecord[], + getMem: (op: PerfOperationRecord) => number | undefined, +): number | null { + const points: Array<[number, number]> = []; + for (const op of operations) { + const mem = getMem(op); + if (mem !== undefined) { + points.push([op.session_operation_index, mem]); + } + } + return leastSquaresSlope(points); +} + +/** + * Derives the per-operation memory slope from operation records (P10 parity). + * Only records carrying the specific memory column contribute. Invoked once + * per run/file. + */ +export function derivePerOperationMemorySlope( + operations: readonly PerfOperationRecord[], +): PerOperationMemorySlope { + return { + rss_bytes_per_operation: slopeFromOps(operations, (o) => o.rss_bytes), + heap_used_bytes_per_operation: slopeFromOps( + operations, + (o) => o.heap_used_bytes, + ), + external_bytes_per_operation: slopeFromOps( + operations, + (o) => o.external_bytes, + ), + array_buffers_bytes_per_operation: slopeFromOps( + operations, + (o) => o.array_buffers_bytes, + ), + }; +} + +/** + * Derives the per-minute memory slope from memory_sample records (P10 parity). + * Regression on uptime_ms, scaled to bytes/min. Invoked once per run/file. + */ +export function derivePerMinuteMemorySlope( + samples: readonly PerfMemorySampleRecord[], +): PerMinuteMemorySlope { + const slope = ( + getMem: (s: PerfMemorySampleRecord) => number, + ): number | null => { + const points: Array<[number, number]> = samples.map((s) => [ + s.uptime_ms, + getMem(s), + ]); + const bytesPerMs = leastSquaresSlope(points); + if (bytesPerMs === null) return null; + return bytesPerMs * MS_PER_MINUTE; + }; + + return { + rss_bytes_per_minute: slope((s) => s.rss_bytes), + heap_used_bytes_per_minute: slope((s) => s.heap_used_bytes), + external_bytes_per_minute: slope((s) => s.external_bytes), + array_buffers_bytes_per_minute: slope((s) => s.array_buffers_bytes), + }; +} diff --git a/packages/telemetry/src/perf/retention.capSelection.behavior.test.ts b/packages/telemetry/src/perf/retention.capSelection.behavior.test.ts new file mode 100644 index 0000000000..300658ad85 --- /dev/null +++ b/packages/telemetry/src/perf/retention.capSelection.behavior.test.ts @@ -0,0 +1,133 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Cap-selection behavioral test for PerfRetention (P08, D5). + * + * Under observed single-writer volume, asserts which of (count cap, byte cap) + * binds first. Uses the P04 benchmark values: + * operation record (with memory): 1220 bytes/line + * memory_sample record: 242 bytes/line + * combined per-operation pair: 1462 bytes + * + * The byte cap (64 MiB) binds for high-volume writers (>~359 pairs/day); + * the file cap (128) binds for low-volume writers (<~359 pairs/day). + * At representative single-writer interactive use (~100 ops/day), the file cap + * is the binding constraint at ~128 days. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { PerfRetention, PERF_MAX_BYTES, PERF_MAX_FILES } from './retention.js'; + +let dir: string; + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'perf-cap-')); +}); + +afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); +}); + +describe('PerfRetention cap selection (D5)', () => { + it('at representative single-writer volume (~100 ops/day) the file cap binds', async () => { + const now = Date.now(); + // Simulate MAX_FILES days × 100 operations/day at ~1220 bytes/op (no memory). + // MAX_FILES × 100 × 1220 ≈ 15.6 MiB < 64 MiB. + // So MAX_FILES files is under the byte cap but at the file cap. + for (let i = 0; i < PERF_MAX_FILES; i++) { + const dayKey = makeSequentialDayKey(i); + const content = + `${JSON.stringify({ idx: 0, pad: '.'.repeat(1100) })}\n`.repeat(100); + fs.writeFileSync(path.join(dir, `perf-${dayKey}-w.jsonl`), content); + fs.utimesSync( + path.join(dir, `perf-${dayKey}-w.jsonl`), + new Date(now - (PERF_MAX_FILES - i) * 86_400_000), + new Date(now - (PERF_MAX_FILES - i) * 86_400_000), + ); + } + + // Verify total bytes is under the byte cap. + let totalBytes = 0; + for (const f of fs.readdirSync(dir)) { + totalBytes += fs.statSync(path.join(dir, f)).size; + } + expect(totalBytes).toBeLessThan(PERF_MAX_BYTES); + + // With one more file, the file cap binds. + const extraKey = makeSequentialDayKey(PERF_MAX_FILES); + fs.writeFileSync( + path.join(dir, `perf-${extraKey}-extra.jsonl`), + `${JSON.stringify({ idx: 0 })}\n`, + ); + fs.utimesSync( + path.join(dir, `perf-${extraKey}-extra.jsonl`), + new Date(now), + new Date(now), + ); + + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000000', + onDiagnostic: () => {}, + }); + await retention.maintain(now); + + // File count should be at most MAX_FILES (128). + const remaining = fs.readdirSync(dir).filter((f) => f.endsWith('.jsonl')); + expect(remaining.length).toBeLessThanOrEqual(PERF_MAX_FILES); + }); + + it('at high single-writer volume the byte cap binds before the file cap', async () => { + const now = Date.now(); + // Create fewer than MAX_FILES files but exceeding MAX_BYTES total. + // 8 files × ~8 MiB each = ~64 MiB ≈ MAX_BYTES. + const bytesPerFile = Math.ceil(PERF_MAX_BYTES / 7); // ~9.6 MiB each + for (let i = 0; i < 8; i++) { + const dayKey = makeSequentialDayKey(i); + const padding = '.'.repeat(bytesPerFile); + fs.writeFileSync(path.join(dir, `perf-${dayKey}-h.jsonl`), padding); + fs.utimesSync( + path.join(dir, `perf-${dayKey}-h.jsonl`), + new Date(now - (8 - i) * 86_400_000), + new Date(now - (8 - i) * 86_400_000), + ); + } + + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000001', + onDiagnostic: () => {}, + }); + await retention.maintain(now); + + let remainingBytes = 0; + const remaining = fs.readdirSync(dir).filter((f) => f.endsWith('.jsonl')); + for (const f of remaining) { + remainingBytes += fs.statSync(path.join(dir, f)).size; + } + // The byte cap should be enforced (files count < MAX_FILES proves byte cap bound). + expect(remaining.length).toBeLessThan(PERF_MAX_FILES); + // Total bytes should be under the byte cap. + expect(remainingBytes).toBeLessThanOrEqual(PERF_MAX_BYTES); + }); +}); + +/** + * Generates sequential valid 8-digit YYYYMMDD day keys starting from a base date. + * Index 0 → 20250101, index 1 → 20250102, etc., rolling over months/years as needed. + */ +function makeSequentialDayKey(index: number): string { + const base = new Date(Date.UTC(2025, 0, 1)); + base.setUTCDate(base.getUTCDate() + index); + const year = base.getUTCFullYear(); + const month = String(base.getUTCMonth() + 1).padStart(2, '0'); + const day = String(base.getUTCDate()).padStart(2, '0'); + return `${year}${month}${day}`; +} diff --git a/packages/telemetry/src/perf/retention.claim.behavior.test.ts b/packages/telemetry/src/perf/retention.claim.behavior.test.ts new file mode 100644 index 0000000000..ec5ea6e3c3 --- /dev/null +++ b/packages/telemetry/src/perf/retention.claim.behavior.test.ts @@ -0,0 +1,353 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Claim-mechanism behavioral tests for PerfRetention (D5, AC-7, D3). + * + * Covers the claim file contract split out of the original + * retention.behavior.test.ts: the retention constants (D5), the claim file + * lifecycle (AC-7, D3), counting of non-stale claims (AC-7, D3), and the + * invariant that claim files are never parsed as perf JSONL. + * + * Real files, real filesystem, no mocks. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { + PerfRetention, + PERF_MAX_BYTES, + PERF_MAX_FILES, + PERF_MAINTENANCE_INTERVAL_MS, + PERF_CLAIM_LEASE_MS, + type PerfScheduler, + type PerfTimerHandle, +} from './retention.js'; +import { readPerfRecords } from './perfRecords.js'; + +let dir: string; + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'perf-retention-')); +}); + +afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); +}); + +function writePerfFile( + name: string, + recordCount: number, + recordBytes = 1220, +): void { + const lines: string[] = []; + for (let i = 0; i < recordCount; i++) { + const padding = '.'.repeat( + Math.max(0, recordBytes - 80 - String(i).length), + ); + lines.push( + JSON.stringify({ + schema_version: 1, + record_type: 'operation', + ts: '2026-08-08T12:00:00.000Z', + pad: padding, + idx: i, + }), + ); + } + fs.writeFileSync(path.join(dir, name), lines.join('\n') + '\n'); +} + +function writePerfFileExact(name: string, content: string): void { + fs.writeFileSync(path.join(dir, name), content); +} + +function createClaimFile(uuid: string, mtimeMs: number): void { + const p = path.join(dir, `${uuid}.claim`); + fs.writeFileSync(p, '', { mode: 0o600 }); + fs.utimesSync(p, new Date(mtimeMs), new Date(mtimeMs)); +} + +function listFiles(): string[] { + return fs.readdirSync(dir).sort(); +} + +class TestScheduler implements PerfScheduler { + callback: (() => Promise) | null = null; + handle: PerfTimerHandle | null = null; + + setInterval(callback: () => Promise, _ms: number): PerfTimerHandle { + this.callback = callback; + this.handle = { unref: () => {}, clear: () => {} }; + return this.handle; + } +} + +describe('PerfRetention constants (D5)', () => { + it('MAX_BYTES is 64 MiB', () => { + expect(PERF_MAX_BYTES).toBe(64 * 1024 * 1024); + }); + + it('MAX_FILES is 128', () => { + expect(PERF_MAX_FILES).toBe(128); + }); + + it('MAINTENANCE_INTERVAL_MS is 60 seconds', () => { + expect(PERF_MAINTENANCE_INTERVAL_MS).toBe(60_000); + }); + + it('CLAIM_LEASE_MS is three maintenance intervals (180s)', () => { + expect(PERF_CLAIM_LEASE_MS).toBe(PERF_MAINTENANCE_INTERVAL_MS * 3); + }); +}); + +describe('PerfRetention claim lifecycle (AC-7, D3)', () => { + it('start() creates a UUID claim file exclusively', async () => { + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000000', + }); + await retention.start(); + + const files = listFiles(); + expect(files).toContain('00000000-0000-4000-8000-000000000000.claim'); + await retention.dispose(); + }); + + it.skipIf(process.platform === 'win32')( + 'start() creates the claim with 0600 permissions', + async () => { + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000001', + }); + await retention.start(); + + const stat = fs.statSync( + path.join(dir, '00000000-0000-4000-8000-000000000001.claim'), + ); + expect(stat.mode & 0o777).toBe(0o600); + await retention.dispose(); + }, + ); + + it('start() creates only the claim — no perf JSONL', async () => { + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000002', + }); + await retention.start(); + + const files = listFiles(); + expect(files).toEqual(['00000000-0000-4000-8000-000000000002.claim']); + await retention.dispose(); + }); + + it('tick() touches the claim mtime (within lease window)', async () => { + const scheduler = new TestScheduler(); + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000003', + scheduler, + maintenanceIntervalMs: 60_000, + claimLeaseMs: 180_000, + }); + await retention.start(); + + const beforeMtime = fs.statSync( + path.join(dir, '00000000-0000-4000-8000-000000000003.claim'), + ).mtimeMs; + + await new Promise((r) => setTimeout(r, 20)); + + await scheduler.callback!(); + + const afterMtime = fs.statSync( + path.join(dir, '00000000-0000-4000-8000-000000000003.claim'), + ).mtimeMs; + + expect(afterMtime).toBeGreaterThan(beforeMtime); + await retention.dispose(); + }); + + it('dispose() removes the claim file cleanly', async () => { + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000004', + }); + await retention.start(); + expect(listFiles()).toContain('00000000-0000-4000-8000-000000000004.claim'); + + await retention.dispose(); + expect(listFiles()).not.toContain( + '00000000-0000-4000-8000-000000000004.claim', + ); + }); + + it('dispose() stops the interval (no further ticks)', async () => { + const scheduler = new TestScheduler(); + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000005', + scheduler, + }); + await retention.start(); + await retention.dispose(); + + // Verify the claim file THIS instance created was removed by dispose(). + expect(listFiles()).not.toContain( + '00000000-0000-4000-8000-000000000005.claim', + ); + + // Verify the interval callback is no longer active: calling it after + // dispose should not re-create the claim file. + if (scheduler.callback) { + await scheduler.callback(); + } + expect(listFiles()).not.toContain( + '00000000-0000-4000-8000-000000000005.claim', + ); + }); + + it('crash (no dispose) leaves a stale claim until sweep', async () => { + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000006', + }); + await retention.start(); + + const files = listFiles(); + expect(files).toContain('00000000-0000-4000-8000-000000000006.claim'); + + expect( + fs.existsSync( + path.join(dir, '00000000-0000-4000-8000-000000000006.claim'), + ), + ).toBe(true); + }); +}); + +describe('PerfRetention countNonStaleClaims (AC-7, D3)', () => { + it('counts only fresh claims within the lease window', async () => { + const now = Date.now(); + + createClaimFile('fresh-uuid', now - 10_000); + + createClaimFile('stale-uuid', now - PERF_CLAIM_LEASE_MS - 1); + + createClaimFile('another-fresh', now - 60_000); + + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000007', + }); + const count = await retention.countNonStaleClaims(now); + expect(count).toBe(2); + }); + + it('returns 0 when only stale claims exist', async () => { + const now = Date.now(); + createClaimFile('old1', now - PERF_CLAIM_LEASE_MS - 1000); + createClaimFile('old2', now - PERF_CLAIM_LEASE_MS - 5000); + + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000008', + }); + const count = await retention.countNonStaleClaims(now); + expect(count).toBe(0); + }); + + it('returns 0 when no claims exist', async () => { + writePerfFile('perf-20260808-uuid.jsonl', 3); + const retention = new PerfRetention({ dir, runUuid: 'uuid' }); + const count = await retention.countNonStaleClaims(Date.now()); + expect(count).toBe(0); + }); + + it('ignores non-claim files when counting', async () => { + const now = Date.now(); + createClaimFile('fresh', now - 10_000); + writePerfFile('perf-20260808-uuid.jsonl', 3); + + const retention = new PerfRetention({ dir, runUuid: 'old' }); + const count = await retention.countNonStaleClaims(now); + expect(count).toBe(1); + }); +}); + +describe('PerfRetention — claims never parsed as JSONL', () => { + it('a .claim file is not picked up by readPerfRecords', async () => { + const now = Date.now(); + + const validRecord = JSON.stringify({ + schema_version: 1, + record_type: 'operation', + ts: '2026-08-08T12:00:00.000Z', + session_id: 'sess-abc', + operation_id: 'sess-abc#agentic-loop#f7e2', + runtime_id: 'rt-main', + parent_runtime_id: null, + subagent_name: null, + project_hash: 'sha256:hash', + llxprt_version: '0.11.0', + git_sha: 'abc1234', + runtime: 'bun-1.3.14', + platform: 'darwin-arm64', + provider: 'openai', + model: 'gpt-4o', + context_tokens: 1000, + output_tokens: 500, + terminal_cols: 120, + terminal_rows: 40, + render_mode: 'incremental', + concurrent_instances: 1, + status: 'completed', + client_prepare_ms: 5, + stream_handler_ms: 10, + ink_render_ms: 20, + ink_render_count: 3, + stdout_bytes: 4096, + stdout_write_calls: 3, + stdout_write_sync_ms: 2, + client_finalize_ms: 1, + provider_attempts: 1, + provider_attempt_sum_ms: 800, + provider_union_ms: 800, + tool_calls: 2, + tool_call_sum_ms: 300, + tool_union_ms: 280, + agent_activity_union_ms: 1000, + operation_elapsed_ms: 1200, + approval_wait_ms: 0, + unclassified_elapsed_ms: 100, + session_operation_index: 1, + uptime_ms: 50000, + }); + writePerfFileExact( + 'perf-20260808-00000000-0000-4000-8000-00000000001c.jsonl', + validRecord + String.fromCharCode(10), + ); + createClaimFile('00000000-0000-4000-8000-00000000001c', now); + + const perfFiles = listFiles().filter( + (f) => f.startsWith('perf-') && f.endsWith('.jsonl'), + ); + expect(perfFiles).toEqual([ + 'perf-20260808-00000000-0000-4000-8000-00000000001c.jsonl', + ]); + + expect(listFiles()).toContain('00000000-0000-4000-8000-00000000001c.claim'); + expect(perfFiles).not.toContain( + '00000000-0000-4000-8000-00000000001c.claim', + ); + + const { counts } = await readPerfRecords(path.join(dir, perfFiles[0])); + expect(counts.parsed).toBe(1); + }); +}); diff --git a/packages/telemetry/src/perf/retention.eviction.behavior.test.ts b/packages/telemetry/src/perf/retention.eviction.behavior.test.ts new file mode 100644 index 0000000000..169e838eef --- /dev/null +++ b/packages/telemetry/src/perf/retention.eviction.behavior.test.ts @@ -0,0 +1,353 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Eviction-policy behavioral tests for PerfRetention (AC-7, D3). + * + * Covers the eviction decisions split out of the original + * retention.behavior.test.ts: live-writer safety (AC-7), claim handling during + * retention (AC-7, D3), oldest-first cap convergence (AC-7), and future-mtime + * boundary protection (AC-7 boundary). + * + * Real files, real filesystem, no mocks. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { + PerfRetention, + PERF_CLAIM_LEASE_MS, + PERF_MAINTENANCE_INTERVAL_MS, +} from './retention.js'; + +let dir: string; + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'perf-retention-')); +}); + +afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); +}); + +function writePerfFile( + name: string, + recordCount: number, + recordBytes = 1220, +): void { + const lines: string[] = []; + for (let i = 0; i < recordCount; i++) { + const padding = '.'.repeat( + Math.max(0, recordBytes - 80 - String(i).length), + ); + lines.push( + JSON.stringify({ + schema_version: 1, + record_type: 'operation', + ts: '2026-08-08T12:00:00.000Z', + pad: padding, + idx: i, + }), + ); + } + fs.writeFileSync(path.join(dir, name), lines.join('\n') + '\n'); +} + +function createClaimFile(uuid: string, mtimeMs: number): void { + const p = path.join(dir, `${uuid}.claim`); + fs.writeFileSync(p, '', { mode: 0o600 }); + fs.utimesSync(p, new Date(mtimeMs), new Date(mtimeMs)); +} + +function setMtime(name: string, mtimeMs: number): void { + const p = path.join(dir, name); + fs.utimesSync(p, new Date(mtimeMs), new Date(mtimeMs)); +} + +function listFiles(): string[] { + return fs.readdirSync(dir).sort(); +} + +describe('PerfRetention live-writer safety (AC-7)', () => { + it('does NOT evict a today file with mtime within the maintenance window', async () => { + const now = Date.now(); + + writePerfFile('perf-20260101-old.jsonl', 5); + setMtime('perf-20260101-old.jsonl', now - 86_400_000); + + const todayKey = utcDayKey(now); + writePerfFile(`perf-${todayKey}-live.jsonl`, 5); + setMtime(`perf-${todayKey}-live.jsonl`, now - 5_000); + + createClaimFile('stalestale', now - PERF_CLAIM_LEASE_MS - 1); + + const retention = new PerfRetention({ + dir, + runUuid: 'stale', + maxFiles: 2, + maxBytes: 10_000, + }); + await retention.maintain(now); + + expect(fs.existsSync(path.join(dir, 'perf-20260101-old.jsonl'))).toBe( + false, + ); + expect(fs.existsSync(path.join(dir, `perf-${todayKey}-live.jsonl`))).toBe( + true, + ); + }); + + it('evicts a today file whose mtime is older than the maintenance window', async () => { + const now = Date.now(); + const todayKey = utcDayKey(now); + + writePerfFile(`perf-${todayKey}-stale.jsonl`, 5); + setMtime( + `perf-${todayKey}-stale.jsonl`, + now - PERF_MAINTENANCE_INTERVAL_MS - 1, + ); + + writePerfFile('perf-20260101-older.jsonl', 5); + setMtime('perf-20260101-older.jsonl', now - 86_400_000 * 2); + + // With maxBytes: 1 the byte cap forces every eligible file to be evicted, + // proving the stale-mtime today file is NOT protected by live-writer safety. + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-0000000000aa', + maxFiles: 10_000_000, + maxBytes: 1, + }); + await retention.maintain(now); + + expect(fs.existsSync(path.join(dir, 'perf-20260101-older.jsonl'))).toBe( + false, + ); + expect(fs.existsSync(path.join(dir, `perf-${todayKey}-stale.jsonl`))).toBe( + false, + ); + }); +}); + +describe('PerfRetention claim handling in retention (AC-7, D3)', () => { + it('claims count toward artifact count and bytes but are never JSONL parsed', async () => { + const now = Date.now(); + const todayKey = utcDayKey(now); + + // Today's live JSONL — protected by live-writer safety. + writePerfFile(`perf-${todayKey}-live.jsonl`, 3); + setMtime(`perf-${todayKey}-live.jsonl`, now - 5_000); + // A stale claim (0 bytes) — counts toward the count cap and is eligible. + createClaimFile( + '00000000-0000-4000-8000-00000000000f', + now - PERF_CLAIM_LEASE_MS - 1, + ); + + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-0000000000a0', + maxFiles: 1, + maxBytes: 10_000_000, + }); + await retention.maintain(now); + + // The stale claim was counted (2 artifacts > 1 cap) and evicted; the + // live JSONL survives. Claims are accounting-only, never parsed as records. + const remaining = listFiles(); + expect(remaining).toEqual([`perf-${todayKey}-live.jsonl`]); + }); + + it('a fresh claim is never evicted', async () => { + const now = Date.now(); + createClaimFile('00000000-0000-4000-8000-00000000000e', now - 10_000); + + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-00000000000d', + maxFiles: 1, + maxBytes: 1, + }); + await retention.maintain(now); + + expect( + fs.existsSync( + path.join(dir, '00000000-0000-4000-8000-00000000000e.claim'), + ), + ).toBe(true); + }); + + it('a stale claim IS eligible for eviction', async () => { + const now = Date.now(); + // A stale claim for a DIFFERENT run (past the lease) — eligible. + createClaimFile( + '00000000-0000-4000-8000-00000000000a', + now - PERF_CLAIM_LEASE_MS - 1, + ); + // The owner's own claim, kept fresh — protected (non-stale). + createClaimFile('00000000-0000-4000-8000-00000000000f', now - 10_000); + + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-00000000000f', + maxFiles: 1, + maxBytes: 1, + }); + await retention.maintain(now); + + expect( + fs.existsSync( + path.join(dir, '00000000-0000-4000-8000-00000000000a.claim'), + ), + ).toBe(false); + expect( + fs.existsSync( + path.join(dir, '00000000-0000-4000-8000-00000000000f.claim'), + ), + ).toBe(true); + }); + + it('a future-mtime claim is protected until it becomes eligible', async () => { + const now = Date.now(); + + createClaimFile('00000000-0000-4000-8000-000000000010', now + 3_600_000); + + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000011', + maxFiles: 1, + maxBytes: 1, + }); + await retention.maintain(now); + + expect( + fs.existsSync( + path.join(dir, '00000000-0000-4000-8000-000000000010.claim'), + ), + ).toBe(true); + }); +}); + +describe('PerfRetention oldest-first cap convergence (AC-7)', () => { + it('evicts oldest-first until under both caps', async () => { + const now = Date.now(); + + for (let i = 0; i < 5; i++) { + writePerfFile(`perf-2026010${i}-file.jsonl`, 1); + setMtime(`perf-2026010${i}-file.jsonl`, now - (5 - i) * 86_400_000); + } + + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000040', + maxFiles: 2, + maxBytes: 10_000_000, + }); + await retention.maintain(now); + + const remaining = listFiles(); + expect(remaining.length).toBe(2); + + expect(remaining).toContain('perf-20260104-file.jsonl'); + expect(remaining).toContain('perf-20260103-file.jsonl'); + }); + + it('stable deterministic tie-break by name when mtimes are equal', async () => { + const now = Date.now(); + + writePerfFile( + 'perf-20260101-00000000-0000-4000-8000-000000000011.jsonl', + 1, + ); + writePerfFile( + 'perf-20260101-00000000-0000-4000-8000-000000000012.jsonl', + 1, + ); + setMtime( + 'perf-20260101-00000000-0000-4000-8000-000000000011.jsonl', + now - 86_400_000, + ); + setMtime( + 'perf-20260101-00000000-0000-4000-8000-000000000012.jsonl', + now - 86_400_000, + ); + + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000099', + maxFiles: 1, + maxBytes: 10_000_000, + }); + await retention.maintain(now); + + const remaining = listFiles(); + // Equal mtimes → tie-break by name: ...011 sorts before ...012, so ...011 + // is evicted first and ...012 survives. + expect(remaining).toContain( + 'perf-20260101-00000000-0000-4000-8000-000000000012.jsonl', + ); + expect(remaining).not.toContain( + 'perf-20260101-00000000-0000-4000-8000-000000000011.jsonl', + ); + }); + + it('converges after enough sweeps under concurrent append overshoot', async () => { + const now = Date.now(); + + for (let i = 0; i < 10; i++) { + writePerfFile(`perf-2026010${i}-f.jsonl`, 1); + setMtime(`perf-2026010${i}-f.jsonl`, now - (10 - i) * 3_600_000); + } + + const retention = new PerfRetention({ + dir, + runUuid: 'new', + maxFiles: 3, + maxBytes: 10_000_000, + }); + + await retention.maintain(now); + + writePerfFile('perf-20260110-new.jsonl', 1); + setMtime('perf-20260110-new.jsonl', now); + + await retention.maintain(now); + + const remaining = listFiles(); + expect(remaining.length).toBeLessThanOrEqual(3); + }); +}); + +describe('PerfRetention future mtime (AC-7 boundary)', () => { + it('a file with materially-future mtime is protected', async () => { + const now = Date.now(); + const todayKey = utcDayKey(now); + writePerfFile(`perf-${todayKey}-future.jsonl`, 3); + setMtime(`perf-${todayKey}-future.jsonl`, now + 3_600_000); + + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-00000000001b', + maxFiles: 1, + maxBytes: 1, + onDiagnostic: () => {}, + }); + await retention.maintain(now); + + expect(fs.existsSync(path.join(dir, `perf-${todayKey}-future.jsonl`))).toBe( + true, + ); + }); +}); + +function utcDayKey(now: number): string { + const date = new Date(now); + const year = date.getUTCFullYear(); + const month = String(date.getUTCMonth() + 1).padStart(2, '0'); + const day = String(date.getUTCDate()).padStart(2, '0'); + return `${year}${month}${day}`; +} diff --git a/packages/telemetry/src/perf/retention.faults.behavior.test.ts b/packages/telemetry/src/perf/retention.faults.behavior.test.ts new file mode 100644 index 0000000000..cae58dc3dc --- /dev/null +++ b/packages/telemetry/src/perf/retention.faults.behavior.test.ts @@ -0,0 +1,315 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Fault-tolerance and diagnostics behavioral tests for PerfRetention + * (AC-7, D6, D-LC-4). + * + * Covers the error-path behavior split out of the original + * retention.behavior.test.ts: failed-unlink accounting that stays intact + * (AC-7, D6) and fail-open diagnostics emitted on filesystem failures + * (D-LC-4). + * + * Real files, real filesystem. Faults are injected through the package-private + * FaultInjectingRetentionFilesystem port (no monkeypatching of node:fs). + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { + PerfRetention, + FaultInjectingRetentionFilesystem, +} from './retention.js'; + +let dir: string; + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'perf-retention-')); +}); + +afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); +}); + +function writePerfFile( + name: string, + recordCount: number, + recordBytes = 1220, +): void { + const lines: string[] = []; + for (let i = 0; i < recordCount; i++) { + const padding = '.'.repeat( + Math.max(0, recordBytes - 80 - String(i).length), + ); + lines.push( + JSON.stringify({ + schema_version: 1, + record_type: 'operation', + ts: '2026-08-08T12:00:00.000Z', + pad: padding, + idx: i, + }), + ); + } + fs.writeFileSync(path.join(dir, name), lines.join('\n') + '\n'); +} + +function createClaimFile(uuid: string, mtimeMs: number): void { + const p = path.join(dir, `${uuid}.claim`); + fs.writeFileSync(p, '', { mode: 0o600 }); + fs.utimesSync(p, new Date(mtimeMs), new Date(mtimeMs)); +} + +function setMtime(name: string, mtimeMs: number): void { + const p = path.join(dir, name); + fs.utimesSync(p, new Date(mtimeMs), new Date(mtimeMs)); +} + +describe('PerfRetention failed unlink — accounting intact (AC-7, D6)', () => { + it('does NOT decrement file/byte count when unlink fails', async () => { + const now = Date.now(); + const firstName = + 'perf-20260101-00000000-0000-4000-8000-0000000000ee.jsonl'; + const secondName = + 'perf-20260102-00000000-0000-4000-8000-0000000000ee.jsonl'; + writePerfFile(firstName, 5); + setMtime(firstName, now - 86_400_000); + writePerfFile(secondName, 5); + setMtime(secondName, now - 43_200_000); + + const diagnostics: string[] = []; + const faultFs = new FaultInjectingRetentionFilesystem({ + failMethod: 'unlink', + code: 'EACCES', + }); + + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000014', + fs: faultFs, + maxFiles: 1, + maxBytes: 10_000_000, + onDiagnostic: (m) => diagnostics.push(m), + }); + await retention.maintain(now); + + expect(retention.evictionCount).toBe(0); + expect(diagnostics.length).toBeGreaterThanOrEqual(1); + expect(diagnostics[0]).toContain('EACCES'); + expect(fs.existsSync(path.join(dir, firstName))).toBe(true); + expect(fs.existsSync(path.join(dir, secondName))).toBe(true); + }); + + it('diagnostics are rate-limited for repeated unlink failures', async () => { + const now = Date.now(); + + for (let i = 0; i < 5; i++) { + writePerfFile(`perf-2026010${i}-fail.jsonl`, 1); + setMtime(`perf-2026010${i}-fail.jsonl`, now - (5 - i) * 86_400_000); + } + + const diagnostics: string[] = []; + const faultFs = new FaultInjectingRetentionFilesystem({ + failMethod: 'unlink', + code: 'EACCES', + }); + + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000015', + fs: faultFs, + maxFiles: 1, + maxBytes: 10_000_000, + diagRateLimitMs: 60_000, + onDiagnostic: (msg) => diagnostics.push(msg), + }); + await retention.maintain(now); + + expect(diagnostics).toHaveLength(1); + }); +}); + +describe('PerfRetention fail-open diagnostics (D-LC-4)', () => { + it('emits a rate-limited diagnostic when stat fails during maintain', async () => { + const now = Date.now(); + writePerfFile('perf-20260101-a.jsonl', 1); + setMtime('perf-20260101-a.jsonl', now - 86_400_000); + + const diagnostics: string[] = []; + const faultFs = new FaultInjectingRetentionFilesystem({ + failMethod: 'stat', + code: 'EACCES', + }); + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-00000000001c', + fs: faultFs, + maxFiles: 1, + maxBytes: 1, + onDiagnostic: (m) => diagnostics.push(m), + }); + await retention.maintain(now); + + expect(diagnostics.length).toBeGreaterThanOrEqual(1); + expect(diagnostics[0]).toContain('EACCES'); + + expect(fs.existsSync(path.join(dir, 'perf-20260101-a.jsonl'))).toBe(true); + }); + + it('emits a diagnostic when readdir fails in countNonStaleClaims', async () => { + const diagnostics: string[] = []; + const faultFs = new FaultInjectingRetentionFilesystem({ + failMethod: 'readdir', + code: 'EACCES', + }); + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-00000000001d', + fs: faultFs, + onDiagnostic: (m) => diagnostics.push(m), + }); + const count = await retention.countNonStaleClaims(Date.now()); + + expect(count).toBe(0); + expect(diagnostics.length).toBe(1); + expect(diagnostics[0]).toContain('EACCES'); + }); + + it('emits a diagnostic when a claim stat fails in countNonStaleClaims', async () => { + const now = Date.now(); + createClaimFile('fresh', now - 10_000); + + const diagnostics: string[] = []; + const faultFs = new FaultInjectingRetentionFilesystem({ + failMethod: 'stat', + code: 'EACCES', + }); + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-00000000001e', + fs: faultFs, + onDiagnostic: (m) => diagnostics.push(m), + }); + const count = await retention.countNonStaleClaims(now); + + expect(count).toBe(0); + expect(diagnostics.length).toBeGreaterThanOrEqual(1); + expect(diagnostics[0]).toContain('EACCES'); + }); + + it('emits a diagnostic for an ENOENT race in countNonStaleClaims', async () => { + const now = Date.now(); + createClaimFile('racy', now - 10_000); + + const diagnostics: string[] = []; + const faultFs = new FaultInjectingRetentionFilesystem({ + failMethod: 'stat', + code: 'ENOENT', + }); + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-00000000001f', + fs: faultFs, + onDiagnostic: (m) => diagnostics.push(m), + }); + const count = await retention.countNonStaleClaims(now); + + expect(count).toBe(0); + expect(diagnostics.length).toBeGreaterThanOrEqual(1); + expect(diagnostics[0]).toContain('ENOENT'); + }); +}); + +// --------------------------------------------------------------------------- +// Claim start failure: truthful retryable state + recovered start (D3, D6) +// --------------------------------------------------------------------------- + +describe('PerfRetention claim start failure — truthful retryable state (D3, D6)', () => { + it('EACCES on openExclusive leaves truthful retryable state; a recovered start creates one claim', async () => { + const runUuid = '00000000-0000-4000-8000-000000000030'; + const faultFs = new FaultInjectingRetentionFilesystem({ + failMethod: 'openExclusive', + code: 'EACCES', + }); + const diagnostics: string[] = []; + const retention = new PerfRetention({ + dir, + runUuid, + fs: faultFs, + onDiagnostic: (m) => diagnostics.push(m), + }); + + // start() must fail-open (no throw) on the errno error. + await retention.start(); + + // Truthful state: no claim file was created, and a diagnostic was emitted. + expect(fs.existsSync(path.join(dir, `${runUuid}.claim`))).toBe(false); + expect(diagnostics.length).toBeGreaterThanOrEqual(1); + expect(diagnostics[0]).toContain('EACCES'); + + // Disposing a failed-start instance is safe (no timer, no claim). + await retention.dispose(); + + // Recovered start: a new instance with a working filesystem creates + // exactly one claim in the same directory. + const recovered = new PerfRetention({ + dir, + runUuid, + onDiagnostic: () => {}, + }); + await recovered.start(); + + const files = fs.readdirSync(dir).sort(); + expect(files).toEqual([`${runUuid}.claim`]); + + await recovered.dispose(); + expect(fs.readdirSync(dir)).toHaveLength(0); + }); + + it('EEXIST (pre-existing claim) leaves truthful retryable state; removing the stale claim allows recovered start', async () => { + const runUuid = '00000000-0000-4000-8000-000000000031'; + const claimPath = path.join(dir, `${runUuid}.claim`); + + // Pre-create the claim file so openExclusive throws EEXIST on the real fs. + fs.writeFileSync(claimPath, '', { mode: 0o600 }); + + const diagnostics: string[] = []; + const retention = new PerfRetention({ + dir, + runUuid, + onDiagnostic: (m) => diagnostics.push(m), + }); + + // start() must fail-open on EEXIST. + await retention.start(); + + expect(diagnostics.length).toBeGreaterThanOrEqual(1); + + // No timer was created; disposing the failed-start instance is safe. + await retention.dispose(); + + // The pre-existing file remains (not a phantom from our failed start). + expect(fs.existsSync(claimPath)).toBe(true); + + // Remove the stale claim → recovered start creates exactly one claim. + fs.unlinkSync(claimPath); + const recovered = new PerfRetention({ + dir, + runUuid, + onDiagnostic: () => {}, + }); + await recovered.start(); + + expect(fs.existsSync(claimPath)).toBe(true); + const files = fs.readdirSync(dir).sort(); + expect(files).toEqual([`${runUuid}.claim`]); + + await recovered.dispose(); + expect(fs.readdirSync(dir)).toHaveLength(0); + }); +}); diff --git a/packages/telemetry/src/perf/retention.lifecycle.behavior.test.ts b/packages/telemetry/src/perf/retention.lifecycle.behavior.test.ts new file mode 100644 index 0000000000..2d5ea36af8 --- /dev/null +++ b/packages/telemetry/src/perf/retention.lifecycle.behavior.test.ts @@ -0,0 +1,707 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Lifecycle / concurrency behavioral tests for PerfRetention (P08). + * + * Covers the lifecycle defects surfaced in independent source inspection: + * D-LC-1: dispose actually cancels the interval (clear), not merely nulls the + * handle. Proven with an auto-firing scheduler that would keep firing + * unless clear() is invoked. + * D-LC-2: dispose awaits an in-flight tick before unlinking the claim. Proven + * with a controllable real-file gate that blocks touch until released. + * D-LC-3: an internal (non-errno) rejection during a tick is observable via a + * deterministic scheduler (await/reject), not silently swallowed. + * + * Real files, real filesystem, no mocks of fs. Gating is achieved by wrapping + * the package-private filesystem port with controllable deferreds. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import { promises as fsp } from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { + PerfRetention, + type PerfScheduler, + type PerfTimerHandle, + type PerfRetentionFilesystem, +} from './retention.js'; + +let dir: string; + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'perf-lifecycle-')); +}); + +afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); +}); + +// --------------------------------------------------------------------------- +// Helpers +// --------------------------------------------------------------------------- + +interface Deferred { + promise: Promise; + resolve: (value: T) => void; + reject: (err: unknown) => void; +} + +function createDeferred(): Deferred { + let resolve!: (value: T) => void; + let reject!: (err: unknown) => void; + const promise = new Promise((res, rej) => { + resolve = res; + reject = rej; + }); + return { promise, resolve, reject }; +} + +/** Real-fs port that delegates to node:fs/promises (test shared dir). */ +function realFilesystem(): PerfRetentionFilesystem { + return { + async ensureDir(d: string): Promise { + try { + await fsp.access(d); + } catch { + await fsp.mkdir(d, { recursive: true, mode: 0o700 }); + } + }, + async openExclusive(p: string, mode: number): Promise { + const h = await fsp.open(p, 'wx', mode); + await h.close(); + }, + async utimes(p: string, atime: Date, mtime: Date): Promise { + await fsp.utimes(p, atime, mtime); + }, + async readdir(d: string): Promise { + return fsp.readdir(d); + }, + async stat(p: string): Promise<{ size: number; mtimeMs: number }> { + const s = await fsp.stat(p); + return { size: s.size, mtimeMs: s.mtimeMs }; + }, + async unlink(p: string): Promise { + await fsp.unlink(p); + }, + }; +} + +/** Captures the interval callback for deterministic firing. */ +class CapturingScheduler implements PerfScheduler { + callback: (() => Promise) | null = null; + + setInterval(callback: () => Promise): PerfTimerHandle { + this.callback = callback; + return { unref: () => {}, clear: () => {} }; + } +} + +class ThrowOnceScheduler implements PerfScheduler { + attempts = 0; + callback: (() => Promise) | null = null; + + setInterval(callback: () => Promise): PerfTimerHandle { + this.attempts += 1; + if (this.attempts === 1) { + throw new Error('scheduler setup failed'); + } + this.callback = callback; + return { unref: () => {}, clear: () => {} }; + } +} + +class ThrowOnceUnrefScheduler implements PerfScheduler { + attempts = 0; + clearCalls = 0; + + setInterval(): PerfTimerHandle { + this.attempts += 1; + const throwOnUnref = this.attempts === 1; + return { + unref: () => { + if (throwOnUnref) throw new Error('scheduler unref failed'); + }, + clear: () => { + this.clearCalls += 1; + }, + }; + } +} + +/** + * A scheduler backed by a REAL native interval that keeps firing until clear() + * is called on the returned handle. Used to prove disposal actually cancels + * firing (behaviorally), not just source-level. + */ +class AutoFiringScheduler implements PerfScheduler { + fireCount = 0; + private native: ReturnType | null = null; + + setInterval(callback: () => Promise, ms: number): PerfTimerHandle { + this.native = setInterval(() => { + this.fireCount += 1; + void callback().catch(() => { + /* internal errors are surfaced by the retention path; ignore here */ + }); + }, ms); + return { + unref: () => { + const h = this.native as unknown as { unref?: () => void }; + if (typeof h.unref === 'function') h.unref(); + }, + clear: () => { + if (this.native !== null) { + clearInterval(this.native); + this.native = null; + } + }, + }; + } +} + +// --------------------------------------------------------------------------- +// Start transaction: scheduler setup failure rolls back claim/state +// --------------------------------------------------------------------------- + +describe('PerfRetention scheduler setup rollback', () => { + it('removes the claim and permits a successful start retry', async () => { + const runUuid = '00000000-0000-4000-8000-00000000000a'; + const claimPath = path.join(dir, `${runUuid}.claim`); + const scheduler = new ThrowOnceScheduler(); + const retention = new PerfRetention({ + dir, + runUuid, + scheduler, + onDiagnostic: () => {}, + }); + + await expect(retention.start()).rejects.toThrow('scheduler setup failed'); + expect(fs.existsSync(claimPath)).toBe(false); + + await expect(retention.start()).resolves.toBeUndefined(); + expect(scheduler.attempts).toBe(2); + expect(scheduler.callback).not.toBeNull(); + expect(fs.existsSync(claimPath)).toBe(true); + + await retention.dispose(); + expect(fs.existsSync(claimPath)).toBe(false); + }); + + it('clears the timer and rolls back the claim when unref fails', async () => { + const runUuid = '00000000-0000-4000-8000-00000000000b'; + const claimPath = path.join(dir, `${runUuid}.claim`); + const scheduler = new ThrowOnceUnrefScheduler(); + const retention = new PerfRetention({ + dir, + runUuid, + scheduler, + onDiagnostic: () => {}, + }); + + await expect(retention.start()).rejects.toThrow('scheduler unref failed'); + expect(scheduler.clearCalls).toBe(1); + expect(fs.existsSync(claimPath)).toBe(false); + + await expect(retention.start()).resolves.toBeUndefined(); + expect(scheduler.attempts).toBe(2); + expect(fs.existsSync(claimPath)).toBe(true); + + await retention.dispose(); + expect(scheduler.clearCalls).toBe(2); + expect(fs.existsSync(claimPath)).toBe(false); + }); +}); + +// --------------------------------------------------------------------------- +// D-LC-1: dispose cancels the interval (clear), not just nulls the handle +// --------------------------------------------------------------------------- + +describe('PerfRetention dispose cancels the interval (D-LC-1)', () => { + it('stops firing after dispose when the scheduler keeps firing until clear', async () => { + const scheduler = new AutoFiringScheduler(); + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000000', + scheduler, + maintenanceIntervalMs: 8, + onDiagnostic: () => {}, + }); + await retention.start(); + + // Let the auto-firing scheduler fire several times. + await new Promise((r) => setTimeout(r, 40)); + const warmed = scheduler.fireCount; + expect(warmed).toBeGreaterThan(0); + + await retention.dispose(); + + // After dispose, the native interval must be cancelled. Wait long enough + // that several more firings WOULD have occurred if clear() were not called. + await new Promise((r) => setTimeout(r, 40)); + expect(scheduler.fireCount).toBe(warmed); + }); + + it('does not touch the claim after dispose (auto-firing scheduler)', async () => { + const scheduler = new AutoFiringScheduler(); + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000001', + scheduler, + maintenanceIntervalMs: 8, + onDiagnostic: () => {}, + }); + await retention.start(); + await retention.dispose(); + + // The claim must be gone, and no amount of waiting recreates it. + await new Promise((r) => setTimeout(r, 30)); + expect( + fs.existsSync( + path.join(dir, '00000000-0000-4000-8000-000000000001.claim'), + ), + ).toBe(false); + }); +}); + +// --------------------------------------------------------------------------- +// D-LC-2: dispose awaits an in-flight tick before unlinking the claim +// --------------------------------------------------------------------------- + +describe('PerfRetention dispose awaits in-flight tick (D-LC-2)', () => { + it('waits for a touch blocked on a gate, then unlinks — no post-dispose touch', async () => { + const touchGate = createDeferred(); + let touchCount = 0; + const gatedFs: PerfRetentionFilesystem = { + ...realFilesystem(), + async utimes(p, atime, mtime): Promise { + touchCount += 1; + // Block until the test releases the gate. + await touchGate.promise; + await fsp.utimes(p, atime, mtime); + }, + }; + const scheduler = new CapturingScheduler(); + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000002', + fs: gatedFs, + scheduler, + onDiagnostic: () => {}, + }); + await retention.start(); + + // Fire the interval tick WITHOUT awaiting — it blocks inside touchClaim. + const tickPromise = scheduler.callback!().catch(() => {}); + await new Promise((r) => setTimeout(r, 15)); + + // Begin dispose while the tick is still in flight (blocked at the gate). + let disposed = false; + const disposePromise = retention.dispose().then(() => { + disposed = true; + }); + await new Promise((r) => setTimeout(r, 15)); + + // Dispose must NOT have resolved yet: the in-flight tick is still running. + expect(disposed).toBe(false); + // The claim must still be present: dispose has not unlinked it yet. + expect( + fs.existsSync( + path.join(dir, '00000000-0000-4000-8000-000000000002.claim'), + ), + ).toBe(true); + + // Release the gate: the in-flight touch completes, then dispose unlinks. + touchGate.resolve(); + await tickPromise; + await disposePromise; + + expect(disposed).toBe(true); + expect( + fs.existsSync( + path.join(dir, '00000000-0000-4000-8000-000000000002.claim'), + ), + ).toBe(false); + // Exactly one touch (the in-flight one). No post-dispose touch occurs. + expect(touchCount).toBe(1); + }); + + it('dispose then a late scheduler fire performs no touch', async () => { + const touchGate = createDeferred(); + let touchCount = 0; + const gatedFs: PerfRetentionFilesystem = { + ...realFilesystem(), + async utimes(p, atime, mtime): Promise { + touchCount += 1; + await touchGate.promise; + await fsp.utimes(p, atime, mtime); + }, + }; + const scheduler = new CapturingScheduler(); + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000003', + fs: gatedFs, + scheduler, + onDiagnostic: () => {}, + }); + await retention.start(); + + // Block a tick, then dispose (which awaits it), then release. + const tickPromise = scheduler.callback!().catch(() => {}); + const disposePromise = retention.dispose(); + await new Promise((r) => setTimeout(r, 10)); + touchGate.resolve(); + await tickPromise; + await disposePromise; + + // Now attempt a LATE fire of the captured callback — disposed, so no touch. + const before = touchCount; + await scheduler.callback!().catch(() => {}); + expect(touchCount).toBe(before); + expect( + fs.existsSync( + path.join(dir, '00000000-0000-4000-8000-000000000003.claim'), + ), + ).toBe(false); + }); +}); + +// --------------------------------------------------------------------------- +// D-LC-3: internal rejection is observable (await/reject), not swallowed +// --------------------------------------------------------------------------- + +describe('PerfRetention internal tick rejection (D-LC-3)', () => { + it('a non-errno throw during touch is awaitable/rejectable via the scheduler', async () => { + // Internal (programming) error: utimes throws a plain Error (no errno code). + const internalFs: PerfRetentionFilesystem = { + ...realFilesystem(), + async utimes(): Promise { + throw new Error('internal touch corruption'); + }, + }; + const scheduler = new CapturingScheduler(); + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000004', + fs: internalFs, + scheduler, + onDiagnostic: () => {}, + }); + await retention.start(); + + // The interval callback returns a promise that rejects with the internal + // error — it is NOT silently swallowed; awaiting observes the rejection. + await expect(scheduler.callback!()).rejects.toThrow( + 'internal touch corruption', + ); + + await retention.dispose(); + }); + + it('a non-errno throw during maintain is awaitable/rejectable', async () => { + // Internal error inside maintain: readdir throws a plain Error. + const internalFs: PerfRetentionFilesystem = { + ...realFilesystem(), + async readdir(): Promise { + throw new Error('internal readdir corruption'); + }, + }; + const scheduler = new CapturingScheduler(); + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000005', + fs: internalFs, + scheduler, + onDiagnostic: () => {}, + }); + await retention.start(); + + await expect(scheduler.callback!()).rejects.toThrow( + 'internal readdir corruption', + ); + + await retention.dispose(); + }); + + it('dispose still completes (unlinks claim) after a tick rejected', async () => { + const internalFs: PerfRetentionFilesystem = { + ...realFilesystem(), + async utimes(): Promise { + throw new Error('internal touch corruption'); + }, + }; + const scheduler = new CapturingScheduler(); + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000006', + fs: internalFs, + scheduler, + onDiagnostic: () => {}, + }); + await retention.start(); + + await scheduler.callback!().catch(() => {}); + await retention.dispose(); + + // Claim cleanup proceeds despite the prior internal rejection. + expect( + fs.existsSync( + path.join(dir, '00000000-0000-4000-8000-000000000006.claim'), + ), + ).toBe(false); + }); +}); + +// --------------------------------------------------------------------------- +// D-LC-4: dispose propagates in-flight tick internal error after cleanup +// --------------------------------------------------------------------------- + +describe('PerfRetention dispose propagates in-flight tick error (D-LC-4)', () => { + it('rejects with the in-flight tick internal error AND removes the claim', async () => { + let touchResolve!: () => void; + const touchGate = new Promise((resolve) => { + touchResolve = resolve; + }); + const gatedFs: PerfRetentionFilesystem = { + ...realFilesystem(), + async utimes(_p, _atime, _mtime): Promise { + await touchGate; + throw new Error('internal touch corruption'); + }, + }; + const scheduler = new CapturingScheduler(); + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000007', + fs: gatedFs, + scheduler, + onDiagnostic: () => {}, + }); + await retention.start(); + + // Fire the tick — it blocks inside touchClaim on the gate. + const tickPromise = scheduler.callback!().catch(() => {}); + await new Promise((r) => setTimeout(r, 15)); + + // Dispose while the tick is in-flight (blocked at the gate). + const disposePromise = retention.dispose(); + + // Release the gate — the tick throws an internal (non-errno) error. + touchResolve(); + await tickPromise; + + // Dispose must reject with the tick's internal error after cleanup. + await expect(disposePromise).rejects.toThrow('internal touch corruption'); + + // Cleanup must still have proceeded despite the tick error. + expect( + fs.existsSync( + path.join(dir, '00000000-0000-4000-8000-000000000007.claim'), + ), + ).toBe(false); + }); + + it('aggregates tick and cleanup internal errors when both fail (D-LC-4)', async () => { + let touchResolve!: () => void; + const touchGate = new Promise((resolve) => { + touchResolve = resolve; + }); + const dualFailFs: PerfRetentionFilesystem = { + ...realFilesystem(), + async utimes(): Promise { + await touchGate; + throw new Error('tick internal error'); + }, + async unlink(): Promise { + throw new Error('cleanup internal error'); + }, + }; + const scheduler = new CapturingScheduler(); + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000008', + fs: dualFailFs, + scheduler, + onDiagnostic: () => {}, + }); + await retention.start(); + + const tickPromise = scheduler.callback!().catch(() => {}); + await new Promise((r) => setTimeout(r, 15)); + + const disposePromise = retention.dispose(); + touchResolve(); + await tickPromise; + + let caught: unknown = undefined; + try { + await disposePromise; + } catch (err) { + caught = err; + } + expect(caught).toBeInstanceOf(AggregateError); + const aggregate = caught as AggregateError; + const messages = aggregate.errors.map((e) => + e instanceof Error ? e.message : String(e), + ); + expect(messages).toContain('tick internal error'); + expect(messages).toContain('cleanup internal error'); + }); + + it('external errno tick failure resolves fail-open during dispose', async () => { + let touchResolve!: () => void; + const touchGate = new Promise((resolve) => { + touchResolve = resolve; + }); + const errnoFs: PerfRetentionFilesystem = { + ...realFilesystem(), + async utimes(): Promise { + await touchGate; + const err = new Error('EACCES') as NodeJS.ErrnoException; + err.code = 'EACCES'; + throw err; + }, + }; + const scheduler = new CapturingScheduler(); + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000009', + fs: errnoFs, + scheduler, + onDiagnostic: () => {}, + }); + await retention.start(); + + const tickPromise = scheduler.callback!().catch(() => {}); + await new Promise((r) => setTimeout(r, 15)); + + const disposePromise = retention.dispose(); + touchResolve(); + await tickPromise; + + // External errno failure resolves fail-open — dispose resolves. + await expect(disposePromise).resolves.toBeUndefined(); + expect( + fs.existsSync( + path.join(dir, '00000000-0000-4000-8000-000000000009.claim'), + ), + ).toBe(false); + }); +}); + +// --------------------------------------------------------------------------- +// Serialization: maybeMaintain and interval tick use one chain (no overlap) +// --------------------------------------------------------------------------- + +describe('PerfRetention serialization — maybeMaintain and tick never overlap', () => { + it('maybeMaintain chains after an in-flight tick with no overlapping maintain work', async () => { + // Gated filesystem: the first readdir (from the tick's maintain) blocks + // until released. While blocked, maybeMaintain is called — it must chain + // behind the tick rather than overlap. + const gate = createDeferred(); + const readdirEntered = createDeferred(); + let concurrentReaddirs = 0; + let maxConcurrent = 0; + let readdirCount = 0; + let utimesCount = 0; + + const gatedFs: PerfRetentionFilesystem = { + ...realFilesystem(), + async readdir(d: string): Promise { + readdirCount += 1; + concurrentReaddirs += 1; + maxConcurrent = Math.max(maxConcurrent, concurrentReaddirs); + if (readdirCount === 1) { + readdirEntered.resolve(); + await gate.promise; + } + const result = await fsp.readdir(d); + concurrentReaddirs -= 1; + return result; + }, + async utimes(p: string, atime: Date, mtime: Date): Promise { + utimesCount += 1; + await fsp.utimes(p, atime, mtime); + }, + }; + + const scheduler = new CapturingScheduler(); + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000040', + fs: gatedFs, + scheduler, + maintenanceIntervalMs: 1, + maxFiles: 1, + maxBytes: 1, + onDiagnostic: () => {}, + }); + await retention.start(); + + const tickPromise = scheduler.callback!(); + await readdirEntered.promise; + + const maybePromise = retention.maybeMaintain(Date.now() + 10_000); + + gate.resolve(); + await tickPromise; + await maybePromise; + + expect(readdirCount).toBe(2); + + // No overlap: at most 1 concurrent readdir across both maintains. + expect(maxConcurrent).toBe(1); + + // No extra claim touch: only the tick called utimes (touchClaim). + // maybeMaintain calls maintain() — NOT tick() — so it does not touch + // the claim. + expect(utimesCount).toBe(1); + + await retention.dispose(); + }); + + it('explicit maybeMaintain now is preserved for the eviction sweep (not overwritten by a concurrent tick)', async () => { + // Create a perf file with today's UTC day key and a known mtime, belonging + // to a DIFFERENT run UUID (so own-run protection does not apply). The file + // is protected (not evicted) when `now - mtimeMs < maintenanceIntervalMs` + // and eligible when `now - mtimeMs >= maintenanceIntervalMs`. Proving the + // file survives at one `now` and is evicted at a later `now` demonstrates + // that maybeMaintain passes its explicit `now` to the sweep. + const today = new Date(); + const dayKey = `${today.getUTCFullYear()}${String(today.getUTCMonth() + 1).padStart(2, '0')}${String(today.getUTCDate()).padStart(2, '0')}`; + const fileName = `perf-${dayKey}-aaaaaaaa-bbbb-4ccc-8ddd-eeeeeeeeeeee.jsonl`; + const filePath = path.join(dir, fileName); + + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000041', + maxFiles: 1, + maxBytes: 1, + maintenanceIntervalMs: 60_000, + onDiagnostic: () => {}, + }); + + // File mtime is exactly 1000 ms before baseNow. + const baseNow = Date.now(); + const fileMtime = baseNow - 1000; + fs.writeFileSync(filePath, 'data' + '\n'); + fs.utimesSync(filePath, new Date(fileMtime), new Date(fileMtime)); + + // maybeMaintain(baseNow): now - mtimeMs = 1000 < 60_000 → file protected. + await retention.maybeMaintain(baseNow); + expect(fs.existsSync(filePath)).toBe(true); + + // maybeMaintain(baseNow + 120_000): now - mtimeMs = 121_000 >= 60_000 → + // file eligible → evicted. The rate-limit also passes: 120_000 >= 60_000. + await retention.maybeMaintain(baseNow + 120_000); + expect(fs.existsSync(filePath)).toBe(false); + + await retention.dispose(); + }); +}); diff --git a/packages/telemetry/src/perf/retention.multiowner.behavior.test.ts b/packages/telemetry/src/perf/retention.multiowner.behavior.test.ts new file mode 100644 index 0000000000..713c893963 --- /dev/null +++ b/packages/telemetry/src/perf/retention.multiowner.behavior.test.ts @@ -0,0 +1,190 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Real multi-owner convergence evidence for PerfRetention (AC-7, D3, §6). + * + * Automatic retention protects a JSONL ONLY as a genuinely-live writer: + * current UTC day-key AND mtime within the maintenance interval. A fresh + * claim protects the claim FILE itself (and counts toward caps) but does NOT + * shield that run's historical JSONL — otherwise a long-running process could + * never evict its own old-day files and converge to the eventual byte/file + * caps. This deliberately differs from explicit /perf delete, which keeps + * claim→JSONL protection to avoid unlinking a file another active process may + * still be appending. + * + * No mocks — real files, real filesystem, distinct standard UUIDs per run. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { PerfRetention } from './retention.js'; +import { utcDayKey } from './perfArtifacts.js'; + +const ONE_MIB = 1_048_576; + +let dir: string; + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'perf-multiowner-')); +}); + +afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); +}); + +function writeExactFile(name: string, byteLength: number): void { + fs.writeFileSync(path.join(dir, name), 'x'.repeat(byteLength)); +} + +function createClaimFile(uuid: string, mtimeMs: number): void { + const p = path.join(dir, `${uuid}.claim`); + fs.writeFileSync(p, '', { mode: 0o600 }); + fs.utimesSync(p, new Date(mtimeMs), new Date(mtimeMs)); +} + +function setMtime(name: string, mtimeMs: number): void { + const p = path.join(dir, name); + fs.utimesSync(p, new Date(mtimeMs), new Date(mtimeMs)); +} + +function listFiles(): string[] { + return fs.readdirSync(dir).sort(); +} + +function exists(name: string): boolean { + return fs.existsSync(path.join(dir, name)); +} + +describe('PerfRetention multi-owner convergence (AC-7, D3, §6)', () => { + it('a 24x7 run evicts its own old-day files until BOTH caps are met while its current live file and fresh claim survive', async () => { + // Fixed, deterministic "now" — no wall-clock dependence. + const now = Date.parse('2026-08-10T12:00:00.000Z'); + const todayKey = utcDayKey(now); + const ownerUuid = '00000000-0000-4000-8000-000000000100'; + + // Multiple OLD-DAY JSONL files from the SAME owner run. Each is exactly + // 1 MiB and belongs to ownerUuid, on a distinct prior UTC day. + const oldNames: string[] = []; + for (let i = 0; i < 5; i++) { + const day = new Date(now - (i + 1) * 86_400_000); + const dayKey = utcDayKey(day.getTime()); + const name = `perf-${dayKey}-${ownerUuid}.jsonl`; + writeExactFile(name, ONE_MIB); + setMtime(name, now - (i + 1) * 86_400_000); + oldNames.push(name); + } + + // One CURRENT-DAY live JSONL file from the same owner — recent mtime. + const liveName = `perf-${todayKey}-${ownerUuid}.jsonl`; + writeExactFile(liveName, ONE_MIB); + setMtime(liveName, now - 5_000); + + // Start over BOTH caps: 5 old files + 1 live file + the owner claim = 7 + // artifacts (~6 MiB). maxFiles 2 and maxBytes 2 MiB are both exceeded. + const retention = new PerfRetention({ + dir, + runUuid: ownerUuid, + maxFiles: 2, + maxBytes: 2 * ONE_MIB, + }); + await retention.start(); + await retention.maintain(now); + + // The fresh claim survives (non-stale; counts toward caps). + expect(exists(`${ownerUuid}.claim`)).toBe(true); + // The current-day live file survives (genuine live writer). + expect(exists(liveName)).toBe(true); + // Every old-day file is evicted — the run's own historical files are NOT + // shielded by the fresh claim or by being the owner run. + for (const name of oldNames) { + expect(exists(name)).toBe(false); + } + + // Exact survivor set: only the protected claim + live writer remain, and + // BOTH caps are satisfied (2 files ≤ 2, 1 MiB ≤ 2 MiB). + expect(listFiles()).toEqual([`${ownerUuid}.claim`, liveName]); + expect(retention.evictionCount).toBe(5); + + await retention.dispose(); + }); + + it('a fresh claim does NOT protect its run old-day JSONL from retention eviction', async () => { + const now = Date.parse('2026-08-10T12:00:00.000Z'); + const ownerUuid = '00000000-0000-4000-8000-000000000110'; + const otherUuid = '00000000-0000-4000-8000-000000000111'; + const yesterdayKey = utcDayKey(now - 86_400_000); + + // Old-day JSONL belonging to otherUuid — would be eligible on its own. + const otherOldName = `perf-${yesterdayKey}-${otherUuid}.jsonl`; + writeExactFile(otherOldName, 1024); + setMtime(otherOldName, now - 86_400_000); + + // Fresh claim for otherUuid. Automatic retention must NOT propagate this + // to shield otherUuid's old-day JSONL (that is delete-only behavior). + createClaimFile(otherUuid, now - 10_000); + + const retention = new PerfRetention({ + dir, + runUuid: ownerUuid, + maxFiles: 1, + maxBytes: 10_000_000, + }); + await retention.start(); + await retention.maintain(now); + + // The fresh claim survives (non-stale; counts toward caps). + expect(exists(`${otherUuid}.claim`)).toBe(true); + expect(exists(`${ownerUuid}.claim`)).toBe(true); + // The old-day JSONL is evicted despite the fresh claim on its run. + expect(exists(otherOldName)).toBe(false); + + await retention.dispose(); + }); + + it('the owner current-day live file survives while its own old-day files converge', async () => { + const now = Date.parse('2026-08-10T12:00:00.000Z'); + const todayKey = utcDayKey(now); + const yesterdayKey = utcDayKey(now - 86_400_000); + const ownerUuid = '00000000-0000-4000-8000-000000000120'; + + // Current-day live file — protected (live writer). + const liveName = `perf-${todayKey}-${ownerUuid}.jsonl`; + writeExactFile(liveName, 4096); + setMtime(liveName, now - 5_000); + + // The owner's OWN old-day file — eligible (not a live writer). + const ownerOldName = `perf-${yesterdayKey}-${ownerUuid}.jsonl`; + writeExactFile(ownerOldName, 4096); + setMtime(ownerOldName, now - 86_400_000); + + // An unaffiliated, older file from a run with no claim — eligible. + const loneName = `perf-${utcDayKey(now - 2 * 86_400_000)}-00000000-0000-4000-8000-000000000121.jsonl`; + writeExactFile(loneName, 4096); + setMtime(loneName, now - 2 * 86_400_000); + + const retention = new PerfRetention({ + dir, + runUuid: ownerUuid, + maxFiles: 1, + maxBytes: 10_000_000, + }); + await retention.start(); + await retention.maintain(now); + + // The lone file and the owner's old-day file are evicted; only the + // current-day live file and the fresh claim survive. + expect(exists(loneName)).toBe(false); + expect(exists(ownerOldName)).toBe(false); + expect(exists(liveName)).toBe(true); + expect(exists(`${ownerUuid}.claim`)).toBe(true); + expect(listFiles()).toEqual([`${ownerUuid}.claim`, liveName]); + + await retention.dispose(); + }); +}); diff --git a/packages/telemetry/src/perf/retention.scheduling.behavior.test.ts b/packages/telemetry/src/perf/retention.scheduling.behavior.test.ts new file mode 100644 index 0000000000..60b63b059a --- /dev/null +++ b/packages/telemetry/src/perf/retention.scheduling.behavior.test.ts @@ -0,0 +1,207 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Scheduling and rate-limiting behavioral tests for PerfRetention (AC-7, D3). + * + * Covers the interval/rate-limit behavior split out of the original + * retention.behavior.test.ts: the single coarse maintenance interval that + * both touches the claim and sweeps old files (AC-7, D3) and the + * maybeMaintain rate-limiting gate. + * + * Real files, real filesystem, no mocks. The interval callback is captured + * via a deterministic TestScheduler. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { + PerfRetention, + type PerfScheduler, + type PerfTimerHandle, +} from './retention.js'; + +let dir: string; + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'perf-retention-')); +}); + +afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); +}); + +function writePerfFile( + name: string, + recordCount: number, + recordBytes = 1220, +): void { + const lines: string[] = []; + for (let i = 0; i < recordCount; i++) { + const padding = '.'.repeat( + Math.max(0, recordBytes - 80 - String(i).length), + ); + lines.push( + JSON.stringify({ + schema_version: 1, + record_type: 'operation', + ts: '2026-08-08T12:00:00.000Z', + pad: padding, + idx: i, + }), + ); + } + fs.writeFileSync(path.join(dir, name), lines.join('\n') + '\n'); +} + +function setMtime(name: string, mtimeMs: number): void { + const p = path.join(dir, name); + fs.utimesSync(p, new Date(mtimeMs), new Date(mtimeMs)); +} + +function listFiles(): string[] { + return fs.readdirSync(dir).sort(); +} + +class TestScheduler implements PerfScheduler { + callback: (() => Promise) | null = null; + handle: PerfTimerHandle | null = null; + + setInterval(callback: () => Promise, _ms: number): PerfTimerHandle { + this.callback = callback; + this.handle = { unref: () => {}, clear: () => {} }; + return this.handle; + } +} + +describe('PerfRetention one coarse interval (AC-7, D3)', () => { + it('the same interval touches the claim and sweeps old files', async () => { + const scheduler = new TestScheduler(); + const now = Date.now(); + const oldNow = now - 3_600_000; + + writePerfFile('perf-20260101-old.jsonl', 3); + setMtime('perf-20260101-old.jsonl', oldNow); + writePerfFile('perf-20260102-older.jsonl', 3); + setMtime('perf-20260102-older.jsonl', oldNow - 3_600_000); + + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000016', + scheduler, + maxFiles: 1, + maxBytes: 10_000_000, + }); + await retention.start(); + + expect(listFiles()).toContain('00000000-0000-4000-8000-000000000016.claim'); + + const beforeMtime = fs.statSync( + path.join(dir, '00000000-0000-4000-8000-000000000016.claim'), + ).mtimeMs; + + await new Promise((r) => setTimeout(r, 20)); + + await scheduler.callback!(); + + expect(fs.existsSync(path.join(dir, 'perf-20260101-old.jsonl'))).toBe( + false, + ); + + const afterMtime = fs.statSync( + path.join(dir, '00000000-0000-4000-8000-000000000016.claim'), + ).mtimeMs; + expect(afterMtime).toBeGreaterThan(beforeMtime); + + await retention.dispose(); + }); + + it('fires via the actual owned interval without restart', async () => { + const scheduler = new TestScheduler(); + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000017', + scheduler, + }); + await retention.start(); + + expect(scheduler.callback).not.toBeNull(); + + await scheduler.callback!(); + await scheduler.callback!(); + await scheduler.callback!(); + + expect(listFiles()).toContain('00000000-0000-4000-8000-000000000017.claim'); + await retention.dispose(); + }); +}); + +describe('PerfRetention maybeMaintain rate-limiting', () => { + it('runs maintain on first call', async () => { + const now = Date.now(); + const evictableName = + 'perf-20260101-11111111-1111-4111-8111-111111111118.jsonl'; + writePerfFile(evictableName, 3); + setMtime(evictableName, now - 86_400_000); + + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000018', + maxFiles: 1, + maxBytes: 1, + onDiagnostic: () => {}, + }); + await retention.maybeMaintain(now); + + expect(fs.existsSync(path.join(dir, evictableName))).toBe(false); + }); + + it('skips when called within the maintenance interval', async () => { + const now = Date.now(); + const evictableName = + 'perf-20260101-11111111-1111-4111-8111-111111111119.jsonl'; + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000019', + maxFiles: 1, + maxBytes: 1, + maintenanceIntervalMs: 60_000, + onDiagnostic: () => {}, + }); + + await retention.maybeMaintain(now); + + writePerfFile(evictableName, 3); + setMtime(evictableName, now - 86_400_000); + await retention.maybeMaintain(now + 1_000); + + expect(fs.existsSync(path.join(dir, evictableName))).toBe(true); + expect(retention.evictionCount).toBe(0); + }); + + it('runs again after the maintenance interval elapses', async () => { + const now = Date.now(); + const evictableName = + 'perf-20260101-11111111-1111-4111-8111-11111111111a.jsonl'; + const retention = new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-00000000001a', + maxFiles: 1, + maxBytes: 1, + maintenanceIntervalMs: 60_000, + onDiagnostic: () => {}, + }); + await retention.maybeMaintain(now); + + writePerfFile(evictableName, 3); + setMtime(evictableName, now - 86_400_000); + + await retention.maybeMaintain(now + 61_000); + expect(fs.existsSync(path.join(dir, evictableName))).toBe(false); + }); +}); diff --git a/packages/telemetry/src/perf/retention.ts b/packages/telemetry/src/perf/retention.ts new file mode 100644 index 0000000000..5a0983f235 --- /dev/null +++ b/packages/telemetry/src/perf/retention.ts @@ -0,0 +1,827 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * PerfRetention — directory retention + per-run claim lifecycle (P08, D3/D5/D6). + * + * A constructible (non-singleton) owner that manages exactly ONE coarse + * maintenance interval. That same interval: + * 1. Touches this run's UUID claim file (lease-window freshness — D3). + * 2. Performs an oldest-first retention sweep over perf-*.jsonl + *.claim. + * + * Claim lifecycle: + * - Created exclusively (`wx`, 0600) at {@link PerfRetention.start}. + * - Touched every interval by {@link PerfRetention.tick}. + * - Removed on clean {@link PerfRetention.dispose}. + * - A crash (no dispose) leaves a stale claim until the next sweep reaps it. + * + * Retention is an EVENTUAL BOUND with documented overshoot + live-writer + * safety (AC-7, §6). It is explicitly NOT an instantaneous no-loss cap. + * The bound permits active-day and claim overshoot (D3/D5). + * + * Error policy (D8): internal/programming errors fail fast. Only genuine + * filesystem persistence/maintenance errors (create/touch/stat/readdir/unlink) + * fail open and are rate-limited. + * + * Filesystem fault injection (D6): tests inject {@link FaultInjectingRetentionFilesystem} + * to produce deterministic EACCES/EROFS/ENOSPC at any boundary — never real-disk + * fill or chmod. + */ + +import { promises as fsp } from 'node:fs'; +import { join } from 'node:path'; +import { + isOwnedArtifact, + isNonStaleClaim, + isClaimFile, + isPerfJsonl, + isLiveWriterFile, + requireValidRunUuid, +} from './perfArtifacts.js'; + +// =========================================================================== +// Constants (D5 — derived from the P04 Bun record-size benchmark) +// =========================================================================== +// +// P04 benchmark output (perfRecordSize.bench.ts, actual v1 schema): +// operation record (WITH memory columns): 1220 bytes/line +// memory_sample record: 242 bytes/line +// combined per-operation pair: 1462 bytes +// +// MAX_BYTES = 64 MiB = 67,108,864 bytes +// At 1462 bytes/pair (memory on), the cap holds ~45,902 operation pairs. +// At 1220 bytes/op (memory off), it holds ~55,008 operations. +// This is a generous budget for LOCAL-ONLY dev telemetry — no network, +// no remote upload. +// +// MAX_FILES = 128 +// One file per writer per UTC day (perf-YYYYMMDD-uuid.jsonl) + one 0-byte +// claim per concurrent run. At single-writer volume, 128 files ≈ 128 days. +// Each claim file is 0 bytes but counts toward the file cap. +// +// Which cap binds at representative single-writer volume? +// Crossover: MAX_BYTES / MAX_FILES = 524,288 bytes/file ≈ 512 KiB. +// 524,288 / 1462 ≈ 359 operation-pairs/file (memory on). +// Below ~359 pairs/day: the FILE cap binds (128 days of data before eviction). +// Above ~359 pairs/day: the BYTE cap binds (64 MiB reached before 128 days). +// At typical interactive use (~50-200 ops/day), the file cap is the binding +// constraint — generous for local-only retention. +// +// MAINTENANCE_INTERVAL_MS = 60,000 (60 s) +// The single owned coarse interval. Also defines the live-writer protection +// window: a perf file with today's day-key whose mtime is within this window +// is never evicted. +// +// CLAIM_LEASE_MS = 180,000 (180 s = 3 × interval) +// A claim is non-stale while now - mtime ≤ CLAIM_LEASE_MS. The interval +// touches the claim every 60 s, so a running process keeps it fresh well +// within the 180 s lease. A crashed run's claim becomes stale within +// 3 minutes — bounded crash overshoot (D3). +// +// DIAG_RATE_LIMIT_MS = 60,000 (60 s) +// At most one diagnostic per window for retention filesystem failures. + +export const PERF_MAX_BYTES = 64 * 1024 * 1024; // 67,108,864 +export const PERF_MAX_FILES = 128; +export const PERF_MAINTENANCE_INTERVAL_MS = 60_000; +export const PERF_CLAIM_LEASE_MS = PERF_MAINTENANCE_INTERVAL_MS * 3; // 180,000 +export const PERF_DIAG_RATE_LIMIT_MS = 60_000; + +// =========================================================================== +// Filesystem port (D6 — package-private for deterministic fault injection) +// =========================================================================== + +/** + * Narrow filesystem port used by PerfRetention. The default implementation + * uses real `node:fs/promises`; tests inject + * {@link FaultInjectingRetentionFilesystem} to produce deterministic + * EACCES/EROFS/ENOSPC failures at the unlink/touch/stat/readdir boundary + * without filling a disk or relying on chmod. + */ +export interface PerfRetentionFilesystem { + ensureDir(dir: string): Promise; + openExclusive(path: string, mode: number): Promise; + utimes(path: string, atime: Date, mtime: Date): Promise; + readdir(dir: string): Promise; + stat(path: string): Promise<{ size: number; mtimeMs: number }>; + unlink(path: string): Promise; +} + +/** Default filesystem port using real `node:fs/promises`. */ +class RealRetentionFilesystem implements PerfRetentionFilesystem { + async ensureDir(dir: string): Promise { + try { + await fsp.access(dir); + } catch { + await fsp.mkdir(dir, { recursive: true, mode: 0o700 }); + } + } + + async openExclusive(filePath: string, mode: number): Promise { + const handle = await fsp.open(filePath, 'wx', mode); + await handle.close(); + } + + async utimes(filePath: string, atime: Date, mtime: Date): Promise { + await fsp.utimes(filePath, atime, mtime); + } + + async readdir(dir: string): Promise { + return fsp.readdir(dir); + } + + async stat(filePath: string): Promise<{ size: number; mtimeMs: number }> { + const s = await fsp.stat(filePath); + return { size: s.size, mtimeMs: s.mtimeMs }; + } + + async unlink(filePath: string): Promise { + await fsp.unlink(filePath); + } +} + +/** + * Deterministic fault-injecting filesystem port for retention tests (D6). + * Fails the configured method with the given errno code on every call, + * delegating all other methods to the real implementation. Package-private + * — tests deep-import it; it is NOT in the public barrel. + */ +export class FaultInjectingRetentionFilesystem + implements PerfRetentionFilesystem +{ + private readonly real = new RealRetentionFilesystem(); + + constructor( + private readonly fault: { + readonly failMethod: + | 'unlink' + | 'utimes' + | 'openExclusive' + | 'readdir' + | 'stat' + | 'ensureDir'; + readonly code: 'EACCES' | 'EROFS' | 'ENOSPC' | 'ENOENT'; + }, + ) {} + + async ensureDir(dir: string): Promise { + if (this.fault.failMethod === 'ensureDir') throw this.makeError(); + await this.real.ensureDir(dir); + } + + async openExclusive(filePath: string, mode: number): Promise { + if (this.fault.failMethod === 'openExclusive') throw this.makeError(); + await this.real.openExclusive(filePath, mode); + } + + async utimes(filePath: string, atime: Date, mtime: Date): Promise { + if (this.fault.failMethod === 'utimes') throw this.makeError(); + await this.real.utimes(filePath, atime, mtime); + } + + async readdir(dir: string): Promise { + if (this.fault.failMethod === 'readdir') throw this.makeError(); + return this.real.readdir(dir); + } + + async stat(filePath: string): Promise<{ size: number; mtimeMs: number }> { + if (this.fault.failMethod === 'stat') throw this.makeError(); + return this.real.stat(filePath); + } + + async unlink(filePath: string): Promise { + if (this.fault.failMethod === 'unlink') throw this.makeError(); + await this.real.unlink(filePath); + } + + private makeError(): NodeJS.ErrnoException { + const err = new Error( + `fault-injected ${this.fault.code}`, + ) as NodeJS.ErrnoException; + err.code = this.fault.code; + return err; + } +} + +// =========================================================================== +// Scheduler port (package-private — for deterministic test firing) +// =========================================================================== + +/** + * Handle for an owned interval timer. The `unref` method prevents the timer + * from keeping the CLI process alive (supported on Node.js/Bun). The `clear` + * method cancels the interval so it stops invoking the callback — disposal + * MUST call this rather than merely nulling the handle, otherwise the native + * interval keeps firing a disposed callback forever. + */ +export interface PerfTimerHandle { + unref(): void; + clear(): void; +} + +/** + * Package-private scheduler seam. The default implementation delegates to + * `setInterval`. Tests inject a custom implementation that captures the + * callback so it can be fired deterministically, then asserts file/mtime/ + * outcome behavior — not just callback invocation. + * + * The callback returns a Promise so test schedulers can `await` the async + * work (touch + maintain) before asserting state. + */ +export interface PerfScheduler { + setInterval(callback: () => Promise, ms: number): PerfTimerHandle; +} + +class RealScheduler implements PerfScheduler { + setInterval(callback: () => Promise, ms: number): PerfTimerHandle { + // Wrap the async callback so that an internal (non-errno) rejection is + // surfaced via an asynchronous rethrow (D8 fail-fast) rather than becoming + // an opaque unhandled rejection. External fs errors are caught inside + // tick()/maintain() (fail-open); only true programming errors reach here. + const fire = (): void => { + Promise.resolve(callback()).catch((err: unknown) => { + queueMicrotask(() => { + throw err; + }); + }); + }; + const nativeHandle = globalThis.setInterval(fire, ms); + return { + unref: () => { + const h = nativeHandle as unknown as { + unref?: () => void; + }; + if (typeof h.unref === 'function') h.unref(); + }, + clear: () => { + globalThis.clearInterval(nativeHandle); + }, + }; + } +} + +// =========================================================================== +// Helpers +// =========================================================================== + +/** + * Determines whether an error carries a Node.js errno code, indicating a + * filesystem persistence failure (fail-open). Errors without an errno code + * are programming errors and must propagate (fail fast). + */ +function isErrnoError(err: unknown): boolean { + if (!(err instanceof Error)) return false; + return typeof (err as NodeJS.ErrnoException).code === 'string'; +} + +// =========================================================================== +// PerfRetention +// =========================================================================== + +export interface PerfRetentionOptions { + readonly dir: string; + readonly runUuid: string; + readonly fs?: PerfRetentionFilesystem; + readonly scheduler?: PerfScheduler; + readonly maxBytes?: number; + readonly maxFiles?: number; + readonly maintenanceIntervalMs?: number; + readonly claimLeaseMs?: number; + readonly diagRateLimitMs?: number; + readonly onDiagnostic?: (message: string) => void; +} + +interface ArtifactInfo { + readonly name: string; + readonly fullPath: string; + readonly size: number; + readonly mtimeMs: number; +} + +/** + * Constructible retention owner. Owns exactly one coarse maintenance interval + * that touches this run's claim and sweeps old artifacts. + * + * Lifecycle: + * - `await start()` — creates the claim exclusively, starts the interval. + * - `await tick()` — interval body: touch claim + maintain (fire-and-forget + * in production, awaitable for deterministic tests). + * - `await dispose()` — clears the interval, removes the claim cleanly. + * + * A crash (no dispose) leaves a stale claim on disk; the next sweep reaps it. + */ +export class PerfRetention { + private readonly sinkDir: string; + private readonly runUuid: string; + private readonly fsPort: PerfRetentionFilesystem; + private readonly scheduler: PerfScheduler; + private readonly maxBytes: number; + private readonly maxFiles: number; + private readonly maintenanceIntervalMs: number; + private readonly claimLeaseMs: number; + private readonly diagRateLimitMs: number; + private readonly onDiagnostic: (message: string) => void; + + private timerHandle: PerfTimerHandle | null = null; + private claimPath: string | null = null; + private lastMaintenanceMs = 0; + private lastDiagMs = 0; + private started = false; + private disposed = false; + // P11 self-health: cumulative count of successful evictions in THIS process. + // Narrow read-only state for the inspect/report self-health surface — NOT + // persisted. + private evictions = 0; + // The latest accepted interval tick promise (serialized chain). Dispose + // awaits this so an in-flight touch+sweep completes before the claim is + // removed — preventing a touch-after-unlink race. + private inflight: Promise | null = null; + + constructor(options: PerfRetentionOptions) { + this.sinkDir = options.dir; + this.runUuid = requireValidRunUuid(options.runUuid); + this.fsPort = options.fs ?? new RealRetentionFilesystem(); + this.scheduler = options.scheduler ?? new RealScheduler(); + this.maxBytes = options.maxBytes ?? PERF_MAX_BYTES; + this.maxFiles = options.maxFiles ?? PERF_MAX_FILES; + this.maintenanceIntervalMs = + options.maintenanceIntervalMs ?? PERF_MAINTENANCE_INTERVAL_MS; + this.claimLeaseMs = options.claimLeaseMs ?? PERF_CLAIM_LEASE_MS; + this.diagRateLimitMs = options.diagRateLimitMs ?? PERF_DIAG_RATE_LIMIT_MS; + this.onDiagnostic = options.onDiagnostic ?? defaultDiagnostic; + this.validateTuning(); + } + + /** + * Fails fast on misconfigured caps/intervals so internal misuse cannot cause + * catastrophic eviction (e.g. NaN/negative caps make every `<=` comparison + * false and delete everything eligible). + */ + private validateTuning(): void { + this.requirePositiveFinite('maxBytes', this.maxBytes); + this.requirePositiveFinite('maxFiles', this.maxFiles); + this.requirePositiveFinite( + 'maintenanceIntervalMs', + this.maintenanceIntervalMs, + ); + this.requirePositiveFinite('claimLeaseMs', this.claimLeaseMs); + if (!Number.isFinite(this.diagRateLimitMs) || this.diagRateLimitMs < 0) { + throw new RangeError( + `PerfRetention: diagRateLimitMs must be a finite nonnegative number (got ${this.diagRateLimitMs})`, + ); + } + } + + private requirePositiveFinite(name: string, value: number): void { + if (!Number.isFinite(value) || value <= 0) { + throw new RangeError( + `PerfRetention: ${name} must be a finite positive number (got ${value})`, + ); + } + } + + /** + * Creates the UUID claim file exclusively and starts the one owned + * maintenance interval. Must be called exactly once before tick/dispose. + * An empty started retention creates ONLY its claim — no perf JSONL. + * + * Only marks `started` after successful claim creation, so an external + * filesystem failure leaves the instance in a truthful, retryable state + * (the next `start()` re-attempts). Phantom claim state is cleared on + * external open failure so dispose/tick do not operate on a non-existent + * file. External errno errors remain fail-open; internal errors propagate. + */ + async start(): Promise { + if (this.started || this.disposed) return; + + this.claimPath = join(this.sinkDir, `${this.runUuid}.claim`); + + try { + await this.fsPort.ensureDir(this.sinkDir); + await this.fsPort.openExclusive(this.claimPath, 0o600); + } catch (err) { + if (isErrnoError(err)) { + this.emitDiagnostic(err); + this.claimPath = null; + return; + } + throw err; + } + + try { + this.timerHandle = this.scheduler.setInterval( + () => this.fireTick(), + this.maintenanceIntervalMs, + ); + this.timerHandle.unref(); + this.started = true; + } catch (error) { + await this.rollbackFailedStart(error); + } + } + + private async rollbackFailedStart(startError: unknown): Promise { + const errors: unknown[] = [startError]; + const timerHandle = this.timerHandle; + this.timerHandle = null; + this.started = false; + if (timerHandle !== null) { + try { + timerHandle.clear(); + } catch (error) { + errors.push(error); + } + } + + const claimPath = this.claimPath; + this.claimPath = null; + if (claimPath !== null) { + try { + await this.fsPort.unlink(claimPath); + } catch (error) { + if (isErrnoError(error)) { + this.emitDiagnostic(error); + } else { + errors.push(error); + } + } + } + + if (errors.length > 1) { + throw new AggregateError( + errors, + 'PerfRetention start and rollback both failed', + ); + } + throw startError; + } + + /** + * Serializes a unit of work onto the single tracked chain so interval ticks + * and triggered maintenance never overlap. Each unit runs regardless of + * whether the prior unit resolved or rejected (a prior internal error is + * surfaced independently by the real-scheduler rejection path). Returns the + * unit's promise so callers can await it. + */ + private serializeWork(work: () => Promise): Promise { + if (this.disposed) return Promise.resolve(); + const prev = this.inflight; + const next: Promise = prev === null ? work() : prev.then(work, work); + this.inflight = next; + next.then( + () => { + if (this.inflight === next) this.inflight = null; + }, + () => { + if (this.inflight === next) this.inflight = null; + }, + ); + return next; + } + + /** + * Interval entry point. Serializes tick work onto a single tracked chain so + * ticks never overlap and {@link dispose} can deterministically await any + * in-flight maintenance before removing the claim. Each firing returns its + * own promise so deterministic test schedulers can await/reject it. + */ + private fireTick(): Promise { + return this.serializeWork(() => this.tick()); + } + + /** + * The interval body: touches this run's claim and runs a retention sweep. + * Called automatically by the owned interval. Exposed as public so + * deterministic tests can fire the actual interval callback and assert + * file/mtime/outcome behavior. + */ + async tick(): Promise { + if (this.disposed) return; + const now = Date.now(); + await this.touchClaim(now); + await this.maintain(now); + } + + /** + * Rate-limited maintenance trigger (called on roll boundary by PerfSink). + * Skips if called within the maintenance interval of the last sweep. + * + * Serialized onto the SAME tracked chain as interval ticks (via + * {@link serializeWork}) so a triggered sweep can never overlap a concurrent + * interval tick. The explicit `now` is preserved for the actual sweep — no + * invented extra claim touch occurs. + */ + async maybeMaintain(now: number): Promise { + if (this.disposed) return; + if (now - this.lastMaintenanceMs < this.maintenanceIntervalMs) return; + this.lastMaintenanceMs = now; + await this.serializeWork(() => this.maintain(now)); + } + + /** + * Scans owned artifacts (perf-YYYYMMDD-*.jsonl + *.claim), computes total + * bytes/files, and evicts oldest-first until BOTH caps are satisfied. + * + * Live-writer protection: a perf file whose day-key is today UTC AND whose + * mtime is within the maintenance interval is never deleted. A non-stale + * claim is never deleted. Stale claims and old-day files are eligible. + * + * Accounting is decremented ONLY on successful unlink (D6 — does not copy + * rotateReports' decrement-on-failure defect). + * + * Genuine filesystem errors (stat/readdir/unlink) fail open and are + * rate-limited. Internal/programming errors fail fast. + */ + async maintain(now: number): Promise { + if (this.disposed) return; + this.lastMaintenanceMs = now; + + let names: string[]; + try { + names = await this.fsPort.readdir(this.sinkDir); + } catch (err) { + if (isErrnoError(err)) { + this.emitDiagnostic(err); + return; + } + throw err; + } + + const owned = names.filter(isOwnedArtifact); + if (owned.length === 0) return; + + // Stat all owned artifacts. Individual stat failures fail open (the + // file is skipped, not counted). + const artifacts: ArtifactInfo[] = []; + for (const name of owned) { + const fullPath = join(this.sinkDir, name); + const statResult = await this.safeStat(fullPath); + if (statResult !== null) { + artifacts.push({ name, fullPath, ...statResult }); + } + } + + let filesLeft = artifacts.length; + let bytesLeft = artifacts.reduce((sum, a) => sum + a.size, 0); + + if (filesLeft <= this.maxFiles && bytesLeft <= this.maxBytes) return; + + // Sort oldest-first with stable deterministic tie-break by name. + // Pre-filter to eligible (non-protected) artifacts to keep the eviction + // loop simple (single break for cap check — no nested continues). + // + // Automatic retention protects a JSONL only as a genuinely-live writer + // (today's UTC day-key AND mtime within the maintenance interval). A fresh + // claim protects the claim FILE itself (and counts toward caps) but does + // NOT shield that run's historical JSONL — otherwise a long-running + // process could never converge to the eventual byte/file caps. Explicit + // /perf delete intentionally keeps claim→JSONL protection + // (perfArtifacts.isPerfJsonlProtected) to avoid unlinking a file another + // active process may still be appending. + const sorted = artifacts + .filter((a) => !this.isProtected(a, now)) + .sort(compareArtifactAge); + + for (const artifact of sorted) { + if (filesLeft <= this.maxFiles && bytesLeft <= this.maxBytes) break; + + const unlinked = await this.safeUnlink(artifact.fullPath); + if (unlinked) { + // Decrement ONLY on successful unlink. + filesLeft -= 1; + bytesLeft -= artifact.size; + this.evictions += 1; + } + } + } + + /** + * Counts non-stale claim files for concurrent_instances (D3). + * A claim is non-stale while (now - mtime) ≤ CLAIM_LEASE_MS. + * + * Genuine filesystem errors (readdir/stat) fail open (return 0 / skip) but + * emit the same rate-limited diagnostic as other maintenance paths. ENOENT + * races count as external fs and are rate-limited. + */ + async countNonStaleClaims(now: number): Promise { + let names: string[]; + try { + names = await this.fsPort.readdir(this.sinkDir); + } catch (err) { + if (isErrnoError(err)) { + this.emitDiagnostic(err); + return 0; + } + throw err; + } + + const claimNames = names.filter((n) => n.endsWith('.claim')); + let count = 0; + for (const name of claimNames) { + try { + const s = await this.fsPort.stat(join(this.sinkDir, name)); + if (now - s.mtimeMs <= this.claimLeaseMs) { + count += 1; + } + } catch (err) { + if (isErrnoError(err)) { + this.emitDiagnostic(err); // skip unreadable claim, fail open + continue; + } + throw err; + } + } + return count; + } + + /** + * P11 self-health: cumulative count of successful evictions in THIS process. + * Narrow read-only state for the inspect/report self-health surface. Not + * persisted. + */ + get evictionCount(): number { + return this.evictions; + } + + /** + * Stops new scheduling, cancels the interval, awaits all accepted/running + * interval tick work, then removes the claim file cleanly. After dispose, + * no further ticks fire and the claim is gone. A crash (no dispose) leaves + * the claim stale until the next sweep. + * + * If the awaited in-flight tick rejected with an internal/programming error + * (non-errno), dispose rejects with that error AFTER cleanup has proceeded. + * If claim cleanup also has an internal failure, both errors are surfaced + * via an AggregateError (project convention for dual-failure scenarios). + * External errno failures from either tick or cleanup remain fail-open / + * rate-limited and do NOT cause dispose to reject. + */ + async dispose(): Promise { + if (this.disposed) return; + // 1. Stop accepting new interval ticks. + this.disposed = true; + // 2. Cancel the timer so the native interval stops firing. + this.timerHandle?.clear(); + this.timerHandle = null; + + // 3. Await any in-flight touch+sweep so it completes before the claim is + // removed (prevents a touch-after-unlink race). External errno errors + // are swallowed (fail-open); internal errors are captured for rethrow + // after cleanup. + let tickError: unknown = null; + if (this.inflight !== null) { + try { + await this.inflight; + } catch (err) { + if (isErrnoError(err)) { + this.emitDiagnostic(err); + } else { + tickError = err; + } + } + } + + // 4. Remove the claim. Cleanup ALWAYS proceeds (try/finally pattern) + // even when the tick rejected — the primary error is surfaced after. + let cleanupError: unknown = null; + if (this.claimPath !== null) { + try { + await this.fsPort.unlink(this.claimPath); + } catch (err) { + if (isErrnoError(err)) { + this.emitDiagnostic(err); + } else { + cleanupError = err; + } + } + } + + // 5. Surface internal errors (fail-fast). If both the tick and cleanup + // failed internally, aggregate both so neither is silently discarded. + if (tickError !== null && cleanupError !== null) { + throw new AggregateError( + [tickError, cleanupError], + 'PerfRetention dispose: in-flight tick and claim cleanup both failed', + ); + } + if (tickError !== null) throw tickError; + if (cleanupError !== null) throw cleanupError; + } + + // ----------------------------------------------------------------------- + // Private + /** + * Attempts to stat a file. Returns null on filesystem errors (fail open, + * rate-limited diagnostic emitted); rethrows internal/programming errors + * (fail fast). ENOENT races (file removed between readdir and stat) count + * as external fs and are rate-limited. + */ + private async safeStat( + fullPath: string, + ): Promise<{ size: number; mtimeMs: number } | null> { + try { + return await this.fsPort.stat(fullPath); + } catch (err) { + if (isErrnoError(err)) { + this.emitDiagnostic(err); + return null; + } + throw err; + } + } + + /** + * Attempts to unlink a file. Returns true on success, false on filesystem + * errors (fail open — accounting NOT decremented). Rethrows internal errors. + */ + private async safeUnlink(fullPath: string): Promise { + try { + await this.fsPort.unlink(fullPath); + return true; + } catch (err) { + if (isErrnoError(err)) { + this.emitDiagnostic(err); + return false; + } + throw err; + } + } + + // ----------------------------------------------------------------------- + + private async touchClaim(now: number): Promise { + if (this.claimPath === null) return; + const date = new Date(now); + try { + await this.fsPort.utimes(this.claimPath, date, date); + } catch (err) { + if (isErrnoError(err)) { + this.emitDiagnostic(err); + return; + } + throw err; + } + } + + /** + * Determines whether an artifact is protected from automatic eviction. + * + * Automatic retention uses the narrow live-writer predicate for JSONL: a + * file is protected only when its day-key is the current UTC day AND its + * mtime is within the maintenance interval. This deliberately differs from + * explicit {@link perfDelete} (perfDelete.ts), which additionally protects + * any JSONL whose run holds a fresh claim — the narrower retention rule is + * what lets a 24×7 process converge to the eventual byte/file caps by + * evicting its own old-day files. + * + * Claims are protected individually by freshness (a non-stale claim is + * never reaped, and counts toward the caps) but a fresh claim never + * shields that run's older JSONL from retention eviction. + */ + private isProtected(artifact: ArtifactInfo, now: number): boolean { + if (isClaimFile(artifact.name)) { + return isNonStaleClaim(artifact.mtimeMs, now, this.claimLeaseMs); + } + if (isPerfJsonl(artifact.name)) { + return isLiveWriterFile( + artifact.name, + artifact.mtimeMs, + now, + this.maintenanceIntervalMs, + ); + } + return false; + } + + private emitDiagnostic(err: unknown): void { + const now = Date.now(); + if (now - this.lastDiagMs < this.diagRateLimitMs) return; + this.lastDiagMs = now; + const code = + err instanceof Error + ? ((err as NodeJS.ErrnoException).code ?? 'UNKNOWN') + : 'UNKNOWN'; + this.onDiagnostic(`perf retention error: ${code}`); + } +} + +// =========================================================================== +// Module-private utilities +// =========================================================================== + +function defaultDiagnostic(message: string): void { + process.stderr.write(`${message}\n`); +} + +/** Stable oldest-first comparison: by mtime, then by name. */ +function compareArtifactAge(a: ArtifactInfo, b: ArtifactInfo): number { + if (a.mtimeMs !== b.mtimeMs) { + return a.mtimeMs < b.mtimeMs ? -1 : 1; + } + if (a.name < b.name) return -1; + if (a.name > b.name) return 1; + return 0; +} diff --git a/packages/telemetry/src/perf/retention.validation.behavior.test.ts b/packages/telemetry/src/perf/retention.validation.behavior.test.ts new file mode 100644 index 0000000000..e1bb6ab113 --- /dev/null +++ b/packages/telemetry/src/perf/retention.validation.behavior.test.ts @@ -0,0 +1,238 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Constructor-tuning validation behavioral tests for PerfRetention (D-LC-5). + * + * Covers the constructor fail-fast validation split out of the original + * retention.behavior.test.ts: rejection of non-positive / non-finite tuning + * options and path-injection-safe runUuid handling. + * + * Real files, real filesystem, no mocks. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { PerfRetention } from './retention.js'; + +let dir: string; + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'perf-retention-')); +}); + +afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); +}); + +describe('PerfRetention constructor tuning validation (D-LC-5)', () => { + it('rejects maxBytes <= 0', () => { + expect( + () => + new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000020', + maxBytes: 0, + }), + ).toThrow(RangeError); + expect( + () => + new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000020', + maxBytes: -1, + }), + ).toThrow(RangeError); + }); + + it('rejects maxFiles <= 0', () => { + expect( + () => + new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000020', + maxFiles: 0, + }), + ).toThrow(RangeError); + expect( + () => + new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000020', + maxFiles: -5, + }), + ).toThrow(RangeError); + }); + + it('rejects non-finite maxBytes/maxFiles', () => { + expect( + () => + new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000020', + maxBytes: NaN, + }), + ).toThrow(RangeError); + expect( + () => + new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000020', + maxBytes: Infinity, + }), + ).toThrow(RangeError); + expect( + () => + new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000020', + maxFiles: NaN, + }), + ).toThrow(RangeError); + }); + + it('rejects maintenanceIntervalMs <= 0', () => { + expect( + () => + new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000020', + maintenanceIntervalMs: 0, + }), + ).toThrow(RangeError); + expect( + () => + new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000020', + maintenanceIntervalMs: -1, + }), + ).toThrow(RangeError); + }); + + it('rejects non-finite maintenanceIntervalMs/claimLeaseMs', () => { + expect( + () => + new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000020', + maintenanceIntervalMs: NaN, + }), + ).toThrow(RangeError); + expect( + () => + new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000020', + claimLeaseMs: Infinity, + }), + ).toThrow(RangeError); + }); + + it('rejects claimLeaseMs <= 0', () => { + expect( + () => + new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000020', + claimLeaseMs: 0, + }), + ).toThrow(RangeError); + }); + + it('rejects negative diagRateLimitMs but allows zero', () => { + expect( + () => + new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000020', + diagRateLimitMs: -1, + }), + ).toThrow(RangeError); + expect( + () => + new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000020', + diagRateLimitMs: NaN, + }), + ).toThrow(RangeError); + + expect( + () => + new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000020', + diagRateLimitMs: 0, + }), + ).not.toThrow(); + }); + + it('accepts the defaults (no tuning options)', () => { + expect( + () => + new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000021', + }), + ).not.toThrow(); + }); + + it('accepts sensible positive finite values', () => { + expect( + () => + new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000022', + maxBytes: 1024, + maxFiles: 10, + maintenanceIntervalMs: 1000, + claimLeaseMs: 3000, + diagRateLimitMs: 5000, + }), + ).not.toThrow(); + }); + + it('rejects a runUuid with path separators (fail-fast internally)', () => { + const slash = String.fromCharCode(0x2f); // '/' + const backslash = String.fromCharCode(0x5c); // '\' + const backspace = String.fromCharCode(0x08); // control char + const nullByte = String.fromCharCode(0x00); // null byte (POSIX truncation) + const del = String.fromCharCode(0x7f); // DEL + expect( + () => new PerfRetention({ dir, runUuid: `..${slash}escape` }), + ).toThrow(TypeError); + expect(() => new PerfRetention({ dir, runUuid: `a${slash}b` })).toThrow( + TypeError, + ); + expect(() => new PerfRetention({ dir, runUuid: `a${backslash}b` })).toThrow( + TypeError, + ); + expect(() => new PerfRetention({ dir, runUuid: `a${backspace}b` })).toThrow( + TypeError, + ); + expect(() => new PerfRetention({ dir, runUuid: `a${nullByte}b` })).toThrow( + TypeError, + ); + expect(() => new PerfRetention({ dir, runUuid: `a${del}b` })).toThrow( + TypeError, + ); + expect(() => new PerfRetention({ dir, runUuid: '..' })).toThrow(TypeError); + expect(() => new PerfRetention({ dir, runUuid: '' })).toThrow(TypeError); + }); + + it('accepts a canonical standard runUuid (no path-injection vectors)', () => { + expect( + () => + new PerfRetention({ + dir, + runUuid: '00000000-0000-4000-8000-000000000023', + }), + ).not.toThrow(); + }); +}); diff --git a/packages/telemetry/src/perf/tokenUsageReader.behavior.test.ts b/packages/telemetry/src/perf/tokenUsageReader.behavior.test.ts new file mode 100644 index 0000000000..16fd767287 --- /dev/null +++ b/packages/telemetry/src/perf/tokenUsageReader.behavior.test.ts @@ -0,0 +1,375 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Behavioral tests for the tolerant streaming token-usage reader (D1, AC-3). + * + * The reader streams the telemetry-owned token-usage JSONL directory one file + * at a time and structurally accepts turn rows. It must NOT import + * packages/agents (it defines its own tolerant structural acceptance). It must + * ignore non-turn lifecycle rows and tolerate malformed/future external JSONL + * with countable self-health — without whole-directory buffering. + * + * All tests use real files and the package-private readable-stream seam — no + * source-text assertions, no mocks. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { Readable } from 'node:stream'; +import { + consumeTokenUsageDirectory, + streamTokenUsageDirectory, + streamTokenUsageRecords, +} from './tokenUsageReader.js'; +import type { TokenUsageStreamEntry } from './tokenUsageReader.js'; +// The controlled-readable seam is package-private (not exported from the +// barrel); same-package tests import it directly from the internal module. +import { streamTokenUsageFromReadable } from './tokenUsageReader.js'; + +let dir: string; + +beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'token-usage-reader-')); +}); + +afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); +}); + +function writeFile(name: string, content: string): void { + fs.writeFileSync(path.join(dir, name), content, 'utf8'); +} + +function turnLine(overrides: Record = {}): string { + return JSON.stringify({ + prompt_id: 'sess-1#agentic-loop#aaaa', + actual_prompt_tokens: 1000, + output_tokens: 500, + ...overrides, + }); +} + +describe('streamTokenUsageRecords — real file classification', () => { + it('yields one turn entry per accepted turn row', async () => { + writeFile( + 'tokens.jsonl', + `${turnLine({ prompt_id: 'p1', actual_prompt_tokens: 100 })}\n` + + `${turnLine({ prompt_id: 'p2', actual_prompt_tokens: 200 })}\n`, + ); + const entries: TokenUsageStreamEntry[] = []; + for await (const e of streamTokenUsageRecords( + path.join(dir, 'tokens.jsonl'), + )) { + entries.push(e); + } + expect(entries.map((e) => e.kind)).toEqual(['turn', 'turn']); + const first = entries[0]; + if (first.kind !== 'turn') throw new Error('unreachable'); + expect(first.row.actualPromptTokens).toBe(100); + }); + + it('ignores non-turn lifecycle rows and counts them', async () => { + writeFile( + 'tokens.jsonl', + `${turnLine()}\n` + + `${JSON.stringify({ record_type: 'compression', before: 10, after: 5 })}\n` + + `${JSON.stringify({ record_type: 'provider_switch', from: 'a', to: 'b' })}\n` + + `${JSON.stringify({ record_type: 'model_switch' })}\n` + + `${turnLine({ prompt_id: 'p2' })}\n`, + ); + const entries = []; + for await (const e of streamTokenUsageRecords( + path.join(dir, 'tokens.jsonl'), + )) { + entries.push(e); + } + expect(entries.map((e) => e.kind)).toEqual([ + 'turn', + 'lifecycle', + 'lifecycle', + 'lifecycle', + 'turn', + ]); + }); + + it('tolerates malformed and blank lines without failing', async () => { + writeFile( + 'tokens.jsonl', + `${turnLine()}\n` + + `this is not json\n` + + `\n` + + ` \n` + + `${turnLine({ prompt_id: 'p2' })}\n` + + `{not an object}\n`, + ); + const entries = []; + for await (const e of streamTokenUsageRecords( + path.join(dir, 'tokens.jsonl'), + )) { + entries.push(e); + } + expect(entries.map((e) => e.kind)).toEqual([ + 'turn', + 'malformed', + 'blank', + 'blank', + 'turn', + 'malformed', + ]); + }); + + it('classifies a final truncated line as truncated', async () => { + writeFile( + 'tokens.jsonl', + `${turnLine()}\n` + `{"prompt_id":"truncated","actual_prompt_tok`, // no newline + ); + const entries = []; + for await (const e of streamTokenUsageRecords( + path.join(dir, 'tokens.jsonl'), + )) { + entries.push(e); + } + expect(entries.map((e) => e.kind)).toEqual(['turn', 'truncated']); + }); + + it('classifies a complete mid-file invalid JSON line as malformed (not truncated)', async () => { + writeFile( + 'tokens.jsonl', + `${turnLine()}\nbad json line\n${turnLine({ prompt_id: 'p2' })}\n`, + ); + const entries = []; + for await (const e of streamTokenUsageRecords( + path.join(dir, 'tokens.jsonl'), + )) { + entries.push(e); + } + expect(entries.map((e) => e.kind)).toEqual(['turn', 'malformed', 'turn']); + }); + + it('treats a non-object JSON value as malformed', async () => { + writeFile('tokens.jsonl', `${turnLine()}\n[1,2,3]\n42\n"string"\nnull\n`); + const entries = []; + for await (const e of streamTokenUsageRecords( + path.join(dir, 'tokens.jsonl'), + )) { + entries.push(e); + } + expect(entries.slice(1).map((e) => e.kind)).toEqual([ + 'malformed', + 'malformed', + 'malformed', + 'malformed', + ]); + }); + + it('omits outputTokens when the field is absent or non-numeric (never zero-filled)', async () => { + writeFile( + 'tokens.jsonl', + `${turnLine({ output_tokens: undefined })}\n` + // omitted + `${turnLine({ prompt_id: 'p2', output_tokens: 'NaN-string' })}\n` + // invalid type + `${turnLine({ prompt_id: 'p3', output_tokens: 0 })}\n`, // legitimate zero + ); + const entries: TokenUsageStreamEntry[] = []; + for await (const e of streamTokenUsageRecords( + path.join(dir, 'tokens.jsonl'), + )) { + entries.push(e); + } + const rows = entries + .filter( + (e): e is Extract => + e.kind === 'turn', + ) + .map((e) => e.row); + expect(rows[0]?.outputTokens).toBeUndefined(); + expect(rows[1]?.outputTokens).toBeUndefined(); + expect(rows[2]?.outputTokens).toBe(0); + }); + + it('rejects negative or non-numeric actual_prompt_tokens as lifecycle', async () => { + writeFile( + 'tokens.jsonl', + `${turnLine({ prompt_id: 'good' })}\n` + + `${JSON.stringify({ prompt_id: 'neg', actual_prompt_tokens: -5 })}\n` + + `${JSON.stringify({ prompt_id: 'str', actual_prompt_tokens: 'lots' })}\n` + + `${JSON.stringify({ prompt_id: '', actual_prompt_tokens: 10 })}\n`, // empty id + ); + const entries = []; + for await (const e of streamTokenUsageRecords( + path.join(dir, 'tokens.jsonl'), + )) { + entries.push(e); + } + expect(entries.map((e) => e.kind)).toEqual([ + 'turn', + 'lifecycle', + 'lifecycle', + 'lifecycle', + ]); + }); + + it('handles an empty file', async () => { + writeFile('empty.jsonl', ''); + const entries = []; + for await (const e of streamTokenUsageRecords( + path.join(dir, 'empty.jsonl'), + )) { + entries.push(e); + } + expect(entries).toEqual([]); + }); + + it('structurally accepts a future-version turn row with the right fields', async () => { + // The reader's acceptance is structural, not version-gated: a future + // schema version carrying the turn fields is accepted. + const future = JSON.stringify({ + schema_version: 99, + prompt_id: 'future-prompt', + actual_prompt_tokens: 777, + output_tokens: 88, + }); + writeFile('future.jsonl', future); + const entries: TokenUsageStreamEntry[] = []; + for await (const e of streamTokenUsageRecords( + path.join(dir, 'future.jsonl'), + )) { + entries.push(e); + } + expect(entries.map((e) => e.kind)).toEqual(['turn']); + const first = entries[0]; + if (first.kind !== 'turn') throw new Error('unreachable'); + expect(first.row.actualPromptTokens).toBe(777); + }); +}); + +describe('consumeTokenUsageDirectory — multi-file sorted reading + self-health', () => { + it('reads multiple sorted files and accumulates rows + counts', async () => { + writeFile( + 'a.jsonl', + `${turnLine({ prompt_id: 'aaa' })}\n` + + `${JSON.stringify({ record_type: 'compression' })}\n`, + ); + writeFile('b.jsonl', `${turnLine({ prompt_id: 'bbb' })}\n` + `not json\n`); + // A non-jsonl file must be ignored entirely. + fs.writeFileSync(path.join(dir, 'notes.txt'), 'ignore me\n'); + + const { rows, counts } = await consumeTokenUsageDirectory(dir); + expect(counts.files).toBe(2); + expect(counts.turns).toBe(2); + expect(counts.lifecycle).toBe(1); + expect(counts.malformed).toBe(1); + expect(rows.map((r) => r.promptId)).toEqual(['aaa', 'bbb']); + }); + + it('visits files in sorted name order for deterministic reading', async () => { + writeFile('z.jsonl', `${turnLine({ prompt_id: 'z' })}\n`); + writeFile('a.jsonl', `${turnLine({ prompt_id: 'a' })}\n`); + writeFile('m.jsonl', `${turnLine({ prompt_id: 'm' })}\n`); + + const { rows } = await consumeTokenUsageDirectory(dir); + expect(rows.map((r) => r.promptId)).toEqual(['a', 'm', 'z']); + }); + + it('a missing directory is an empty dataset (fail open)', async () => { + const { rows, counts } = await consumeTokenUsageDirectory( + path.join(dir, 'does-not-exist'), + ); + expect(rows).toEqual([]); + expect(counts.files).toBe(0); + expect(counts.turns).toBe(0); + }); + + it('distinguishes absent lifecycle (zero) from present-but-ignored', async () => { + // Pure turn file: lifecycle count is a KNOWN zero, distinguishable from + // an absent directory (which yields all-zero counts but empty rows). The + // counts object always carries explicit numeric fields. + writeFile('pure.jsonl', `${turnLine({ prompt_id: 'p1' })}\n`); + const { counts } = await consumeTokenUsageDirectory(dir); + expect(counts.lifecycle).toBe(0); + expect(counts.malformed).toBe(0); + expect(counts.truncated).toBe(0); + expect(counts.turns).toBe(1); + }); +}); + +describe('streamTokenUsageFromReadable — incremental yield proof', () => { + it('yields the first turn before the second chunk is pushed', async () => { + const line1 = `${turnLine({ prompt_id: 'first' })}\n`; + const line2 = `${turnLine({ prompt_id: 'second' })}\n`; + + const readable = new Readable({ read() {} }); + const iter = streamTokenUsageFromReadable(readable); + + readable.push(Buffer.from(line1)); + + const first = await iter.next(); + expect(first.done).toBe(false); + expect(first.value?.kind).toBe('turn'); + + readable.push(Buffer.from(line2)); + readable.push(null); + + const second = await iter.next(); + expect(second.done).toBe(false); + expect(second.value?.kind).toBe('turn'); + + const third = await iter.next(); + expect(third.done).toBe(true); + }); + + it('processes a large file incrementally without accumulating it all first', async () => { + const N = 5000; + const lines: string[] = []; + for (let i = 0; i < N; i++) { + lines.push(turnLine({ prompt_id: `p-${i}`, actual_prompt_tokens: i })); + } + writeFile('big.jsonl', lines.join('\n') + '\n'); + let count = 0; + let firstId = ''; + for await (const entry of streamTokenUsageRecords( + path.join(dir, 'big.jsonl'), + )) { + if (entry.kind !== 'turn') continue; + count++; + if (count === 1) firstId = entry.row.promptId; + } + expect(count).toBe(N); + expect(firstId).toBe('p-0'); + }); + + it('proves streaming by interleaving pushes and pulls', async () => { + const readable = new Readable({ read() {} }); + const iter = streamTokenUsageFromReadable(readable); + for (let i = 0; i < 3; i++) { + readable.push( + Buffer.from( + `${turnLine({ prompt_id: `p-${i}`, actual_prompt_tokens: i })}\n`, + ), + ); + const result = await iter.next(); + expect(result.done).toBe(false); + expect(result.value?.kind).toBe('turn'); + } + readable.push(null); + const done = await iter.next(); + expect(done.done).toBe(true); + }); +}); + +describe('streamTokenUsageDirectory — per-file source attribution', () => { + it('annotates each entry with its source file name', async () => { + writeFile('a.jsonl', `${turnLine({ prompt_id: 'a' })}\n`); + writeFile('b.jsonl', `${turnLine({ prompt_id: 'b' })}\n`); + const sources: string[] = []; + for await (const ce of streamTokenUsageDirectory(dir)) { + if (ce.entry.kind === 'turn') sources.push(ce.sourceFile); + } + expect(sources).toEqual(['a.jsonl', 'b.jsonl']); + }); +}); diff --git a/packages/telemetry/src/perf/tokenUsageReader.ts b/packages/telemetry/src/perf/tokenUsageReader.ts new file mode 100644 index 0000000000..8978d48039 --- /dev/null +++ b/packages/telemetry/src/perf/tokenUsageReader.ts @@ -0,0 +1,327 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Tolerant streaming reader for telemetry-owned token-usage JSONL (D1, AC-3). + * + * The perf operation record carries only `operation_id` (derived from the + * initial prompt-id prefix). Token-usage rows each carry their own `prompt_id` + * (one per send, including continuations). This reader streams the token-usage + * JSONL directory one file at a time and structurally accepts turn rows + * (`prompt_id` + `actual_prompt_tokens` [+ optional `output_tokens`]) so the + * report can join N continuation rows to the SINGLE perf operation at read time + * — without copying any child id onto the perf record and WITHOUT importing + * packages/agents (the canonical schema lives there; this reader defines its + * own tolerant structural acceptance so the telemetry layer never depends on + * the agents layer). + * + * Tolerance (external fs/JSONL input): + * - Non-turn lifecycle rows (compression / provider_switch / model_switch / + * session_resume / context_truncation, or any structured object lacking + * the turn fields) are ignored and counted, never fatal. + * - Malformed / truncated final lines are counted, never fatal. + * - A missing directory is an empty dataset (fail open). + * + * Genuine filesystem errors (permission denied, …) propagate; those are not + * line-content problems. No whole-directory buffering: each file is streamed + * line-by-line and entries are yielded incrementally. + */ + +import { promises as fsp } from 'node:fs'; +import { createReadStream } from 'node:fs'; +import { join } from 'node:path'; +import { StringDecoder } from 'node:string_decoder'; + +// =========================================================================== +// Types +// =========================================================================== + +/** + * A structurally-accepted token-usage turn row — the read-time join input. + * + * `promptId` is the initial prompt id or a continuation + * (`${initial}#continuation#${n}`). `actualPromptTokens` is the per-send + * prompt/context token count; `outputTokens` is optional (omitted, never + * zero-filled, when the provider did not report it). + */ +export interface TokenUsageTurnRow { + readonly promptId: string; + readonly actualPromptTokens: number; + readonly outputTokens?: number; +} + +/** + * Per-line streaming classification of a token-usage JSONL line. + * + * `turn` carries the accepted row; `lifecycle` is a structured non-turn row + * (ignored); `malformed` is a complete non-JSON / non-object line; + * `truncated` is a final unterminated non-JSON line (SIGKILL mid-append); + * `blank` is a whitespace-only line. + */ +export type TokenUsageStreamEntry = + | { readonly kind: 'turn'; readonly row: TokenUsageTurnRow } + | { readonly kind: 'lifecycle' } + | { readonly kind: 'malformed' } + | { readonly kind: 'truncated' } + | { readonly kind: 'blank' }; + +/** A streaming entry annotated with its source file name. */ +export interface TokenUsageConsumerEntry { + readonly entry: TokenUsageStreamEntry; + readonly sourceFile: string; +} + +/** Aggregate self-health counters across the token-usage directory. */ +export interface TokenUsageReaderCounts { + /** Accepted turn rows (join input). */ + readonly turns: number; + /** Structured non-turn rows, ignored (lifecycle records). */ + readonly lifecycle: number; + /** Complete lines that failed to parse or were non-objects. */ + readonly malformed: number; + /** Final unterminated non-JSON lines. */ + readonly truncated: number; + /** Blank / whitespace-only lines. */ + readonly blank: number; + /** Number of `*.jsonl` files read. */ + readonly files: number; +} + +export interface TokenUsageReaderResult { + readonly rows: readonly TokenUsageTurnRow[]; + readonly counts: TokenUsageReaderCounts; +} + +// =========================================================================== +// Helpers +// =========================================================================== + +function isPlainObject(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +function isFiniteNonNegNumber(value: unknown): value is number { + return typeof value === 'number' && Number.isFinite(value) && value >= 0; +} + +function hasErrnoCode(err: unknown, code: string): boolean { + return err instanceof Error && (err as NodeJS.ErrnoException).code === code; +} + +/** + * Structurally classifies a parsed JSON value from a token-usage JSONL line. + * + * A turn row is any plain object with a non-empty string `prompt_id` and a + * finite, non-negative numeric `actual_prompt_tokens`. An optional + * `output_tokens` is accepted only when it is a finite non-negative number; + * any other shape (missing, negative, NaN, a string) is treated as "not + * reported" (undefined) so the report never zero-fills an unreported cost. + * + * Any plain object that is NOT a turn row is `lifecycle` (ignored). Never + * throws. + */ +export function classifyTokenUsageLine(value: unknown): TokenUsageStreamEntry { + if (!isPlainObject(value)) { + return { kind: 'malformed' }; + } + const promptId = value['prompt_id']; + const actualPromptTokens = value['actual_prompt_tokens']; + if ( + typeof promptId === 'string' && + promptId.length > 0 && + isFiniteNonNegNumber(actualPromptTokens) + ) { + const rawOutput = value['output_tokens']; + return { + kind: 'turn', + row: { + promptId, + actualPromptTokens, + outputTokens: isFiniteNonNegNumber(rawOutput) ? rawOutput : undefined, + }, + }; + } + return { kind: 'lifecycle' }; +} + +/** + * Classifies a complete or final text line into a {@link TokenUsageStreamEntry}. + * + * `truncated` is reserved for a final nonblank line that is NOT valid JSON + * (realistic cause: SIGKILL mid-append). A complete (newline-terminated) + * non-JSON line is `malformed`. Blank/whitespace-only lines are `blank`. + * Never throws. + */ +function classifyTextLine( + text: string, + isFinal: boolean, +): TokenUsageStreamEntry { + if (text.trim() === '') { + return { kind: 'blank' }; + } + let value: unknown; + try { + value = JSON.parse(text); + } catch { + return isFinal ? { kind: 'truncated' } : { kind: 'malformed' }; + } + return classifyTokenUsageLine(value); +} + +function toBuffer(chunk: Buffer | string): Buffer { + return Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk); +} + +// =========================================================================== +// Package-private streaming seam (deep-imported by behavior tests; NOT in the +// public barrel). Mirrors perfRecordsStream so an incremental-yield proof can +// interleave pushes and pulls against a controlled readable. +// =========================================================================== + +/** + * Streams token-usage entries from any readable stream, yielding + * classification outcomes incrementally WITHOUT accumulating the entire stream. + * + * Genuine I/O failures propagate as a rejection; those are not line-content + * problems. + */ +export async function* streamTokenUsageFromReadable( + readable: NodeJS.ReadableStream, +): AsyncGenerator { + const decoder = new StringDecoder('utf8'); + let leftover = ''; + + for await (const chunk of readable) { + const data = leftover + decoder.write(toBuffer(chunk)); + const parts = data.split('\n'); + leftover = parts.pop() ?? ''; + for (const line of parts) { + yield classifyTextLine(line, false); + } + } + leftover += decoder.end(); + + if (leftover !== '') { + yield classifyTextLine(leftover, true); + } +} + +// =========================================================================== +// File-level streaming reader +// =========================================================================== + +/** + * Streams token-usage entries from a single JSONL file path, yielding + * classification outcomes incrementally WITHOUT reading the whole file. + * + * Genuine I/O failures (missing file, permission denied) propagate. + */ +export async function* streamTokenUsageRecords( + filePath: string, +): AsyncGenerator { + yield* streamTokenUsageFromReadable(createReadStream(filePath)); +} + +// =========================================================================== +// Directory-level reader (sorted, one file at a time, no directory buffering) +// =========================================================================== + +interface TokenUsageFileInfo { + readonly name: string; + readonly path: string; +} + +/** + * Lists sorted `*.jsonl` files in a directory one at a time. A missing + * directory (ENOENT) yields nothing — an empty dataset. Other errors + * propagate. + */ +async function* listTokenUsageFiles( + dir: string, +): AsyncGenerator { + let names: string[]; + try { + names = await fsp.readdir(dir); + } catch (err) { + if (hasErrnoCode(err, 'ENOENT')) return; + throw err; + } + const sorted = names.filter((n) => n.endsWith('.jsonl')).sort(); + for (const name of sorted) { + yield { name, path: join(dir, name) }; + } +} + +/** + * Streams token-usage entries from all `*.jsonl` files in a directory, one + * file at a time. Each entry carries its source file name. Files are visited + * in sorted order for deterministic reading; each file is streamed line-by-line + * (no whole-directory buffering). + * + * A missing directory yields nothing (empty dataset). Other genuine filesystem + * errors propagate. + */ +export async function* streamTokenUsageDirectory( + dir: string, +): AsyncGenerator { + for await (const file of listTokenUsageFiles(dir)) { + for await (const entry of streamTokenUsageRecords(file.path)) { + yield { entry, sourceFile: file.name }; + } + } +} + +/** + * Accumulating directory consumer: streams all `*.jsonl` token-usage files and + * collects every accepted turn row with aggregate self-health counts. The + * returned rows are the read-time join input for {@link buildReport}. + * + * Missing directory = empty result. Other genuine filesystem errors propagate. + */ +export async function consumeTokenUsageDirectory( + dir: string, +): Promise { + const rows: TokenUsageTurnRow[] = []; + let turns = 0; + let lifecycle = 0; + let malformed = 0; + let truncated = 0; + let blank = 0; + let files = 0; + + for await (const file of listTokenUsageFiles(dir)) { + files += 1; + for await (const entry of streamTokenUsageRecords(file.path)) { + switch (entry.kind) { + case 'turn': + rows.push(entry.row); + turns += 1; + break; + case 'lifecycle': + lifecycle += 1; + break; + case 'malformed': + malformed += 1; + break; + case 'truncated': + truncated += 1; + break; + case 'blank': + blank += 1; + break; + default: { + const _exhaustive: never = entry; + return _exhaustive; + } + } + } + } + + return { + rows, + counts: { turns, lifecycle, malformed, truncated, blank, files }, + }; +} diff --git a/packages/telemetry/src/telemetry/events/tool-events.ts b/packages/telemetry/src/telemetry/events/tool-events.ts index e33a1bb068..4d60188875 100644 --- a/packages/telemetry/src/telemetry/events/tool-events.ts +++ b/packages/telemetry/src/telemetry/events/tool-events.ts @@ -40,6 +40,8 @@ export class ToolCallEvent { start_ms?: number; /** Monotonic end timestamp (ms) for interval unioning */ end_ms?: number; + #perfStartMs?: number; + #perfEndMs?: number; constructor(call: CompletedToolCallShape) { this['event.name'] = 'tool_call'; @@ -60,14 +62,11 @@ export class ToolCallEvent { this.agent_id = call.request.agentId ?? DEFAULT_AGENT_ID; this.call_id = call.request.callId; - // Preserve caller-supplied start/end timestamps when available; only - // derive from duration as a fallback so interval unioning works even - // when callers don't provide explicit monotonic timestamps. - const hasExplicitStartEnd = - call.startMs !== undefined && call.endMs !== undefined; - if (hasExplicitStartEnd) { + if (call.startMs !== undefined && call.endMs !== undefined) { this.start_ms = call.startMs; this.end_ms = call.endMs; + this.#perfStartMs = call.startMs; + this.#perfEndMs = call.endMs; } else if (call.durationMs !== undefined && call.durationMs > 0) { const endMs = performance.now(); this.end_ms = endMs; @@ -87,6 +86,16 @@ export class ToolCallEvent { } } } + + /** + * Returns only caller-supplied monotonic boundaries for performance interval + * unioning. The public event fields retain their historical duration-based + * fallback for telemetry compatibility, but that estimate is not an honest + * overlap interval. + */ + getPerfBoundaries(): { startMs?: number; endMs?: number } { + return { startMs: this.#perfStartMs, endMs: this.#perfEndMs }; + } } function resolveHookName( diff --git a/packages/telemetry/src/telemetry/events/toolEvents.boundaries.behavior.test.ts b/packages/telemetry/src/telemetry/events/toolEvents.boundaries.behavior.test.ts new file mode 100644 index 0000000000..e935704000 --- /dev/null +++ b/packages/telemetry/src/telemetry/events/toolEvents.boundaries.behavior.test.ts @@ -0,0 +1,249 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Behavioral tests proving ToolCallEvent preserves honest tool-call + * boundaries (P07 contract, issue #3167 review finding B). + * + * Real ToolCallEvent construction + real logToolCall + real PerfPhaseObserver. + * + * Proves: + * - Caller-supplied startMs/endMs are preserved exactly. + * - Historical public event fields retain their duration-based fallback. + * - A completed call without explicit boundaries still carries durationMs + * (count/sum contribution) but contributes no performance union endpoints. + * - Production-conversion: staggered/completed calls lacking boundaries and + * explicit-boundary controls. + */ + +import { describe, it, expect, beforeEach, afterEach, vi } from 'bun:test'; +import { logToolCall } from '../loggers.js'; +import { ToolCallEvent } from './tool-events.js'; +import type { CompletedToolCallShape } from '../../internal/interfaces.js'; +import * as sdk from '../sdk.js'; +import * as uiTelemetry from '../uiTelemetry.js'; +import { + setPerfPhaseObserver, + type PerfPhaseObserver, + type PerfToolCallCompletedInfo, +} from '../../perf/perfPhaseObserver.js'; + +function makeCompletedCall( + overrides: Partial & { + callId?: string; + promptId?: string; + startMs?: number; + endMs?: number; + durationMs?: number; + } = {}, +): CompletedToolCallShape { + return { + status: 'success', + request: { + name: 'test_tool', + args: {}, + callId: overrides.callId ?? 'call-1', + isClientInitiated: true, + prompt_id: overrides.promptId ?? 'sess-1#agentic-loop#uuid', + agentId: 'primary', + }, + response: { + callId: overrides.callId ?? 'call-1', + responseParts: [{ text: 'done' }], + }, + tool: {}, + durationMs: overrides.durationMs, + startMs: overrides.startMs, + endMs: overrides.endMs, + ...overrides, + }; +} + +function capturingObserver(): { + observer: PerfPhaseObserver; + toolCalls: PerfToolCallCompletedInfo[]; +} { + const toolCalls: PerfToolCallCompletedInfo[] = []; + const observer: PerfPhaseObserver = { + onProviderAttemptStart: () => undefined, + onProviderAttemptEnd: () => undefined, + onToolCallCompleted: (info) => toolCalls.push(info), + }; + return { observer, toolCalls }; +} + +const mockConfig = { + getSessionId: () => 'test-session-id', + getTargetDir: () => 'target-dir', + getUsageStatisticsEnabled: () => true, + getTelemetryEnabled: () => true, + getTelemetryLogPromptsEnabled: () => true, +} as unknown as Parameters[0]; + +describe('ToolCallEvent honest boundaries (P07 contract, finding B)', () => { + beforeEach(() => { + vi.spyOn(sdk, 'isTelemetrySdkInitialized').mockReturnValue(false); + vi.spyOn(uiTelemetry.uiTelemetryService, 'addEvent').mockImplementation( + () => undefined, + ); + setPerfPhaseObserver(null); + }); + + afterEach(() => { + vi.restoreAllMocks(); + setPerfPhaseObserver(null); + }); + + // --- Explicit boundaries are preserved --- + + it('preserves caller-supplied startMs/endMs exactly', () => { + const event = new ToolCallEvent( + makeCompletedCall({ startMs: 1000, endMs: 1050, durationMs: 50 }), + ); + expect(event.start_ms).toBe(1000); + expect(event.end_ms).toBe(1050); + expect(event.duration_ms).toBe(50); + }); + + it('passes explicit startMs/endMs through to the observer', () => { + const { observer, toolCalls } = capturingObserver(); + setPerfPhaseObserver(observer); + + logToolCall( + mockConfig, + new ToolCallEvent( + makeCompletedCall({ callId: 'c1', startMs: 200, endMs: 350 }), + ), + ); + + expect(toolCalls).toHaveLength(1); + expect(toolCalls[0].startMs).toBe(200); + expect(toolCalls[0].endMs).toBe(350); + }); + + // --- Missing boundaries: NO invented interval --- + + it('retains event compatibility without exposing fallback boundaries to perf', () => { + const event = new ToolCallEvent(makeCompletedCall({ durationMs: 100 })); + expect(event.duration_ms).toBe(100); + expect(typeof event.start_ms).toBe('number'); + expect(typeof event.end_ms).toBe('number'); + expect((event.end_ms ?? 0) - (event.start_ms ?? 0)).toBe(100); + expect(event.getPerfBoundaries()).toEqual({ + startMs: undefined, + endMs: undefined, + }); + }); + + it('does NOT invent start_ms/end_ms when durationMs is zero', () => { + const event = new ToolCallEvent(makeCompletedCall({ durationMs: 0 })); + expect(event.duration_ms).toBe(0); + expect(event.start_ms).toBeUndefined(); + expect(event.end_ms).toBeUndefined(); + }); + + it('does NOT invent start_ms/end_ms when durationMs is undefined', () => { + const event = new ToolCallEvent(makeCompletedCall({})); + expect(event.duration_ms).toBe(0); + expect(event.start_ms).toBeUndefined(); + expect(event.end_ms).toBeUndefined(); + }); + + it('observer receives undefined startMs/endMs (no invented endpoints) for boundary-less call', () => { + const { observer, toolCalls } = capturingObserver(); + setPerfPhaseObserver(observer); + + logToolCall( + mockConfig, + new ToolCallEvent(makeCompletedCall({ callId: 'c2', durationMs: 250 })), + ); + + expect(toolCalls).toHaveLength(1); + expect(toolCalls[0].startMs).toBeUndefined(); + expect(toolCalls[0].endMs).toBeUndefined(); + // duration still preserved — the call still contributes count/sum. + expect(toolCalls[0].durationMs).toBe(250); + }); + + // --- Partial boundaries: only both-present counts as explicit --- + + it('does not expose a partial start boundary to the perf observer', () => { + const event = new ToolCallEvent( + makeCompletedCall({ startMs: 500, durationMs: 100 }), + ); + expect(event.duration_ms).toBe(100); + expect(event.getPerfBoundaries()).toEqual({ + startMs: undefined, + endMs: undefined, + }); + }); + + it('does not expose a partial end boundary to the perf observer', () => { + const event = new ToolCallEvent( + makeCompletedCall({ endMs: 600, durationMs: 100 }), + ); + expect(event.duration_ms).toBe(100); + expect(event.getPerfBoundaries()).toEqual({ + startMs: undefined, + endMs: undefined, + }); + }); + + // --- Production conversion: staggered calls --- + + it('staggered completed calls without boundaries contribute duration but no endpoints', () => { + const { observer, toolCalls } = capturingObserver(); + setPerfPhaseObserver(observer); + + // Three tool calls completing at different times, none with explicit + // monotonic boundaries — the production conversion path. + for (const [i, dur] of [50, 120, 200].entries()) { + logToolCall( + mockConfig, + new ToolCallEvent( + makeCompletedCall({ callId: `stagger-${i}`, durationMs: dur }), + ), + ); + } + + expect(toolCalls).toHaveLength(3); + // Every call contributes its duration (count/sum)... + expect(toolCalls.map((t) => t.durationMs)).toEqual([50, 120, 200]); + // ...but none carries an invented interval endpoint. + for (const info of toolCalls) { + expect(info.startMs).toBeUndefined(); + expect(info.endMs).toBeUndefined(); + } + }); + + it('mixed: explicit-boundary and boundary-less calls coexist honestly', () => { + const { observer, toolCalls } = capturingObserver(); + setPerfPhaseObserver(observer); + + // Call with explicit boundaries. + logToolCall( + mockConfig, + new ToolCallEvent( + makeCompletedCall({ callId: 'explicit', startMs: 10, endMs: 60 }), + ), + ); + // Call without boundaries. + logToolCall( + mockConfig, + new ToolCallEvent( + makeCompletedCall({ callId: 'implicit', durationMs: 90 }), + ), + ); + + expect(toolCalls).toHaveLength(2); + const explicit = toolCalls.find((t) => t.callId === 'explicit')!; + const implicit = toolCalls.find((t) => t.callId === 'implicit')!; + expect(explicit.startMs).toBe(10); + expect(explicit.endMs).toBe(60); + expect(implicit.startMs).toBeUndefined(); + expect(implicit.endMs).toBeUndefined(); + }); +}); diff --git a/packages/telemetry/src/telemetry/index.ts b/packages/telemetry/src/telemetry/index.ts index b88490158f..8337927c62 100644 --- a/packages/telemetry/src/telemetry/index.ts +++ b/packages/telemetry/src/telemetry/index.ts @@ -100,6 +100,7 @@ export { type ModelBreakdown, type ApiAttemptRecord, } from './sessionMetricsAggregator.js'; +export { IntervalUnion } from './intervalUnion.js'; export { ToolConfirmationOutcome } from '../internal/interfaces.js'; export type { TelemetryConfig, diff --git a/packages/telemetry/src/telemetry/intervalUnion.behavior.test.ts b/packages/telemetry/src/telemetry/intervalUnion.behavior.test.ts new file mode 100644 index 0000000000..13a7e32478 --- /dev/null +++ b/packages/telemetry/src/telemetry/intervalUnion.behavior.test.ts @@ -0,0 +1,250 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, it, expect, beforeEach } from 'bun:test'; +import { IntervalUnion } from './intervalUnion.js'; + +/** + * Independent brute-force merge of intervals, used only to cross-check the + * incrementally maintained durationMs(). It re-derives the union from the raw + * recorded adds rather than trusting the class under test. + */ +function bruteForceDuration( + intervals: ReadonlyArray, +): number { + const valid = intervals.filter( + ([s, e]) => Number.isFinite(s) && Number.isFinite(e) && e > s, + ); + if (valid.length === 0) return 0; + const sorted = [...valid].sort((a, b) => a[0] - b[0]); + let total = 0; + let curStart = sorted[0][0]; + let curEnd = sorted[0][1]; + for (let i = 1; i < sorted.length; i++) { + if (sorted[i][0] <= curEnd) { + curEnd = Math.max(curEnd, sorted[i][1]); + } else { + total += curEnd - curStart; + curStart = sorted[i][0]; + curEnd = sorted[i][1]; + } + } + total += curEnd - curStart; + return total; +} + +describe('IntervalUnion', () => { + let u: IntervalUnion; + + beforeEach(() => { + u = new IntervalUnion(); + }); + + describe('empty state', () => { + it('reports zero duration and zero count before any add', () => { + expect(u.durationMs()).toBe(0); + expect(u.count()).toBe(0); + expect(u.latestEnd).toBe(0); + }); + }); + + describe('EVIDENCE-AC5a: disjoint intervals', () => { + it('sums disjoint interval durations and grows count', () => { + u.add(0, 10); + u.add(20, 30); + u.add(40, 50); + expect(u.durationMs()).toBe(30); + expect(u.count()).toBe(3); + }); + + it('keeps a gap between disjoint intervals uncounted', () => { + u.add(0, 100); + u.add(200, 300); + expect(u.durationMs()).toBe(200); + }); + }); + + describe('overlapping intervals merge without double-counting', () => { + it('counts the overlap once', () => { + u.add(0, 1000); + u.add(500, 1500); + expect(u.durationMs()).toBe(1500); + expect(u.count()).toBe(1); + }); + + it('collapses identical intervals to a single span', () => { + u.add(0, 2000); + u.add(0, 2000); + expect(u.durationMs()).toBe(2000); + expect(u.count()).toBe(1); + }); + }); + + describe('adjacent (touching) intervals merge', () => { + it('merges [0,10) and [10,20) into a single 20ms span', () => { + u.add(0, 10); + u.add(10, 20); + expect(u.durationMs()).toBe(20); + expect(u.count()).toBe(1); + }); + }); + + describe('nested interval adds no duration', () => { + it('an interval fully inside another changes nothing', () => { + u.add(0, 100); + expect(u.durationMs()).toBe(100); + u.add(10, 20); + expect(u.durationMs()).toBe(100); + expect(u.count()).toBe(1); + }); + + it('a later larger interval absorbs earlier nested ones', () => { + u.add(10, 20); + u.add(30, 40); + u.add(0, 100); + expect(u.durationMs()).toBe(100); + expect(u.count()).toBe(1); + }); + }); + + describe('incremental duration equals brute-force recompute', () => { + it('matches an independent merge after each mixed insert', () => { + const recorded: Array<[number, number]> = []; + const pattern: ReadonlyArray = [ + [0, 10], + [5, 15], + [100, 110], + [105, 200], + [50, 60], + [55, 58], + [300, 310], + [0, 400], + [1000, 1010], + [1005, 1020], + ]; + for (const [s, e] of pattern) { + u.add(s, e); + recorded.push([s, e]); + expect(u.durationMs()).toBe(bruteForceDuration(recorded)); + } + }); + + it('stays exact across 250 disjoint intervals', () => { + const recorded: Array<[number, number]> = []; + for (let i = 0; i < 250; i++) { + const s = i * 20; + u.add(s, s + 10); + recorded.push([s, s + 10]); + } + expect(u.durationMs()).toBe(bruteForceDuration(recorded)); + expect(u.durationMs()).toBe(250 * 10); + }); + + it('stays exact with out-of-order overlapping inserts', () => { + const recorded: Array<[number, number]> = []; + const starts = [400, 10, 200, 5, 350, 100, 0, 250]; + for (const s of starts) { + u.add(s, s + 50); + recorded.push([s, s + 50]); + expect(u.durationMs()).toBe(bruteForceDuration(recorded)); + } + }); + }); + + describe('union of two sets', () => { + it('merges two unions into one', () => { + const a = new IntervalUnion(); + a.add(0, 10); + a.add(20, 30); + const b = new IntervalUnion(); + b.add(5, 25); + const merged = a.union(b); + expect(merged.durationMs()).toBe(30); + expect(merged.count()).toBe(1); + }); + + it('does not mutate the operands', () => { + const a = new IntervalUnion(); + a.add(0, 10); + const b = new IntervalUnion(); + b.add(100, 110); + const merged = a.union(b); + expect(merged.durationMs()).toBe(20); + expect(a.durationMs()).toBe(10); + expect(b.durationMs()).toBe(10); + }); + + it('handles empty operands', () => { + const a = new IntervalUnion(); + const b = new IntervalUnion(); + b.add(0, 10); + expect(a.union(b).durationMs()).toBe(10); + expect(b.union(a).durationMs()).toBe(10); + expect(a.union(new IntervalUnion()).durationMs()).toBe(0); + }); + }); + + describe('invalid, zero-length and degenerate intervals', () => { + it('ignores zero-length intervals', () => { + u.add(0, 0); + expect(u.durationMs()).toBe(0); + expect(u.count()).toBe(0); + }); + + it('ignores negative-length intervals', () => { + u.add(10, 5); + expect(u.durationMs()).toBe(0); + expect(u.count()).toBe(0); + }); + + it('ignores non-finite endpoints', () => { + u.add(Number.NaN, 10); + u.add(0, Number.POSITIVE_INFINITY); + u.add(Number.NEGATIVE_INFINITY, 10); + expect(u.durationMs()).toBe(0); + expect(u.count()).toBe(0); + }); + + it('leaves an existing union untouched when a degenerate add follows', () => { + u.add(0, 10); + u.add(5, 5); + u.add(20, 10); + u.add(Number.NaN, 100); + expect(u.durationMs()).toBe(10); + expect(u.count()).toBe(1); + }); + }); + + describe('latestEnd', () => { + it('tracks the maximum end across intervals', () => { + u.add(0, 10); + expect(u.latestEnd).toBe(10); + u.add(100, 200); + expect(u.latestEnd).toBe(200); + u.add(5, 15); + expect(u.latestEnd).toBe(200); + }); + }); + + describe('clear', () => { + it('resets duration and count to zero', () => { + u.add(0, 10); + u.add(20, 30); + u.clear(); + expect(u.durationMs()).toBe(0); + expect(u.count()).toBe(0); + expect(u.latestEnd).toBe(0); + }); + + it('allows reuse after clear', () => { + u.add(0, 10); + u.clear(); + u.add(100, 110); + expect(u.durationMs()).toBe(10); + expect(u.count()).toBe(1); + }); + }); +}); diff --git a/packages/telemetry/src/telemetry/intervalUnion.ts b/packages/telemetry/src/telemetry/intervalUnion.ts new file mode 100644 index 0000000000..121c07e8f6 --- /dev/null +++ b/packages/telemetry/src/telemetry/intervalUnion.ts @@ -0,0 +1,121 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +interface Interval { + start: number; + end: number; +} + +/** + * Incrementally maintained, sorted, non-overlapping interval list. + * + * Each insertion is O(n) worst-case (binary-search position + neighbour + * merge). The total covered duration is maintained incrementally so + * durationMs() is O(1): add() adjusts the cached total by the net change of + * the merge only, never re-walking every interval. This replaces the + * previously private quadratic implementation that recomputed the full + * duration on every insert. + * + * Merge semantics (preserved from the original private implementation): + * - touching (adjacent) intervals merge, e.g. [0,10) + [10,20) => [0,20) + * - overlapping intervals merge, counting the overlap once + * - an interval fully nested inside another adds no duration + * - gaps are never bridged + * - degenerate (end <= start), zero-length and non-finite intervals are ignored + */ +class IntervalUnion { + private readonly intervals: Interval[] = []; + private cachedDurationMs = 0; + + add(start: number, end: number): void { + if (!Number.isFinite(start) || !Number.isFinite(end)) return; + if (end <= start) return; + + const list = this.intervals; + + if (list.length === 0) { + list.push({ start, end }); + this.cachedDurationMs += end - start; + return; + } + + // Binary search: first index whose start >= `start`. + let lo = 0; + let hi = list.length; + while (lo < hi) { + const mid = (lo + hi) >> 1; + if (list[mid].start < start) { + lo = mid + 1; + } else { + hi = mid; + } + } + + let from = lo; + let mergedStart = start; + let mergedEnd = end; + + // Left neighbour overlaps or is adjacent (touching merges). + if (lo > 0 && list[lo - 1].end >= start) { + from = lo - 1; + mergedStart = list[from].start; + mergedEnd = Math.max(list[from].end, end); + } + + // Absorb every interval forward that overlaps or touches the merge. + let to = from; + while (to < list.length && list[to].start <= mergedEnd) { + mergedEnd = Math.max(mergedEnd, list[to].end); + to++; + } + + if (from === to) { + // No overlap: pure insert, the full span is net-new. + list.splice(lo, 0, { start, end }); + this.cachedDurationMs += end - start; + return; + } + + // Replace [from, to) with the merged span, adjusting the cached + // duration by the net change (merged span - sum of removed spans). + let removedDuration = 0; + for (let k = from; k < to; k++) { + removedDuration += list[k].end - list[k].start; + } + list.splice(from, to - from, { start: mergedStart, end: mergedEnd }); + this.cachedDurationMs += mergedEnd - mergedStart - removedDuration; + } + + durationMs(): number { + return this.cachedDurationMs; + } + + count(): number { + return this.intervals.length; + } + + get latestEnd(): number { + return this.intervals[this.intervals.length - 1]?.end ?? 0; + } + + union(other: IntervalUnion): IntervalUnion { + const result = new IntervalUnion(); + for (const iv of this.intervals) { + result.add(iv.start, iv.end); + } + for (const iv of other.intervals) { + result.add(iv.start, iv.end); + } + return result; + } + + clear(): void { + this.intervals.length = 0; + this.cachedDurationMs = 0; + } +} + +export { IntervalUnion }; diff --git a/packages/telemetry/src/telemetry/loggers.perf.behavior.test.ts b/packages/telemetry/src/telemetry/loggers.perf.behavior.test.ts new file mode 100644 index 0000000000..ca81d1250b --- /dev/null +++ b/packages/telemetry/src/telemetry/loggers.perf.behavior.test.ts @@ -0,0 +1,183 @@ +/** + * @license + * Copyright 2026 Vybestack LLC + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Behavioral tests proving logToolCall invokes the PerfPhaseObserver at the + * tool-call-completion lifecycle boundary (P07, EVIDENCE-AC5). + * + * Real ToolCallEvent construction + real logToolCall + real PerfPhaseObserver seam. + * + * Proves: + * - Tool completion metrics use the real call_id/start_ms/end_ms/duration seam + * - SDK-disabled mode still notifies + * - D8: observer invoked outside try/catch (fail-fast on observer error) + * - Default-off: null observer → no notification, no crash + * - Characterizes missing call_id/timing honestly (no invented IDs) + */ + +import { describe, it, expect, beforeEach, afterEach, vi } from 'bun:test'; +import { logToolCall } from './loggers.js'; +import { ToolCallEvent } from './types.js'; +import type { CompletedToolCallShape } from '../internal/interfaces.js'; +import * as sdk from './sdk.js'; +import * as uiTelemetry from './uiTelemetry.js'; +import { + setPerfPhaseObserver, + getPerfPhaseObserver, + type PerfPhaseObserver, + type PerfToolCallCompletedInfo, +} from '../perf/perfPhaseObserver.js'; + +function makeCompletedCall( + overrides: Partial & { + callId?: string; + promptId?: string; + startMs?: number; + endMs?: number; + durationMs?: number; + } = {}, +): CompletedToolCallShape { + return { + status: 'success', + request: { + name: 'test_tool', + args: {}, + callId: overrides.callId ?? 'call-1', + isClientInitiated: true, + prompt_id: overrides.promptId ?? 'sess-1#agentic-loop#uuid', + agentId: 'primary', + }, + response: { + callId: overrides.callId ?? 'call-1', + responseParts: [{ text: 'done' }], + }, + tool: {}, + durationMs: overrides.durationMs ?? 50, + startMs: overrides.startMs, + endMs: overrides.endMs, + ...overrides, + }; +} + +function capturingObserver(): { + observer: PerfPhaseObserver; + toolCalls: PerfToolCallCompletedInfo[]; +} { + const toolCalls: PerfToolCallCompletedInfo[] = []; + const observer: PerfPhaseObserver = { + onProviderAttemptStart: () => undefined, + onProviderAttemptEnd: () => undefined, + onToolCallCompleted: (info) => toolCalls.push(info), + }; + return { observer, toolCalls }; +} + +const mockConfig = { + getSessionId: () => 'test-session-id', + getTargetDir: () => 'target-dir', + getUsageStatisticsEnabled: () => true, + getTelemetryEnabled: () => true, + getTelemetryLogPromptsEnabled: () => true, +} as unknown as Parameters[0]; + +describe('logToolCall perf phase observer (P07)', () => { + beforeEach(() => { + vi.spyOn(sdk, 'isTelemetrySdkInitialized').mockReturnValue(false); + vi.spyOn(uiTelemetry.uiTelemetryService, 'addEvent').mockImplementation( + () => undefined, + ); + setPerfPhaseObserver(null); + }); + + afterEach(() => { + vi.restoreAllMocks(); + setPerfPhaseObserver(null); + }); + + it('notifies observer with real call_id/start_ms/end_ms/duration', () => { + const { observer, toolCalls } = capturingObserver(); + setPerfPhaseObserver(observer); + + const call = makeCompletedCall({ + callId: 'call-abc', + promptId: 'sess-1#agentic-loop#uuid#continuation#1', + startMs: 1000, + endMs: 1050, + durationMs: 50, + }); + logToolCall(mockConfig, new ToolCallEvent(call)); + + expect(toolCalls).toHaveLength(1); + expect(toolCalls[0].callId).toBe('call-abc'); + expect(toolCalls[0].startMs).toBe(1000); + expect(toolCalls[0].endMs).toBe(1050); + expect(toolCalls[0].durationMs).toBe(50); + expect(toolCalls[0].promptId).toBe( + 'sess-1#agentic-loop#uuid#continuation#1', + ); + }); + + it('continuation prompt_id is carried for D1 association', () => { + const { observer, toolCalls } = capturingObserver(); + setPerfPhaseObserver(observer); + + const call = makeCompletedCall({ + promptId: 'sess-1#agentic-loop#uuid#continuation#2', + }); + logToolCall(mockConfig, new ToolCallEvent(call)); + + expect(toolCalls[0].promptId).toBe( + 'sess-1#agentic-loop#uuid#continuation#2', + ); + }); + + it('SDK-disabled mode still notifies', () => { + vi.spyOn(sdk, 'isTelemetrySdkInitialized').mockReturnValue(false); + const { observer, toolCalls } = capturingObserver(); + setPerfPhaseObserver(observer); + + logToolCall(mockConfig, new ToolCallEvent(makeCompletedCall())); + + expect(toolCalls).toHaveLength(1); + }); + + it('default-off: null observer produces no notification and no crash', () => { + setPerfPhaseObserver(null); + expect(getPerfPhaseObserver()).toBeNull(); + logToolCall(mockConfig, new ToolCallEvent(makeCompletedCall())); + // No crash. + }); + + it('D8: observer error propagates (fail-fast, not swallowed)', () => { + const throwingObserver: PerfPhaseObserver = { + onProviderAttemptStart: () => undefined, + onProviderAttemptEnd: () => undefined, + onToolCallCompleted: () => { + throw new Error('observer internal error'); + }, + }; + setPerfPhaseObserver(throwingObserver); + + expect(() => + logToolCall(mockConfig, new ToolCallEvent(makeCompletedCall())), + ).toThrow('observer internal error'); + }); + + it('characterizes missing call_id honestly (undefined, not invented)', () => { + const { observer, toolCalls } = capturingObserver(); + setPerfPhaseObserver(observer); + + // Construct the event with a properly typed request, then override + // call_id to undefined to test the optional-field code path without + // erasing the required type on ToolCallRequest.callId. + const event = new ToolCallEvent(makeCompletedCall()); + (event as { call_id?: string }).call_id = undefined; + logToolCall(mockConfig, event); + + // call_id is undefined when the request has none — no invented ID. + expect(toolCalls[0].callId).toBeUndefined(); + }); +}); diff --git a/packages/telemetry/src/telemetry/loggers.ts b/packages/telemetry/src/telemetry/loggers.ts index 57ce26914b..1cbe738266 100644 --- a/packages/telemetry/src/telemetry/loggers.ts +++ b/packages/telemetry/src/telemetry/loggers.ts @@ -74,6 +74,7 @@ import { isTelemetrySdkInitialized } from './sdk.js'; import { uiTelemetryService, type UiEvent } from './uiTelemetry.js'; import { safeJsonStringify } from '../utils/safeJsonStringify.js'; import { debugLogger } from '../utils/debugLogger.js'; +import { getPerfPhaseObserver } from '../perf/perfPhaseObserver.js'; type Config = TelemetryConfig; type SessionConfig = Pick; @@ -214,6 +215,22 @@ export function logToolCall( debugLogger.error(`[TELEMETRY] logToolCall: ${event.function_name}`); } + // Perf phase observer (P07): notify at the exact tool-completion boundary, + // BEFORE the SDK/export gate so SDK-disabled mode still notifies. Invoked + // directly (no try/catch) so internal errors propagate fail-fast (D8). + // Default-off: null observer short-circuits with no allocation. + const perfObserver = getPerfPhaseObserver(); + if (perfObserver !== null) { + const boundaries = event.getPerfBoundaries(); + perfObserver.onToolCallCompleted({ + promptId: event.prompt_id, + callId: event.call_id, + startMs: boundaries.startMs, + endMs: boundaries.endMs, + durationMs: event.duration_ms, + }); + } + // Local aggregation always runs, regardless of SDK/export state const uiEvent = { ...event, diff --git a/packages/telemetry/src/telemetry/sessionMetricsAggregator.ts b/packages/telemetry/src/telemetry/sessionMetricsAggregator.ts index 252c57812e..9c3e005ee3 100644 --- a/packages/telemetry/src/telemetry/sessionMetricsAggregator.ts +++ b/packages/telemetry/src/telemetry/sessionMetricsAggregator.ts @@ -4,6 +4,8 @@ * SPDX-License-Identifier: Apache-2.0 */ +import { IntervalUnion } from './intervalUnion.js'; + /** * Canonical session-level metrics aggregator. * @@ -126,11 +128,6 @@ export interface SessionMetricsSnapshot { models: Record; } -interface Interval { - start: number; - end: number; -} - interface CompactTimingSums { sumInputPlusOutput: number; sumDurationMs: number; @@ -188,89 +185,6 @@ class DedupSet { } } -/** - * Incrementally maintained, sorted, non-overlapping interval list. - * - * Each insertion is O(n) worst-case (binary-search position + neighbor - * merge) — never a full O(n log n) re-sort. The union is kept exact for - * the entire session lifetime: no intervals are evicted or merged across - * gaps, so a late-arriving out-of-order interval that overlaps an earlier - * one is always applied correctly. Gaps are never bridged. - */ -class IntervalUnion { - private intervals: Interval[] = []; - private cachedDuration = 0; - - add(start: number, end: number): void { - // Reject non-finite endpoints so NaN/Infinity cannot poison the union - if (!Number.isFinite(start) || !Number.isFinite(end)) return; - if (end <= start) return; - this.insertSorted({ start, end }); - this.recomputeDuration(); - } - - get duration(): number { - return this.cachedDuration; - } - - get count(): number { - return this.intervals.length; - } - - get latestEnd(): number { - return this.intervals[this.intervals.length - 1]?.end ?? 0; - } - - getMerged(): readonly Interval[] { - return this.intervals; - } - - clear(): void { - this.intervals = []; - this.cachedDuration = 0; - } - - private insertSorted(interval: Interval): void { - const list = this.intervals; - if (list.length === 0) { - list.push(interval); - return; - } - - let lo = 0; - let hi = list.length; - while (lo < hi) { - const mid = (lo + hi) >> 1; - if (list[mid].start < interval.start) { - lo = mid + 1; - } else { - hi = mid; - } - } - - if (lo > 0 && list[lo - 1].end >= interval.start) { - lo--; - list[lo].end = Math.max(list[lo].end, interval.end); - } else { - list.splice(lo, 0, interval); - } - - const i = lo + 1; - while (i < list.length && list[i].start <= list[lo].end) { - list[lo].end = Math.max(list[lo].end, list[i].end); - list.splice(i, 1); - } - } - - private recomputeDuration(): void { - let total = 0; - for (const iv of this.intervals) { - total += iv.end - iv.start; - } - this.cachedDuration = total; - } -} - export class SessionMetricsAggregator { private readonly seenAttemptIds = new DedupSet(); private readonly seenToolCallIds = new DedupSet(); @@ -741,7 +655,7 @@ export class SessionMetricsAggregator { } private computeAgentActiveTimeMs(sessionCurrentMs: number): number { - const rawAgentActiveTimeMs = this.activeIntervals.duration; + const rawAgentActiveTimeMs = this.activeIntervals.durationMs(); // When sessionStartMs is null (no positive timestamps recorded), we // cannot compute a meaningful wall-clock clamp. Return the raw union // duration to avoid artificially zeroing out activity time. diff --git a/project-plans/issue3167/.completed/P13.md b/project-plans/issue3167/.completed/P13.md new file mode 100644 index 0000000000..a4fd8df6fd --- /dev/null +++ b/project-plans/issue3167/.completed/P13.md @@ -0,0 +1,82 @@ +# Phase P13 Completion Evidence + +Plan ID: PLAN-20260808-PERFTREND.P13 +Issue: #3167 +Completion timestamp: 2026-08-10T05:50:00Z + +## Whole-suite verification + +- `npm run test`: passed with exit status 0 on the definitive final post-review, post-format source state. Principal package runners reported 368/368 core test files, 561/561 isolated provider files, 365/365 agents files, and 706/706 CLI files; every remaining workspace runner also passed. + - Local host note: Bun 1.3.14 intermittently spins or traps in isolated child processes under JIT on this macOS host. The successful full run used host-only files under `/private/tmp`: agents children ran with JIT disabled, while the one agents test whose performance assertions require JIT ran separately with DFG/FTL disabled and its unfiltered JUnit cases were merged. Other packages used DFG/FTL disabled. CLI and agents runners used supported concurrency 1. No repository source, test, timeout, or policy was changed for this host workaround. + - The temporary root Bun symlink was restored to `../bun/bin/bun.exe`, no test child remained, and no host-only wrapper is part of the repository diff. + - Evidence: `/tmp/issue3167-finaltree-full-test.log`, `/tmp/issue3167-finaltree-full-test.status` (`0`). +- `npm run lint`: passed with exit status 0 on the final formatted tree. Evidence: `/tmp/issue3167-final-lint.log` and `.status` (`0`). +- `npm run typecheck`: passed with exit status 0 before the final build. Evidence: `/tmp/issue3167-postocr-typecheck.log`. +- `npm run format`: passed with exit status 0. It formatted four issue-related files; the definitive full test and final lint gates ran afterward. Evidence: `/tmp/issue3167-postocr-format.log`. +- `npm run build`: passed with exit status 0. Evidence: `/tmp/issue3167-postocr-build.log`. +- Post-build `npm run typecheck`: passed with exit status 0 against freshly generated declarations. Evidence: `/tmp/issue3167-postbuild-typecheck.log`. +- `bun scripts/start.ts --profile-load stepfun-37 "write me a haiku and nothing else"`: passed with exit status 0 and returned a three-line haiku beginning `Terminal aglow`. + +### AC-7 convergence correction verification + +- Automatic retention now protects JSONL only when the filename has today's UTC + day-key and its mtime is within the maintenance interval. A fresh claim remains + protected and counted, but does not shield that run's old-day JSONL. +- Explicit `/perf delete` intentionally retains broader fresh-claim-to-JSONL + protection so a manual delete does not unlink a file another process may append. +- `bun test src/perf/`: 456 passed, 0 failed, 1,130 assertions across 26 files. + The deterministic 24×7 fixture starts over both caps and proves that five + historical files from a continuously claimed owner are evicted while its + current/recent file and claim survive. Evidence: + `/tmp/issue3167-retention-perf-tests.log` and `.status` (`0`). +- The post-correction full `npm run test` gate passed with exit status 0 using the + documented host-only Bun/JSC workaround. Evidence: + `/tmp/issue3167-remediation-full-test-final.log` and `.status` (`0`). A later + ordinary-JIT diagnostic run stalled only the first case in two unrelated agents + files; rerunning those complete files with JIT disabled passed 16/16. Evidence: + `/tmp/issue3167-retention-agents-rerun.log` and `.status` (`0`). +- `npm run lint` passed after the correction. Evidence: + `/tmp/issue3167-remediation-lint.log` and `.status` (`0`). After the final + comment-only clarification, direct ESLint over every changed TypeScript file + also passed (`/tmp/issue3167-retention-changed-eslint.status`, `0`). +- Current-tree `npm run typecheck` and `npm run build` passed. Evidence: + `/tmp/issue3167-retention-{typecheck,build}.status` (`0`). The final + post-tracking-update `npm run format` also passed; evidence: + `/tmp/issue3167-retention-format-final.status` (`0`). +- Current-tree StepFun smoke passed and returned a three-line haiku beginning + `Terminal hums bright`. Evidence: `/tmp/issue3167-retention-smoke.status` (`0`). + +## TUI verification + +- `bun scripts/tmux-harness.ts --script /tmp/issue3167-tmux-perf.json`: passed with exit status 0 in a real 100x28 tmux TTY. +- Bare `/perf` rendered the honest default-off message: `Perf telemetry is not active in this process`. +- `/perf inspect` rendered `Perf Inspect`, the canonical local directory, schema version 1, privacy/default-off details, and empty local artifact counts. +- `/perf report` rendered `Perf Report`, self-health, and `Files scanned: 0 (0 B)` through the production command loader. +- Captures: `/var/folders/qd/962lhrjj0232rjykgg3lgmrw0000gn/T/llxprt-tmux-harness-1786340605494/004-perf-default-off-screen.txt`, `009-perf-inspect-screen.txt`, and `014-perf-report-screen.txt`. + +## Mechanical and scope verification + +- `git diff --check`: passed. +- `bun scripts/check-eslint-guard.ts`: passed. +- `bun scripts/check-no-vitest.ts`: passed. +- `bun scripts/check-no-new-js-files.ts`: passed. +- `bun scripts/check-doc-placement.ts`: passed. +- Added-line scan found no TODO/HACK/STUB/placeholder marker, ESLint suppression, or TypeScript suppression. +- Diffs are empty for `.llxprt/`, `packages/agents/`, `.github/`, root dependency manifests/lockfiles, ESLint config, Prettier ignore, and Git ignore. +- No generated JUnit file is part of the issue diff or eligible for staging. +- No gzip/size sub-rolling, `contended`, `records_dropped`, retry threshold, child-ID arrays, FileOutput inheritance, or agents-to-telemetry dependency was introduced. + +## Acceptance-criteria sign-off + +- AC-1: P04 schema/writer/reader round trip verified. +- AC-2: P09/P12 default-off no-side-effect path verified. +- AC-3: P06 operation identity and continuation joins verified. +- AC-4: P06 seven exactly-once terminal statuses, including `superseded`, verified. +- AC-5: P07 direct phases and overlap semantics verified. +- AC-6: P05 Ink render versus stdout write behavior verified. +- AC-7: P08 retention, claim, concurrent-writer, clock, live-writer, and unlink behavior verified. +- AC-8: P04 fail-open and rate-limited EACCES/EROFS/ENOSPC behavior verified. +- AC-9: P11 streaming report, token join, inspect, delete, and `/perf` behavior verified. +- AC-10: P10 memory omission and both read-time slopes verified. +- AC-11: P10 fixed-capacity ring and continued monitoring verified. +- AC-12: P12 enabled/disabled p50/p95/p99 observer-effect harness verified without a wall-clock threshold. diff --git a/project-plans/issue3167/acceptance-criteria.md b/project-plans/issue3167/acceptance-criteria.md new file mode 100644 index 0000000000..3e93bf70f9 --- /dev/null +++ b/project-plans/issue3167/acceptance-criteria.md @@ -0,0 +1,321 @@ +# Acceptance Criteria — Client-Side Performance Telemetry + +Plan ID: PLAN-20260808-PERFTREND +Issue: #3167 +Binding source: `specification.md` (§1–§9). Where `PLAN.md` REQ text is broader, +spec §9 reduced delivery wins; excluded items are marked **[EXCLUDED]**, optional +compression **[DEFERRED]**. + +**Preflight correction:** source fact-checking found blockers against the original +artifacts; the resolved implementation contract is recorded here as decisions +**D1–D8** and applied consistently across the domain model, pseudocode, phase +files, and execution tracker. The authoritative `specification.md` and `PLAN.md` +are **not** rewritten; where a resolved decision diverges from a spec detail +(notably D1 — child-id arrays removed; D2 — nested settings), that divergence is +recorded in the relevant AC. + +- **D1** child correlation — `operation_id` only; no `prompt_ids`/`turn_ids` + arrays/true-count on the record; join at read time. Child ids do **not** arrive + via `AgentEvent`. +- **D2** settings — nested `telemetry.perf.enabled` + `telemetry.perf.memory`. +- **D3** concurrent_instances — per-run claim file, lease-window semantics. +- **D4** PerfSink — no FileOutput inheritance; serialized no-drop promise chain. +- **D5** retention constants — derived from a P04 Bun record-size benchmark. +- **D6** fs-failure testing — package-private port; never real-disk fill/chmod. +- **D7** report baseline — exact version/sha; matched-dimension; unmatched never pooled. +- **D8** stdout observer — internal errors fail fast; no swallow. + +All planned tests are **Bun / `bun:test` only, behavioral, integration-first**, +and prove real outputs/state/files — never mock calls. Evidence tag format: +`EVIDENCE-ACn` referenced from phase files. + +--- + +## AC-1 — Single-schema real writer/read round trip (REQ-3167-5, -7) + +**Given** perf enabled and a real `PerfSink` writing to a tmpdir perf dir. +**When** N terminal operations complete. +**Then** exactly N `operation` records exist as one JSON object per line, +**each parseable by the real tolerant reader** (`parsePerfRecord`) back to the +exact field values the writer produced. + +- **Evidence**: a real write → fs read → parse loop; the round-trip asserts + against output produced by the actual writer (no hand-authored fixture). +- **Boundary**: record spanning midnight (UTC day-key rolls the file on the next + write); empty operation set writes nothing. +- **Anti-mock**: no `vi.mock(fs)`; use real files. + +## AC-2 — Default-off: no files, no listeners, no overhead (REQ-3167-8) + +**Decision (D2 — settings):** persisted shape is nested +`telemetry.perf.enabled` (master) and `telemetry.perf.memory`, both default +**false**; `telemetry.perf` is **not** a boolean. Memory requires the master. + +**Given** default settings (perf disabled). +**When** llxprt runs one or more operations. +**Then** **no** `perf-*.jsonl` file is created, **no** claim file is written, +**no** stdout observer is installed, **no** onRender wiring is added, **no** +memory ring is allocated. + +- **Evidence**: after a full scenario run with default settings, assert the perf + dir does not exist / is empty AND no extra write-proxy observer is active. + +## AC-3 — Identity joins and continuation grouping (REQ-3167-3, settled §3/§9) + +**Decision (D1 — child correlation):** the v1 perf record carries **no** child +`prompt_ids`/`turn_ids` arrays and **no** true-count/cap fields. Source +fact-checking found those child ids do **not** arrive in the CLI via `AgentEvent` +(event-types.ts defines ~20 event variants — text/tool/usage/done/… — none carry +a per-continuation `prompt_id`). Collecting them would require minting+propagating +ids through `packages/agents`, which is excluded. The reduced, zero-plumbing +resolution: `operation_id` (derived from the top-level prompt-id prefix) is the +sole join key, and the report performs the exact join at read time. + +**Given** one user submission producing an initial prompt id +`${sessionId}#agentic-loop#${uuid}` and continuations `…#continuation#1`, +`…#continuation#2` (real `AgenticLoop.generateContinuationPromptId`). +**When** the operation record is produced. +**Then** `operation_id === deriveOperationId(initialPromptId) === initialPromptId` +(no continuation suffix); the record carries **no** child-id arrays; and +`runtime_id`/`parent_runtime_id`/`subagent_name` reuse #3130's key names verbatim +so the perf log joins the token-usage log and session recording. + +- **Evidence**: a real continuation stream through the integrated recorder; + assert the prefix invariant `deriveOperationId(c) === initialPromptId` for + every continuation id observed, and assert the produced perf record contains + **no** `prompt_ids`/`turn_ids`/`prompt_ids_total`/`turn_ids_total` fields. +- **Read-time join evidence (D1)**: write one perf `operation` record, plus the + corresponding token-usage rows (one per continuation, each carrying its own + `prompt_id`); the report derives `operation_id` from each token row's + `prompt_id` and joins every continuation row to the **single** perf operation + — proving multi-continuation rows join to one perf operation **without** any + child id copied into the perf record. +- **Separate behavioural test**: `operation_id = promptId.split('#continuation#')[0]` + so a future change to `generateContinuationPromptId` fails loudly (the existing + `agenticLoop.prompt-id.test.ts` is extended, not duplicated). +- **[EXCLUDED]** minting+propagating operation_id through `packages/agents`; + [EXCLUDED] a new agents→telemetry dependency edge; [EXCLUDED] `prompt_ids`/ + `turn_ids` arrays and true-count/cap fields on the perf record. + +## AC-4 — Every terminal operation status incl. superseded (REQ-3167-3) + +**Given** the operation lifecycle registry wired into `useSubmitQuery` acquire/ +release + a finalisation sweep. +**When** an operation terminates via each path. +**Then** a record is written **exactly once** with the correct `status`: +`completed`, `error`, `cancelled_before_send`, `cancelled_during_api`, +`cancelled_during_tool`, `cancelled_during_approval`, `superseded`. + +- **Evidence**: drive each path through the integrated lifecycle (fixture + provider + abort controllers); assert one record per path with the right + status. **Superseded** specifically: a newer turn displaces the AbortController; + assert the displaced op is finalised as `superseded` exactly once even though + its guarded `finally` (isCurrentTurn==false) never runs. +- **Boundary**: double-finalise is a no-op (exactly-once). + +## AC-5 — Direct phase measurement + overlap semantics (REQ-3167-1, -2) + +**Given** a streaming operation with concurrent/nested provider and tool activity. +**When** the record is produced. +**Then** each client phase (`client_prepare_ms`, `stream_handler_ms`, +`ink_render_ms`, `stdout_write_sync_ms`, `client_finalize_ms`) is a directly +measured non-negative duration; `provider_attempt_sum_ms`/`provider_union_ms` and +`tool_call_sum_ms`/`tool_union_ms` are reported separately; and the record does +NOT claim they sum to elapsed. `unclassified_elapsed_ms` is the honest residual, +reported (not clamped, not labelled "llxprt time"). + +- **Evidence**: a real streaming turn; assert no client phase is computed by + subtraction of provider/tool; assert `provider_union_ms ≤ provider_attempt_sum_ms` + is allowed (union can be < sum when retries overlap); assert + `agent_activity_union_ms` is the merged provider∪tool union. +- **[EXCLUDED]** `llxprtMs = elapsed − provider − tool` (algebraically invalid). + +## AC-6 — Ink render/write distinction + stdout overload/backpressure/bytes (REQ-3167-4) + +**Decision (D8 — stdout observer):** an internal observer/programming error is +**never** swallowed. There is no try/catch around the internal observer callback; +it fails fast. Only the **filesystem writer** (external I/O) fails open. The +previous "an observer that throws must not corrupt the write (fire-and-forget)" +claim is removed. + +**Given** the stdout observer installed on the **interactive** Ink instance only. +**When** Ink renders frames and writes bytes. +**Then** `ink_render_count` (onRender passes) ≠ `stdout_write_calls` (write +invocations); `stdout_bytes` counts **encoded bytes** (`Buffer.byteLength`/ +`Uint8Array.byteLength`, never string length); the write wrapper preserves every +overload (string|Uint8Array, encoding, callback) and the backpressure boolean; +and Zed's separate `createInkStdio()` produces **uncounted** writes. + +- **Evidence**: real Ink render with a fixture stream; assert render_count and + write_calls diverge on a coalesced/throttled frame; assert a `Uint8Array` write + counts its byte length, not its property count; assert the wrapper returns the + real `writeToStdout` boolean (backpressure) and invokes callbacks. +- **Fail-fast boundary**: an internal observer that throws propagates (no + swallow); filesystem writer failures remain fail-open as external I/O (AC-8). + +## AC-7 — Retention under concurrency / 24×7 / clock changes / live writers / unlink failures (REQ-3167-6) + +**Decision (D3 — concurrent_instances / claim files):** a per-run claim file in +the global perf dir is touched by the single owned coarse maintenance interval +and removed on clean dispose. `concurrent_instances` counts non-stale claims at +operation finalization; the name is kept but it has **lease-window semantics with +bounded crash overshoot** (a crashed run leaves a stale claim until the next +maintenance sweep). Claim files are included in retention artifact accounting but +are **never mistaken for JSONL records**. + +**Given** the eventual-bound retention with live-writer safety and claim-file +concurrency accounting. +**When** records accumulate under: many concurrent writers; a long-running +process (maintenance on roll + coarse interval, not startup-only); a clock step +backwards then forwards; an active live writer; and an `unlink` that fails. +**Then** the bound is enforced as an **eventual bound with documented overshoot** +that explicitly permits active-day and claim overshoot; a live writer's file +(today's day-key, mtime within the maintenance window) is **never** deleted; +failed unlinks do not corrupt accounting (decrement only on success); and the +guarantee degrades to "no further growth" on a read-only/full volume. + +- **Evidence**: real multi-writer tmpdir; assert (a) a file with today's day-key + and recent mtime survives a sweep that evicts older files; (b) a 24×7-style + process triggers maintenance via the coarse interval without restart; + (c) a filesystem-port-injected unlink failure leaves accounting intact (D6); + (d) total bytes eventually falls under the cap after enough evictions (overshoot + documented, not zero); (e) claim files count toward artifact accounting but are + not parsed as JSONL records. +- **Boundary**: materially-future mtime delays eligibility (benign). +- **[EXCLUDED]** instantaneous no-loss hard cap (impossible with no coordination); + [EXCLUDED] a drift-probe timer. + +## AC-8 — Writer fail-open + bounded/rate-limited diagnostics under EACCES/EROFS/ENOSPC (REQ-3167-7) + +**Decision (D4 / D6 — sink + fault injection):** PerfSink uses a serialized +no-drop promise chain (it does **not** inherit FileOutput's bounded/drop queue). +Internal observer/programming errors fail fast; only **filesystem** +persistence/maintenance errors fail open and are rate-limited. Tests never fill +the real disk and do **not** rely on chmod semantics (non-portable). They use a +narrow package-private filesystem port / deterministic failing file handle to +produce EACCES/EROFS/ENOSPC — boundary fault injection, not mock theater. + +**Given** a writer facing EACCES, EROFS, or ENOSPC on append. +**When** perf attempts to write. +**Then** the session is unaffected (no throw escapes to the operation path); +diagnostics are **rate-limited** (not unbounded `console.error`); and the +serialized no-drop write chain provides its own back-pressure. + +- **Evidence**: real tmpdir + a narrow package-private filesystem port (or + deterministic failing file handle) that injects each errno at the append + boundary; assert the operation completes normally and that repeated failures + emit at most one diagnostic per rate-limit window to stderr. Separate real-file + integration tests prove the actual round-trip and concurrency behavior. +- **[EXCLUDED]** a `records_dropped` counter; [EXCLUDED] a retry-threshold + self-disable state machine; [EXCLUDED] FileOutput's bounded-queue drop policy. + +## AC-9 — Cross-platform streaming consumer + inspect/delete + /perf (REQ-3167-9) + +**Decision (D7 — report baseline):** `--baseline` accepts an exact `llxprt_version` +or `git_sha`. Without it, the report prints grouped matched-dimension p50/sample/ +self-health data and **no** delta. With it, each non-baseline version/commit group +is compared **only** against the selected baseline rows sharing +provider/model/render-mode/terminal-geometry buckets; unmatched groups are +reported as **unmatched, never pooled**. `/perf` is a current-process snapshot; +the report is longitudinal. `inspect` shows path/schema/privacy/record counts; +`delete` respects live claims (D3). + +**Given** accumulated real perf files (mixed schema versions, a malformed line, +and a truncated final line from a simulated SIGKILL). +**When** the report streams them. +**Then** it dispatches on `schema_version`, **ignores unknown fields** (a field +add is not a bump), **skips + counts** records above the known version (never +coerces), **tolerates the truncated final line** (counts it), groups by +version/commit within matched dimensions, reports self-health (skipped/truncated +counts, last write error, evictions — **not** records_dropped), and works +cross-platform (no shell pipeline). + +- **Evidence**: real multi-version fileset in tmpdir; assert known-version + records parse, unknown-version records are counted-not-crashed, the truncated + tail is counted, and the report output includes the counts. `inspect` shows the + dir/schema/privacy/record counts; `delete` removes files respecting live claims. +- **Baseline**: without `--baseline` → grouped p50/sample/self-health, no delta; + with `--baseline ` → matched-dimension delta vs baseline only, + unmatched groups reported as unmatched. +- **[EXCLUDED]** gzip; [EXCLUDED] `contended` (use `concurrent_instances`). + +## AC-10 — Memory fields omitted when disabled; two read-time slopes when enabled (§7) + +**Decision (D2 — settings):** the persisted shape is **nested** +`telemetry.perf.enabled` and `telemetry.perf.memory`, both default **false**; +memory requires `enabled`. `telemetry.perf` is **not** itself a boolean. (Note: +spec §7.4 names the master `telemetry.perf`; the resolved persisted contract is +the nested shape below — recorded here because spec is not rewritten.) + +**Given** perf master (`telemetry.perf.enabled`) on. +**When** `telemetry.perf.memory` is **off**. +**Then** operation records **omit** the memory columns (absent, not zero) and no +`memory_sample` records are written; the 60 s monitor reverts to warn-only. +**When** memory is **on**. +**Then** operation records carry the four memory columns; `memory_sample` rows +carry `uptime_ms` + `ms_since_last_operation`; and the reader can derive **two +slopes** (per-operation on `session_operation_index`; per-minute on `uptime_ms` +using the sample rows). Slopes are derived at read time, never stored. + +- **Evidence**: real records with memory on/off; assert field **presence/absence** + (not values==0); assert the per-minute slope uses idle samples + (`ms_since_last_operation` large) to expose a leak signature. +- **[EXCLUDED]** a stored slope; [EXCLUDED] a new timer; [EXCLUDED] memory + collected when the master is off. + +## AC-11 — Fixed-capacity sample ring + continued warning monitor (§7.3) + +**Given** the extended 60 s monitor. +**When** it runs for many ticks. +**Then** the live `/perf` ring is **fixed-capacity with overwrite** (never a +growing array); and the monitor **continues sampling after warning once** (the +warn-once latch is separated from the sampling loop — it no longer +`clearInterval`s itself). + +- **Evidence**: drive M > CAPACITY ticks; assert the ring length == CAPACITY and + oldest entries are overwritten; assert the interval is still active after a + warning fired. +- **[EXCLUDED]** piggybacking Footer.tsx's 2 s interval. + +## AC-12 — Observer-effect harness: enabled/disabled, p50/p95/p99, no wall-clock assertion (PLAN §"Measured facts" withdrawal) + +**Given** the Bun harness exercising the **real** integrated pipeline. +**When** a streaming-load scenario runs perf-enabled and perf-disabled. +**Then** the harness **reports** p50/p95/p99 per-op overhead for both and the +delta, and **asserts stable invariants only**: disabled ⇒ no perf file; enabled ⇒ +record count == operation count; disabled path produces zero side-effects. It +does **not** assert an unstable wall-clock threshold. + +- **Evidence**: harness output prints the percentiles as evidence; assertions + gate only the invariants. +- **[EXCLUDED]** a µs/turn budget asserted as a pass/fail gate. + +--- + +## Scope ledger (resolved against PLAN.md using spec §9) + +| Item | Status | +|---|---| +| Schema + Zod single declaration (writer & reader derive) | **Accepted** | +| PerfSink reuses narrow file/path/append primitives; does **not** inherit FileOutput's bounded/drop queue (D4) | **Accepted** | +| IntervalUnion extracted + incremental duration | **Accepted** | +| operation_id derived from prompt-id prefix; **no** child-id arrays in the record (D1) | **Accepted (halved + reduced)** | +| Operation lifecycle registry incl. superseded | **Accepted** | +| Directly measured client phases + honest residual | **Accepted** | +| Ink onRender + interactive-only stdout observer; observer fails fast (D8) | **Accepted** | +| Retention eventual-bound + live-writer safe; claim-file concurrency accounting (D3) | **Accepted** | +| Tolerant streaming reader + report + /perf + inspect + delete | **Accepted (plain, no gzip)** | +| Settings nested `telemetry.perf.enabled` + `telemetry.perf.memory` (D2) | **Accepted** | +| Retention constants derived from P04 Bun record-size benchmark (D5) | **Accepted** | +| Fault-injected fs failures via package-private port, never real-disk fill/chmod (D6) | **Accepted** | +| Report `--baseline` exact version/sha; matched-dimension delta; unmatched never pooled (D7) | **Accepted** | +| Memory trend: two slopes, zero new timers, own off-switch | **Accepted (one PR)** | +| Overhead harness (real integration, no wall-clock gate) | **Accepted** | +| `contended` drift probe | **[EXCLUDED]** | +| `records_dropped` / bounded-queue drop policy | **[EXCLUDED]** | +| `prompt_ids`/`turn_ids` arrays + true-count/cap on the perf record | **[EXCLUDED — D1]** | +| retry-threshold self-disable state machine | **[EXCLUDED]** | +| gzip (reader + archive) | **[DEFERRED — optional, last]** | +| size sub-rolling | **[DEFERRED — optional, last]** | +| agents→telemetry dependency edge | **[EXCLUDED]** | +| mint+propagate operation_id through agents | **[EXCLUDED]** | diff --git a/project-plans/issue3167/analysis/domain-model.md b/project-plans/issue3167/analysis/domain-model.md new file mode 100644 index 0000000000..cb9c429b9c --- /dev/null +++ b/project-plans/issue3167/analysis/domain-model.md @@ -0,0 +1,252 @@ +# Domain Model — Client-Side Performance Telemetry + +Plan ID: PLAN-20260808-PERFTREND +Issue: #3167 +Status: analysis (derived from `specification.md` §1–§9, binding; `PLAN.md` requirements reconciled where they conflict) + +> This document models the *settled* (spec §9 reduced) delivery. Where +> `PLAN.md` REQ text is broader than `specification.md` §9, the narrower +> spec decision wins and the broadened part is recorded here as **excluded**, +> not silently dropped. +> +> **Decisions D1–D8 (source-verified blockers, resolved):** applied throughout. +> Where a decision diverges from a `specification.md` detail — D1 (no child-id +> arrays on the record; join at read time) and D2 (nested +> `telemetry.perf.enabled`/`.memory`, not a boolean `telemetry.perf`) — the +> divergence is recorded here and in `acceptance-criteria.md`; the authoritative +> spec is not rewritten. + +--- + +## 1. Scope resolution (spec §9 binding) + +| Concern | PLAN.md text | Settled decision (spec) | Status | +|---|---|---|---| +| `operation_id` | "minted at submission, propagated to every child send; new agents→telemetry edge" | Derived from prompt-id prefix at read/derive time: `promptId.split('#continuation#')[0]`. Zero plumbing; `packages/agents` untouched; **no agents→telemetry edge**. **D1:** no child-id arrays on the record — the join is performed at read time from token-usage `prompt_id` metadata. | **Accepted (halved + reduced)** | +| `contended` drift probe | comparison dimension + ~10 Hz probe | **Excluded** — `concurrent_instances` carries the signal at zero timer cost | Excluded | +| `records_dropped` | bounded-queue drop policy field | **Excluded** — no bounded-queue drop counter; one record/operation through a serialized chain | Excluded | +| retry-threshold self-disable | "disables itself after a defined failure threshold" | **Excluded** as a state machine — fail-open + rate-limited diagnostics only | Excluded | +| gzip in reader/consumer | "streams plain and gzipped records" | **Excluded** — plain records only | Excluded | +| size sub-rolling / gzip archive | Phase 7 compression | **Deferred** (optional, last; not a prerequisite) | Deferred | +| memory trend | (separate concern in PLAN) | **In scope, one PR** — §7 first-class | Accepted | + +**Net:** all accepted #3167 work — schema, writer, retention, lifecycle, client +phases, reader/consumer, `/perf`, settings, **and the memory trend** — ships as +**one issue / one PR**. + +--- + +## 2. Package layering (spec §4, §8 — verified against `package.json` edges) + +``` +storage < telemetry < core < agents < cli +``` + +| Package | Owns for this feature | Must NOT | +|---|---|---| +| `telemetry` | Zod schema (`perfRecords.ts`), the JSONL sink (PerfSink reuses narrow FileOutput primitives, does **not** inherit — D4), directory retention, the extracted `IntervalUnion` | import core/cli/agents | +| `core` | the stdout byte/duration counter hook only (`utils/stdio.ts` lives here and cannot move) | import cli; the writer stays in telemetry | +| `cli` | operation lifecycle + finalisation registry, Ink `onRender` wiring, stdout observer install, opt-in setting reads, `/perf` + report command, overhead harness | add an agents→telemetry edge | +| `agents` | **Nothing.** | acquire a telemetry dependency for this feature | + +No new dependency edges are added. The writer is reachable from core's stdout +seam because telemetry sits *below* core; the cli installs the observer it owns. + +--- + +## 3. Entities + +### 3.1 PerfOperationRecord (record_type: "operation") +One per terminal top-level operation. Fields per spec §1.1–§1.6. **D1 (resolved):** +the record carries **no** `prompt_ids`/`turn_ids` arrays and **no** +true-count/cap fields — source fact-checking confirmed child continuation ids do +not arrive in the CLI via `AgentEvent`, so collecting them would require plumbing +through `packages/agents`. `operation_id` (derived from the initial prompt-id +prefix) is the **sole** join key; the report derives the same `operation_id` from +token-usage/session-recording `prompt_id` metadata at read time to join +multi-continuation rows to one operation. Memory columns (§7) ride this record +and are **omitted when disabled**, never zero-filled. + +### 3.2 MemorySampleRecord (record_type: "memory_sample") +Bare sample: four memory values + `uptime_ms` + `ms_since_last_operation` + +envelope. Gives the per-minute axis. Emitted by the **existing** 60 s monitor, +never a new timer. + +### 3.3 IntervalUnion (extracted) +Currently private + quadratic (`recomputeDuration` per `add`). Extracted to its +own exported module in `telemetry`; duration maintained **incrementally** during +the sorted merge so each insert is O(n) worst-case with no full re-walk. Shared +by `SessionMetricsAggregator` (provider/tool unions) and the perf recorder +(provider/tool + agent-activity unions). + +### 3.4 PerfSink (reuses narrow FileOutput primitives; does NOT inherit — D4) +**D4 (resolved):** PerfSink does **not** extend `FileOutput` and does **not** +inherit its bounded/drop queue, batch+interval flush, or singleton machinery. +Narrow file/path/append primitives are extracted/reused where practical, while +`FileOutput`'s public singleton/debug behaviour is preserved unchanged. PerfSink +uses a **serialized no-drop promise chain** (own back-pressure, so **no +bounded-queue drop counter** — spec §9), **one exclusive-create day file per run +UUID** (`wx`), UTC roll on the next record after midnight, **no gzip and no size +sub-rolling**. stat-once + in-memory byte counter. Internal observer/programming +errors **fail fast**; only filesystem persistence/maintenance errors fail open +and are rate-limited. Directory retention bound by count + total bytes. Filename +`perf--.jsonl` in `Storage.getGlobalLogDir()/perf`. + +### 3.5 OperationLifecycleRegistry (cli) +Owns the finalisation sweep that the ownership release cannot provide: a +superseded turn never reaches `useSubmitQuery`'s guarded `finally` +(`isCurrentTurn` is false), so without a registry those operations are dropped. +Registers an operation at acquire (`useSubmitQuery` ~`:627`), finalises it +exactly once on any terminal status (completed / error / +cancelled_before_send / cancelled_during_api / cancelled_during_tool / +cancelled_during_approval / **superseded**). At finalisation it derives +`concurrent_instances` from non-stale claim files (D3). + +### 3.6 StdoutWriteObserver (core seam + cli install) +The byte/duration counter hook lives in `core/utils/stdio.ts`. The interactive +instance from `inkRenderOptions.ts` carries it; Zed's separate `createInkStdio()` +call does **not** (global application would double-count Zed). Counts **encoded +bytes** (`Buffer.byteLength`/`Uint8Array.byteLength`), sync invocation duration +only, and write-call count — distinct from Ink's `onRender` render passes. +**D8:** the internal observer callback is **not** wrapped in try/catch — an +observer/programming error fails fast. Only filesystem writer failures fail open +(external I/O). + +### 3.7 PerfReader / ReportConsumer (telemetry reader + cli command) +Tolerant JSONL stream reader: ignore unknown fields (field-add = no bump); skip ++ count records whose `schema_version` exceeds the known max; tolerate a +truncated final line (SIGKILL mid-append) by counting it; normalise unversioned +legacy to `{schema_version:0}`. Plain records only (no gzip). **D1 read-time +join:** the reader derives `operation_id` from each token-usage/session row's +`prompt_id` (`promptId.split('#continuation#')[0]`) and joins multi-continuation +rows to the single perf `operation` record sharing that `operation_id`. **D7 +report baseline:** `--baseline` accepts an exact `llxprt_version` or `git_sha`; +without it the report prints grouped matched-dimension p50/sample/self-health and +no delta; with it, each non-baseline group is compared only against baseline rows +sharing provider/model/render-mode/terminal-geometry buckets, and unmatched +groups are reported as unmatched, never pooled. `/perf` is a current-process +snapshot; the report is longitudinal. `inspect` shows path/schema/privacy/record +counts; `delete` respects live claims (D3). Self-health surfaces last write error, +evictions, skipped/truncated counts — **not** `records_dropped`. + +### 3.8 MemoryRing (cli, live /perf view) +Fixed-capacity overwrite ring fed by the 60 s monitor — the leak detector must +not leak. Bounded; never a growing array. + +### 3.9 PerfClaimRegistry (cli/telemetry — D3) +A per-run claim file in the global perf dir (`.claim`) is created when +perf is enabled, touched by the **single** owned coarse maintenance interval, and +removed on clean dispose. `concurrent_instances` is derived at operation +finalisation by counting **non-stale** claims (mtime within the maintenance +window); the name is retained but the value has **lease-window semantics with +bounded crash overshoot** (a crashed run leaves a stale claim until the next +maintenance sweep reaps it). This reuses the one owned maintenance timer — there +is **no drift-probe timer and no additional memory timer**. Claim files are +included in retention artifact accounting but are never parsed as JSONL records. + +--- + +## 4. Key invariants + +1. **operation_id = promptId.split('#continuation#')[0]** — recovers grouping + from existing ids and is the **sole** join key on the perf record (D1). No + `prompt_ids`/`turn_ids` arrays or true-count/cap fields exist on the record; + the report derives the same id from token-usage `prompt_id` metadata at read + time. A behavioural test asserts the prefix invariant so a future change to + `AgenticLoop.generateContinuationPromptId` fails loudly. +2. **No phase is computed by subtraction of provider/tool from elapsed.** + Client phases are directly measured; the residual is `unclassified_elapsed_ms`, + reported honestly, never clamped, never labelled "llxprt time". +3. **Provider/tool are sum + union pairs; they do NOT sum to elapsed.** +4. **Memory disabled ⇒ fields omitted, not zeroed.** A zero is indistinguishable + from a measurement; an absent field is unambiguous (§2 readers tolerate absent). +5. **Retention is an eventual bound with documented overshoot + live-writer + safety**, not an instantaneous no-loss cap. The bound explicitly permits + active-day and claim overshoot (D3/D5). A live writer's file is skipped when + day-key is today AND mtime is within the maintenance interval. +6. **Persistent perf telemetry is opt-in / default-off**, inspectable, deletable. +7. **Zero new timers.** Memory sampling rides the existing 60 s monitor; the one + owned maintenance timer also touches claim files (D3). +8. **Internal errors fail fast; only external I/O fails open (D8).** The stdout + observer callback has no try/catch; filesystem writer/retention errors fail + open and are rate-limited. + +--- + +## 5. State transitions — operation lifecycle + +``` +acquire ──► preparing ──► sending(api/tool/approval interleaved) ──► terminal + │ │ │ │ + │ ▼ ▼ ▔▔▔▔ + │ cancelled_before cancelled_during_* │ + │ _send (api/tool/approval) │ + │ │ + └──────────── superseded (newer turn replaced abortControllerRef) ───┘ + ↑ never reaches ownership release; needs registry sweep + terminal ∈ {completed, error, cancelled_before_send, + cancelled_during_api, cancelled_during_tool, + cancelled_during_approval, superseded} +``` + +Exactly-once finalisation per registered operation id, regardless of path. + +## 6. State transitions — memory monitor (existing, extended) + +``` +60s tick ──► rss check ──► [warn latch (once, unchanged behaviour)] + │ + ▼ + sample(rss/heap/external/arrayBuffers + uptime) + │ + ┌───────┴────────┐ + ▼ ▼ + push MemoryRing (if perf+memory on) emit memory_sample record +``` +Defects fixed: (1) separate the warn-once latch from the sampling loop (it no +longer `clearInterval`s itself after warning); (2) MemoryRing is fixed-capacity. + +## 7. Failure model (fail-fast in-process; defensive parsing only for external input) + +| Boundary | Behaviour | +|---|---| +| Reader on JSONL/filesystem (external) | Tolerant: skip+count corrupt/truncated/unknown-version; never throw | +| Writer fs error (EACCES/EROFS/ENOSPC) | Fail open (session unaffected); rate-limited diagnostic; **no** retry-threshold state machine, **no** records_dropped counter (D4) | +| Internal observer/programming error | **Fail fast** — no try/catch around the observer callback (D8) | +| Retention on read-only/full volume | Degrades to "no further growth"; eviction cannot run | +| Sink construction | Real exclusive create (`wx`); stat-once; no check-then-use existsSync | + +**Testing fs failures (D6):** tests never fill the real disk and do not rely on +chmod semantics (non-portable). A narrow package-private filesystem port or a +deterministic failing file handle injects EACCES/EROFS/ENOSPC at the append +boundary — boundary fault injection, not mock theater. Separate real-file +integration tests prove actual round-trip/concurrency behavior. + +No speculative guards. The only swallowed exceptions are external I/O +(filesystem) failures, which are fail-open and rate-limited. + +## 8. Open preflight decisions (resolved — D1/D3/D5) + +1. **Stdout observer injection mechanism** — `createInkStdio` gains an optional + observer; interactive instance carries it, Zed does not; module-scope + `sharedStdio` becomes lazy/cached so the late (settings-gated) install works. +2. **concurrent_instances (D3)** — a per-run claim file in the global perf dir is + touched by the single owned coarse maintenance interval and removed on clean + dispose; `concurrent_instances` counts non-stale claims at operation + finalisation. Lease-window semantics with bounded crash overshoot. One owned + maintenance timer; no drift-probe timer, no extra memory timer. +3. **prompt_ids/turn_ids (D1) — RESOLVED to no arrays.** Source fact-checking + confirmed child continuation ids do **not** arrive via `AgentEvent`; collecting + them would require an excluded agents→telemetry edge. The record carries + `operation_id` only; the join is performed at read time. +4. **Retention constants (D5)** — P04 first adds a Bun record-size benchmark for + the actual schema; P08 derives concrete max-bytes/max-files/maintenance-interval + /diagnostic-rate-limit defaults from that benchmark and operational evidence, + with no placeholders at implementation time. + +## 9. Out of scope (explicit) + +CI perf gating; tool-level attribution; OTLP export; session-cleanup/stale-lock +defects (#3164); gzip; size sub-rolling; drift probe; bounded-queue drop policy; +retry-threshold self-disable; `prompt_ids`/`turn_ids` arrays + true-count/cap on +the perf record (D1). diff --git a/project-plans/issue3167/analysis/pseudocode/01-schema-and-reader.md b/project-plans/issue3167/analysis/pseudocode/01-schema-and-reader.md new file mode 100644 index 0000000000..d594cd3d02 --- /dev/null +++ b/project-plans/issue3167/analysis/pseudocode/01-schema-and-reader.md @@ -0,0 +1,129 @@ +# Pseudocode 01 — Schema, derivation, and tolerant reader + +Plan ID: PLAN-20260808-PERFTREND +Applies to: `packages/telemetry/src/perf/perfRecords.ts` (new), reader in same package. + +**Contract-first.** Inputs: an `operation` payload (cli) + a `memory_sample` +payload (cli). Outputs: a versioned JSONL line. Reader input: arbitrary lines +from disk (external). Reader output: parsed record or null (never throws). + +``` +10: CONST PERF_SCHEMA_VERSION = 1 +11: CONST PERF_RECORD_TYPE_OPERATION = "operation" +12: CONST PERF_RECORD_TYPE_MEMORY_SAMPLE = "memory_sample" +13: +14: // --- Envelope (shared) --- +15: Zod PerfEnvelopeSchema = +16: schema_version: z.number() +17: record_type: z.enum(["operation","memory_sample"]) +18: ts: z.string() // ISO 8601, operation/sample end +19: +20: // --- Identity (reuses #3130 key names verbatim) --- +21: PerfIdentitySchema = +22: session_id: z.string() +23: operation_id: z.string() // DERIVED, see lines 60-66 +24: runtime_id: z.string() +25: parent_runtime_id: z.string().nullable() +26: subagent_name: z.string().nullable() +27: project_hash: z.string() +28: // D1: NO prompt_ids/turn_ids arrays and NO true-count/cap fields. The +29: // child continuation ids do not arrive via AgentEvent; operation_id is the +30: // sole join key. The report derives it from token-usage prompt_id at +31: // read time (see READ-TIME JOIN, lines 104-115). +29: +30: // --- Build identity (x-axis) --- +31: PerfBuildSchema = +32: llxprt_version: z.string() +33: git_sha: z.string() +34: runtime: z.string() +35: platform: z.string() +36: +37: // --- Comparison dimensions (compare like-with-like) --- +38: PerfDimensionsSchema = +39: provider: z.string() +40: model: z.string() +41: context_tokens: z.number() +42: output_tokens: z.number() +43: terminal_cols: z.number() +44: terminal_rows: z.number() +45: render_mode: z.string() +46: concurrent_instances: z.number() +47: +48: // --- operation record: discriminated union keyed on record_type --- +49: PerfOperationRecordSchema = PerfEnvelope ⊕ { record_type: "operation" } +50: ⊕ PerfIdentity ⊕ PerfBuild ⊕ PerfDimensions +51: ⊕ { status: z.enum([7 terminal values]) } // §1.3 incl. "superseded" +52: ⊕ client-measurement fields (see pseudocode 05, lines 10-30) +53: ⊕ provider/tool sum+union fields // see pseudocode 05 lines 40-55 +54: ⊕ { operation_elapsed_ms, approval_wait_ms, unclassified_elapsed_ms } +55: ⊕ memory fields (OPTIONAL — present iff memory enabled) // pseudocode 07 +56: ⊕ session_operation_index, uptime_ms +57: // D1: NO prompt_ids/turn_ids/totals here. operation_id (identity) is the +58: // sole join key. +58: +59: // --- operation_id derivation (settled §3 / §9 — NOT minted+propagated) --- +60: FUNCTION deriveOperationId(promptId: string): string +61: RETURN promptId.split("#continuation#")[0] +62: END +63: // First/initial prompt id has no continuation suffix ⇒ operation_id === it. +64: // Continuations are "...#continuation#n" ⇒ prefix recovered. Subagents use +65: // runtime_id/parent_runtime_id/subagent_name (separate namespace). +66: +67: // --- Tolerant reader (external input — defensive parsing justified) --- +68: FUNCTION isStringRecord(v): boolean // type guard +69: FUNCTION parsePerfRecord(line: unknown): PerfRecord | null +70: IF NOT isStringRecord(line) RETURN null +71: hasVersion = "schema_version" in line +72: hasType = "record_type" in line +73: IF NOT hasVersion AND NOT hasType +74: // legacy/unversioned → normalise to v0, validate as operation +75: normalized = { ...line, schema_version: 0, record_type: "operation" } +76: r = PerfOperationRecordSchema.safeParse(normalized) +77: RETURN r.success ? r.data : null +78: END +79: IF line.schema_version > PERF_SCHEMA_VERSION +80: // unknown future version → skip+count, NEVER coerce +81: RETURN null // caller counts via the "skipped" path +82: END +83: r = PerfRecordUnionSchema.safeParse(line) // discriminated on record_type +84: RETURN r.success ? r.data : null +85: END +86: +87: // --- Streaming line iterator with truncation tolerance --- +88: FUNCTION* readPerfLines(filePath): +89: open file for reading +90: leftover = "" +91: FOR EACH chunk in stream: +92: data = leftover + decode(chunk) +93: parts = data.split("\n") +94: leftover = parts.pop() // last partial line kept +95: FOR EACH line in parts: +96: yield parsePerfRecord(line) +97: // truncated final line: if leftover is non-empty AND not valid JSON +98: // → count as truncated (SIGKILL mid-append); if it IS valid, parse it. +99: IF leftover.trim() != "": +100: r = parsePerfRecord(leftover) +101: yield r // null ⇒ caller counts truncated +102: END +103: +104: // --- READ-TIME JOIN (D1 — zero-plumbing correlation) --- +105: // The perf operation record carries operation_id = deriveOperationId(initialPromptId). +106: // Token-usage / session-recording rows each carry their own prompt_id (one per +107: // send, incl. continuations). The report derives operation_id from each such +108: // prompt_id and joins multi-continuation rows to the SINGLE perf operation. +109: FUNCTION joinKeyFromPromptId(promptId: string): string +110: RETURN promptId.split("#continuation#")[0] // same derivation as line 61 +111: END +112: // Behavioural evidence: N continuation rows (prompt_ids S#..#U#continuation#1..N) +113: // all derive to operation_id === S#..#U and join to the one perf operation — +114: // with NO child id copied into the perf record. This avoids inventing an +115: // unobservable array (the child ids do not arrive via AgentEvent). +``` + +**Anti-patterns (must NOT):** +- `as PerfRecord` on external input (use the schema). +- Throw on a bad line (return null + count). +- Mint/propagate operation_id through agents (derive at line 60-66). +- Add `prompt_ids`/`turn_ids` arrays or true-count/cap fields to the perf record + (D1 — child ids do not arrive via AgentEvent; join at read time, lines 104-115). +- Write zeros for memory-disabled fields (omit them — line 56). diff --git a/project-plans/issue3167/analysis/pseudocode/02-perfsink-and-interval-union.md b/project-plans/issue3167/analysis/pseudocode/02-perfsink-and-interval-union.md new file mode 100644 index 0000000000..28af422e37 --- /dev/null +++ b/project-plans/issue3167/analysis/pseudocode/02-perfsink-and-interval-union.md @@ -0,0 +1,127 @@ +# Pseudocode 02 — PerfSink (reuses narrow FileOutput primitives; D4) + IntervalUnion extraction + +Plan ID: PLAN-20260808-PERFTREND +Applies to: `packages/telemetry/src/debug/FileOutput.ts` (refactored), new +`packages/telemetry/src/perf/PerfSink.ts`, extracted +`packages/telemetry/src/telemetry/intervalUnion.ts`. + +## Part A — IntervalUnion (extracted, incremental duration) + +``` +10: CLASS IntervalUnion: +11: private intervals: Array<[startMs, endMs]> // sorted, non-overlapping +12: private cachedDurationMs: number = 0 // maintained incrementally +13: +14: METHOD add(startMs, endMs): +15: IF endMs <= startMs RETURN // degenerate, ignore +16: // find insert position; merge overlapping/adjacent neighbours +17: // WHILE merging, SUBTRACT removed interval durations from cachedDurationMs +18: // THEN ADD the merged span to cachedDurationMs +19: inserted = mergeIntoSorted(intervals, startMs, endMs, ref cachedDurationMs) +20: END +21: +22: METHOD union(other): IntervalUnion +23: result = new IntervalUnion() +24: FOR EACH [s,e] in this.intervals: result.add(s,e) +25: FOR EACH [s,e] in other.intervals: result.add(s,e) +26: RETURN result +27: END +28: +29: METHOD durationMs(): number +30: RETURN cachedDurationMs // O(1), no full re-walk +31: END +32: +33: METHOD count(): number +34: RETURN intervals.length +35: END +``` +**Bug fixed (vs current private impl):** the current `add()` calls +`recomputeDuration()` walking every interval on each insert (O(n²) over a 24/7 +session). Lines 12/17-18 maintain the total incrementally. `SessionMetricsAggregator` +is refactored to import this exported class. + +## Part B — PerfSink (D4: reuses narrow FileOutput primitives; does NOT inherit) + +**D4 (resolved):** PerfSink does **not** extend `FileOutput` and does **not** +inherit its bounded/drop queue, batch+interval flush, or singleton. Narrow +file/path/append primitives are extracted/reused where practical while +`FileOutput`'s public singleton/debug behaviour is preserved. PerfSink uses a +**serialized no-drop promise chain** (one record per operation; own back-pressure +⇒ no drop counter), **one exclusive-create day file per run UUID**, UTC roll on +the next record, **no gzip and no size sub-rolling**. Internal observer/programming +errors fail fast; only filesystem persistence/maintenance errors fail open and are +rate-limited. + +``` +50: CLASS PerfSink: // CONSTRUCTIBLE — not singleton +51: private dir: string // Storage.getGlobalLogDir() + "/perf" +52: private runUuid: string // per-run id +53: private fileDayKey: string | null // YYYYMMDD from last record ts (UTC) +54: private currentPath: string | null +55: private bytesSinceStat: number // in-memory byte counter (stat once) +56: private writeChain: Promise = Promise.resolve() // serialized, NO drop +57: private lastDiagMs: number = 0 // rate-limit diagnostics +58: private disposed: boolean = false +59: +60: CONSTRUCT(dir, runUuid): +61: this.dir = dir; this.runUuid = runUuid +62: END +63: +64: METHOD write(record): // returns a Promise; never drops +65: IF disposed RETURN writeChain // no-op after dispose +66: dayKey = utcDayKey(record.ts) // UTC day from record's own ts +67: IF dayKey != fileDayKey OR currentPath == null: +68: rollToNewFile(dayKey) // UTC midnight-rollover on next record +69: END +70: payload = JSON.stringify(record) + "\n" +71: // serialized no-drop promise chain — own back-pressure, NO bounded queue +72: writeChain = writeChain +73: .then(() => appendFile(currentPath, payload, { mode: 0o600 })) +74: .then(() => { bytesSinceStat += Buffer.byteLength(payload) }) // in-mem count +75: .catch(err => failOpenDiag(err)) // ONLY filesystem errors fail open +76: RETURN writeChain +77: END +78: +79: PRIVATE METHOD rollToNewFile(dayKey): +80: // ONE exclusive-create day file per run UUID (no seq, no size sub-roll) +81: name = `perf-${dayKey}-${runUuid}.jsonl` +82: path = join(dir, name) +83: fd = openExclusive(path, "wx", mode 0o600) // O_EXCL: no check-then-use +84: currentPath = path +85: bytesSinceStat = 0 // new file ⇒ counter from zero +86: fileDayKey = dayKey +87: END +88: +89: METHOD dispose(): +90: disposed = true +91: await writeChain // drain the chain on clean exit +92: removeClaimFile(runUuid) // D3: remove claim on clean dispose +93: END +94: +95: // --- fail-open + rate-limited diagnostics (§9/D4: NO retry-threshold machine, +96: // NO records_dropped counter) --- +97: PRIVATE METHOD failOpenDiag(err): +98: now = Date.now() +99: IF now - lastDiagMs < DIAG_RATE_LIMIT_MS RETURN // throttle +100: lastDiagMs = now +101: writeToStderrRateLimited(`perf telemetry write failed: ${err.code ?? err.message}`) +102: // ONLY filesystem persistence errors are caught here. Internal observer / +103: // programming errors are NOT caught — they propagate (fail fast, D8). +104: END +105: +106: // stat-once: FileOutput stats EVERY flush; PerfSink stats only at exclusive-open +107: // (line 84) and counts bytes in memory thereafter (line 74). No gzip, no size +108: // sub-rolling — UTC day-key roll is the only segmentation. +``` + +**Anti-patterns (must NOT):** +- Inherit/extend `FileOutput` (D4 — reuse narrow primitives only; preserve its + public singleton/debug behaviour untouched). +- Carry over FileOutput's bounded queue / drop policy / batch+interval flush / + per-flush stat (lines 56/72-74 serialize without dropping; line 74 counts in + memory). +- A `records_dropped` counter or retry-threshold self-disable (excluded §9). +- `existsSync` check-then-create (line 83 uses exclusive `wx`). +- gzip or size sub-rolling (excluded §9 — UTC day-key only). +- Wrap internal observer/programming errors in try/catch (line 75 catches + filesystem errors only; D8). diff --git a/project-plans/issue3167/analysis/pseudocode/03-stdout-observer.md b/project-plans/issue3167/analysis/pseudocode/03-stdout-observer.md new file mode 100644 index 0000000000..86731676e8 --- /dev/null +++ b/project-plans/issue3167/analysis/pseudocode/03-stdout-observer.md @@ -0,0 +1,71 @@ +# Pseudocode 03 — Stdout write observer (core seam, cli install) + +Plan ID: PLAN-20260808-PERFTREND +Applies to: `packages/core/src/utils/stdio.ts`, `packages/cli/src/ui/inkRenderOptions.ts`. + +**Why core owns only this:** `utils/stdio.ts` lives in core and cannot move. +The interactive Ink instance is built here; Zed builds its own — the counter +must attach to the interactive instance ONLY. + +``` +10: // --- core: the seam (no cli import; cli supplies the observer) --- +11: INTERFACE StdoutWriteObserver: +12: onWrite(encodedBytes: number, syncDurationMs: number): void +13: END +14: +15: // createInkStdio gains an OPTIONAL observer; absence ⇒ current behaviour. +16: FUNCTION createInkStdio(observer?: StdoutWriteObserver): { stdout, stderr } +17: attach error handlers (unchanged) +18: inkStdout = new Proxy(process.stdout, { +19: get(target, prop, receiver): +20: IF prop === "write": +21: RETURN function write(...args): +22: // count encoded bytes WITHOUT altering the call +23: chunk = args[0] +24: encodedBytes = byteLength(chunk) // Buffer/Uint8Array, never str len +25: t0 = performance.now() +26: ok = writeToStdout(...args) // delegate; preserve overload/enc/cb/backpressure +27: syncDurationMs = performance.now() - t0 +28: observer?.onWrite(encodedBytes, syncDurationMs) // D8: NO try/catch — fail fast +29: RETURN ok +30: END +31: END +32: // ...rest unchanged (bind methods) +33: }) +34: inkStderr = new Proxy(...) // unchanged (no counting) +35: RETURN { stdout: inkStdout, stderr: inkStderr } +36: END +37: +38: // --- cli: lazy/cached interactive stdio (fixes module-scope blocker) --- +39: // inkRenderOptions.ts line ~24 currently: const sharedStdio = createInkStdio(); +40: // at MODULE SCOPE, before any settings exist. Replace with a lazy cache. +41: let sharedStdio: ReturnType | null = null +42: let sharedStdioObserver: StdoutWriteObserver | null = null +43: +44: FUNCTION setInteractiveStdoutObserver(observer: StdoutWriteObserver | null): +45: // called once perf telemetry is resolved (settings-gated), before first render +46: sharedStdioObserver = observer +47: sharedStdio = null // invalidate cache so next build carries it +48: END +49: +50: FUNCTION getInteractiveStdio(): +51: IF sharedStdio == null: +52: sharedStdio = createInkStdio(sharedStdioObserver ?? undefined) +53: END +54: RETURN sharedStdio +55: END +56: +57: // inkRenderOptions(config, settings) uses getInteractiveStdio() instead of +58: // the module-scope constant. Zed's runZedIntegration.ts keeps calling +58: // createInkStdio() with NO observer ⇒ its writes are uncounted. +``` + +**Anti-patterns (must NOT):** +- Globally monkey-patch `process.stdout.write` (Zed would be double-counted). +- Count `string.length` as bytes (line 24 uses byte length). +- Include drain/terminal-flush time in `stdout_write_sync_ms` (line 27 is the + synchronous `writeToStdout` invocation only). +- Wrap the observer callback in try/catch or swallow its exceptions (line 28 is a + direct call — D8: internal observer/programming errors fail fast; only + filesystem writer failures fail open as external I/O). +- Conflate write calls with Ink render passes (`onRender` covers renders). diff --git a/project-plans/issue3167/analysis/pseudocode/04-operation-lifecycle.md b/project-plans/issue3167/analysis/pseudocode/04-operation-lifecycle.md new file mode 100644 index 0000000000..3f006493a3 --- /dev/null +++ b/project-plans/issue3167/analysis/pseudocode/04-operation-lifecycle.md @@ -0,0 +1,78 @@ +# Pseudocode 04 — Operation lifecycle registry + identity collection + +Plan ID: PLAN-20260808-PERFTREND +Applies to: `packages/cli/src/...` (OperationLifecycleRegistry), `useSubmitQuery.ts`. + +**Settled (§3/§9, D1):** operation_id is DERIVED, not minted+propagated. The CLI +holds the initial prompt id (`turn.promptId`/`resolvedPromptId`) before +`runStream(...)`; `deriveOperationId` recovers grouping. Child prompt/turn ids are +**NOT collected** — they do not arrive via `AgentEvent`, so `operation_id` is the +sole join key and the report correlates at read time (pseudocode 01 lines +104-115). `concurrent_instances` is derived from claim files at finalization (D3). + +``` +10: TYPE OperationStatus = +11: "completed" | "error" +12: | "cancelled_before_send" | "cancelled_during_api" +13: | "cancelled_during_tool" | "cancelled_during_approval" +14: | "superseded" +15: +16: CLASS OperationLifecycleRegistry: +17: private active: Map // keyed by turn's signal +18: private sink: PerfSink | null // null ⇒ perf disabled +19: private finalised: WeakSet // exactly-once guard +20: +21: METHOD begin(turn): OperationHandle +22: signal = turn.abortSignal +23: op = new PendingOp({ +24: operationId: deriveOperationId(turn.promptId), // pseudocode 01 lines 60-66 +25: sessionId, runtimeId, parentRuntimeId, subagentName, projectHash, +26: startedAtMs: performance.now(), +27: // D1: NO promptIds/turnIds sets — child ids not collected +28: identity snapshotted from config/runtime/build, +29: phases: { prepare, streamHandler, finalize }, // see pseudocode 05 +30: providerIntervals: IntervalUnion, toolIntervals: IntervalUnion, +31: memory: sampled-at-end (optional), +32: }) +33: active.set(signal, op) +34: RETURN { signal, op } +35: END +36: +37: METHOD finalise(signal, status: OperationStatus): +38: op = active.get(signal) +39: IF op == null OR finalised.has(signal): RETURN // exactly-once +40: finalised.add(signal) +41: active.delete(signal) +42: IF sink == null: RETURN // disabled ⇒ no file +43: concurrentInstances = countNonStaleClaims(dir, now) // D3 — see pseudocode 06 +44: record = buildRecord(op, status, concurrentInstances) // pseudocode 05 lines 50-74 +45: sink.write(record) +46: END +49: +50: // --- wiring into useSubmitQuery (acquire ~:627, release ~:650-659) --- +51: // acquire: registry.begin(turn) -> handle +52: // pre-send failure / no-send path: +53: // registry.finalise(signal, "cancelled_before_send" | "error") +54: // during-API abort: registry.finalise(signal, "cancelled_during_api") +55: // during-tool abort: registry.finalise(signal, "cancelled_during_tool") +56: // approval reject: registry.finalise(signal, "cancelled_during_approval") +57: // normal completion: registry.finalise(signal, "completed") +58: // error: registry.finalise(signal, "error") +59: // SUPERSEDED sweep: a newer turn replaces abortControllerRef.current; +60: // the older signal's guarded finally (isCurrentTurn==false) never runs, +61: // so when begin() detects an already-replaced signal on a NEW acquire, +62: // it finalises the displaced op as "superseded" exactly once (line 42 guard). +63: // +64: // queued submission drain: each drained submission begins its own op; +65: // a requeued-and-later-consumed submission finalises the prior op as +66: // appropriate before beginning the new one. +``` + +**Anti-patterns (must NOT):** +- Mint+propagate operation_id through `packages/agents` (line 24 derives it). +- Collect child prompt/turn ids (D1 — they do not arrive via AgentEvent; the + record carries `operation_id` only). +- Hang finalisation off the ownership release alone (lines 59-62: superseded + never reaches it). +- Finalise twice (line 39 guard). +- Block the UI thread on finalisation (line 45 is the sink's serialized chain). diff --git a/project-plans/issue3167/analysis/pseudocode/05-client-phases.md b/project-plans/issue3167/analysis/pseudocode/05-client-phases.md new file mode 100644 index 0000000000..0733897a0c --- /dev/null +++ b/project-plans/issue3167/analysis/pseudocode/05-client-phases.md @@ -0,0 +1,76 @@ +# Pseudocode 05 — Directly measured client phases + record assembly + +Plan ID: PLAN-20260808-PERFTREND +Applies to: cli operation recorder; Ink `onRender`; stdout observer (pseudocode 03). + +**Core principle (§1.6, REQ-3167-1):** client phases are DIRECTLY measured; +`unclassified_elapsed_ms` is the honest residual, never clamped, never "llxprt time". + +``` +10: // --- measurement fields (all directly measured, additive among themselves) --- +11: client_prepare_ms // performance.now() delta: submit acquire → first send +12: stream_handler_ms // Σ synchronous CPU inside delta handling +13: ink_render_ms // Σ from Ink onRender (Ink computes it; pure accumulate) +14: ink_render_count // count of onRender callbacks (render passes) +15: stdout_bytes // Σ encoded bytes from stdout observer +16: stdout_write_calls // count of write invocations from observer +17: stdout_write_sync_ms // Σ sync invocation duration from observer +18: client_finalize_ms // performance.now() delta: last send → finalise +19: +20: // --- Ink onRender wiring (REQ-3167-4) --- +21: // VERIFIED against installed @jrichman/ink@6.4.8: RenderMetrics = { renderTime: number } +22: // (Ink computes performance.now() delta around the render computation). +23: // The pseudocode's earlier "renderDurationMs" name is corrected to "renderTime". +24: FUNCTION onRender(metrics: RenderMetrics): +25: op.ink_render_ms += metrics.renderTime // Ink's own clock (renderTime) +26: op.ink_render_count += 1 +27: END +25: // installed on the interactive Ink render options, per-operation accumulate. +26: // Writes and renders are DISTINCT: one write != one frame (Ink throttles/coalesces). +27: +28: // --- stream handler accumulation --- +29: FUNCTION onDeltaProcessed(syncCpuMs): +30: op.stream_handler_ms += syncCpuMs +31: END +32: +40: // --- provider/tool: sum + union (overlapping, NOT additive with client phases) --- +41: provider_attempts, tool_calls: counters +42: provider_attempt_sum_ms = Σ attempt durations +43: tool_call_sum_ms = Σ tool durations +44: provider_union_ms = providerIntervals.durationMs() // IntervalUnion +45: tool_union_ms = toolIntervals.durationMs() +46: agent_activity_union_ms = providerIntervals.union(toolIntervals).durationMs() +47: +50: // --- record assembly at finalise (REQ-3167-1/-2/-4) --- +51: FUNCTION buildRecord(op, status, concurrentInstances): +52: elapsedMs = op.startedAtMs → performance.now() +53: unclassified = elapsedMs +54: - client_prepare_ms - stream_handler_ms - ink_render_ms +55: - stdout_write_sync_ms - client_finalize_ms +56: - approval_wait_ms +57: // provider/tool unions are OVERLAPPING with client phases (work happens +58: // inside them), so they are NOT subtracted. unclassified may be small +59: // or large; report it HONESTLY, never clamp, never zero. +60: RETURN PerfOperationRecord({ +61: ...envelope, ...identity(op.operationId), ...build, +62: ...dimensions(concurrentInstances), // incl. concurrent_instances (D3) +63: status, +64: client_prepare_ms, stream_handler_ms, ink_render_ms, ink_render_count, +65: stdout_bytes, stdout_write_calls, stdout_write_sync_ms, client_finalize_ms, +66: provider_attempts, provider_attempt_sum_ms, provider_union_ms, +67: tool_calls, tool_call_sum_ms, tool_union_ms, agent_activity_union_ms, +68: operation_elapsed_ms: elapsedMs, approval_wait_ms, +69: unclassified_elapsed_ms: unclassified, +70: // D1: NO prompt_ids/turn_ids here — operation_id is the sole join key. +71: ...(memoryEnabled ? memoryColumns(op) : {}), // OMIT when disabled +72: session_operation_index, uptime_ms, +73: }) +74: END +``` + +**Anti-patterns (must NOT):** +- Compute any client phase as `elapsed − provider − tool` (the rev.1 error). +- Subtract provider/tool unions from elapsed (lines 57-59: overlapping). +- Clamp `unclassified_elapsed_ms` (line 59). +- Write zeros for memory when disabled (line 71 omits the field). +- Use Ink render count as a proxy for stdout writes (distinct — lines 14/16). diff --git a/project-plans/issue3167/analysis/pseudocode/06-retention.md b/project-plans/issue3167/analysis/pseudocode/06-retention.md new file mode 100644 index 0000000000..5dcf295586 --- /dev/null +++ b/project-plans/issue3167/analysis/pseudocode/06-retention.md @@ -0,0 +1,94 @@ +# Pseudocode 06 — Directory retention (eventual bound, live-writer safe) + +Plan ID: PLAN-20260808-PERFTREND +Applies to: `packages/telemetry/src/perf/retention.ts` (new), triggered by PerfSink. + +**Settled (§6, REQ-3167-6):** eventual bound with documented overshoot + live-writer +safety — NOT an instantaneous no-loss cap. Shape reused from +`errorReporting.rotateReports()` but NOT its weaker guarantees (it protects only +in-process paths and decrements accounting even when unlink fails). + +``` +10: // --- maintenance triggers: roll boundary + coarse interval (not startup-only) --- +11: METHOD maybeMaintain(now: number): +12: IF now - lastMaintenanceMs < MAINTENANCE_INTERVAL_MS RETURN +13: lastMaintenanceMs = now +14: maintain(now) +15: END +16: // called from PerfSink.rollToNewFile (pseudocode 02 line 68) AND on the +17: // coarse interval so a 24/7 process (never restarts) still bounds growth. +18: // The SAME interval touches this run's claim file (D3) — one owned timer, +19: // no drift-probe timer, no extra memory timer. +20: +21: // --- D3: per-run claim file (concurrent_instances accounting) --- +22: FUNCTION createClaim(dir, runUuid): // on perf enable +23: writeExclusive(join(dir, `${runUuid}.claim`), "", "wx", 0o600) +24: END +25: FUNCTION touchClaim(dir, runUuid): // by the maintenance interval +26: utimes(join(dir, `${runUuid}.claim`), now, now) // best-effort; fail-open +27: END +28: FUNCTION countNonStaleClaims(dir, now): number // ⇒ concurrent_instances +29: claims = readdir(dir).filter(name => name.endsWith(".claim")) +30: RETURN claims.filter(c => (now - stat(c).mtimeMs) <= CLAIM_LEASE_MS).length +31: END +32: // Lease-window semantics: a value counts a run as concurrent while its claim +33: // is fresh. A crashed run leaves a stale claim until the next sweep, so the +34: // count is a lease-window estimate with BOUNDED crash overshoot. Claim files +35: // are included in artifact accounting (below) but never parsed as JSONL. + +40: FUNCTION maintain(now: number): +41: files = readdirSortedByMtime(dir) // perf-*.jsonl AND *.claim +42: totalBytes = 0 +43: FOR EACH f in files: totalBytes += stat(f).size // claims counted, not parsed +44: IF files.length <= maxFiles AND totalBytes <= maxBytes: RETURN +45: +46: // evict oldest-first until BOTH caps satisfied +47: FOR EACH f in files (oldest first): +48: IF filesLeft <= maxFiles AND bytesLeft <= maxBytes: BREAK +49: IF isLiveWriter(f, now): CONTINUE // skip — see lines 62-70 +50: TRY: +51: await unlink(f) +52: bytesLeft -= stat(f).size // decrement ONLY on success +53: filesLeft -= 1 +54: CATCH err: +55: // do NOT decrement accounting on failure (rotateReports' defect) +56: logRateLimited(`perf retention unlink failed: ${err.code}`) +57: END +58: END +59: END +60: +61: // --- live-writer claim (no lock; pure function of filename + one stat) --- +62: FUNCTION isLiveWriter(file, now): boolean +63: dayKey = parseDayKeyFromName(file) // perf--... +64: IF dayKey != todayUtcDayKey(now): RETURN false // not today ⇒ not live +65: mtime = stat(file).mtimeMs +66: IF (now - mtime) <= MAINTENANCE_INTERVAL_MS: RETURN true // within window +67: RETURN false +68: END +69: // On a read-only/full volume unlink throws (line 54) ⇒ guarantee degrades +70: // to "no further growth" (new files still bound by per-day segmentation). +71: // The guarantee is eventual-with-overshoot, never instantaneous, and +72: // explicitly permits active-day and claim overshoot (D5). +``` + +**Documented overshoot sources (acknowledged, not hidden; eventual bound +explicitly permits active-day and claim overshoot — D5):** +- Concurrent appends between scan and delete (line 41→51). +- N active live files / fresh claims can collectively exceed the dir cap + (line 49 skips live files; fresh claims are not reaped). +- A crashed run leaves a stale claim until the next maintenance sweep + (bounded crash overshoot — D3). +- A materially-future mtime (NTP step) delays eligibility until `mtime + + interval` — a benign delay, not a correctness bug. + +**Anti-patterns (must NOT):** +- Decrement accounting on unlink failure (line 52 only on success). +- Delete a file whose day-key is today AND mtime within window (line 66). +- Parse a `.claim` file as a JSONL record (claims are accounting-only — D3). +- Add a drift-probe timer or an extra memory timer (D3 reuses the one owned + maintenance interval to touch claims). +- Assert zero loss + hard cap + no coordination simultaneously (impossible). +- Run only at startup (line 16: 24/7 process never restarts). +- Fill the real disk or rely on chmod to test failures (D6 — use a + package-private filesystem port / failing file handle; real files for + round-trip/concurrency). diff --git a/project-plans/issue3167/analysis/pseudocode/07-memory-trend.md b/project-plans/issue3167/analysis/pseudocode/07-memory-trend.md new file mode 100644 index 0000000000..49a7f8703a --- /dev/null +++ b/project-plans/issue3167/analysis/pseudocode/07-memory-trend.md @@ -0,0 +1,85 @@ +# Pseudocode 07 — Memory trend (in scope, one PR; zero new timers) + +Plan ID: PLAN-20260808-PERFTREND +Applies to: `packages/cli/src/ui/hooks/useMemoryMonitor.ts` (extend), PerfSink, +PerfOperationRecord memory columns, `/perf` live view. + +**Settled (§7):** two slopes (per-operation + per-minute) separate legitimate +growth (tracks work) from a leak (tracks uptime — the #3114 signature). Slopes +are DERIVED at read time, never stored. Zero new timers — extend the existing +60 s monitor. Two defects in it are fixed. Memory is independently disableable. + +``` +10: // --- existing monitor: useMemoryMonitor.ts --- +11: // MEMORY_CHECK_INTERVAL_MS = 60_000 (unchanged cadence) +12: // DEFECT 1 (line ~warning branch): clearInterval(intervalId) after warning +13: // once ⇒ it stops monitoring exactly when memory is known bad. +14: // FIX: separate the warn-once latch from the sampling loop. The interval +15: // keeps running; the latch only suppresses duplicate WARNINGS. +16: +17: let warnedOnce = false +18: FUNCTION on60sTick(): +19: sample = process.memoryUsage() // { rss, heapUsed, external, arrayBuffers } +20: IF sample.rss >= RSS_WARN_THRESHOLD AND NOT warnedOnce: +21: warnOnce() // UI warning (existing behaviour) +22: warnedOnce = true +23: END +24: // sampling continues regardless of the latch: +25: pushToMemoryRing(sample) // line 30 — bounded ring (defect 2 fix) +26: IF perfEnabled AND memoryEnabled: +27: emitMemorySample(sample) // line 40 +28: END +29: END +30: +31: // --- DEFECT 2: live view needs a bounded ring, not a growing array --- +32: CLASS MemoryRing: +33: private buf: MemorySample[] // FIXED CAPACITY (overwrite oldest) +34: private head: number = 0 +35: private len: number = 0 +36: METHOD push(s): +37: buf[head] = s; head = (head+1) % CAPACITY; len = min(len+1, CAPACITY) +38: END +39: METHOD snapshot(): MemorySample[] // ordered oldest→newest +40: END +41: // the leak detector must not leak. +42: +50: // --- memory_sample record (record_type discriminator earns its keep) --- +51: FUNCTION emitMemorySample(sample): +52: sink.write({ +53: record_type: "memory_sample", schema_version: PERF_SCHEMA_VERSION, +54: ts: new Date().toISOString(), +55: rss_bytes: sample.rss, heap_used_bytes: sample.heapUsed, +56: external_bytes: sample.external, array_buffers_bytes: sample.arrayBuffers, +57: uptime_ms: performance.now(), +58: ms_since_last_operation: performance.now() - lastOperationEndMs, +59: }) +60: END +61: // ms_since_last_operation makes an IDLE sample identifiable — idle samples +62: // are the ones that expose the #3114 "tracks uptime" leak signature. +63: +70: // --- memory columns on the operation record (per-operation axis) --- +71: // Present IFF perfEnabled AND memoryEnabled; OMITTED otherwise (never zeros). +72: FUNCTION memoryColumns(op): +73: s = process.memoryUsage() +74: RETURN { +75: rss_bytes: s.rss, heap_used_bytes: s.heapUsed, +76: external_bytes: s.external, array_buffers_bytes: s.arrayBuffers, +77: } +78: END +79: // sampled once at operation END; rides the record already being written +80: // ⇒ the per-operation axis is FREE (no extra sampling). +81: +90: // --- read-time slope derivation (never stored) --- +91: // per-operation slope: regress memory columns on session_operation_index +92: // per-minute slope: regress on uptime_ms using memory_sample rows +93: // a fix to the regression maths never requires re-collecting data. +``` + +**Anti-patterns (must NOT):** +- Add a new timer (line 11 reuses the 60 s interval). +- Piggyback Footer.tsx's 2 s interval (gated on showMemoryUsage + mounted ⇒ + would silently collect nothing). +- Store a computed slope (lines 90-93 derive at read time). +- Write zeros when memory disabled (line 72 omits the field). +- Let the ring grow unbounded (line 33 fixed-capacity overwrite). +- Stop the monitor after warning once (line 15 fix). diff --git a/project-plans/issue3167/analysis/pseudocode/08-consumer-and-perf-command.md b/project-plans/issue3167/analysis/pseudocode/08-consumer-and-perf-command.md new file mode 100644 index 0000000000..02a0935247 --- /dev/null +++ b/project-plans/issue3167/analysis/pseudocode/08-consumer-and-perf-command.md @@ -0,0 +1,103 @@ +# Pseudocode 08 — Reader/consumer + /perf command + settings + +Plan ID: PLAN-20260808-PERFTREND +Applies to: `packages/telemetry/src/perf/perfReader.ts`, cli report command, +`/perf` slash command, `TelemetrySettings` + `resolveTelemetrySettings`. + +**Settled (§9):** plain records only (NO gzip); contamination via +`concurrent_instances` (NO contended). Privacy-first: opt-in/default-off, +inspectable, deletable. + +``` +10: // --- settings (REQ-3167-8, D2): nested shape, both default false --- +11: INTERFACE TelemetrySettings: // configTypes.ts — ADD: +12: perf?: { enabled?: boolean; memory?: boolean } // nested: telemetry.perf is NOT a boolean +13: END +14: // resolveTelemetrySettings() hierarchy (unchanged): CLI flags > env > +15: // workspace .llxprt/settings.json > user settings > defaults (false). +16: FUNCTION resolvePerf(settings): { enabled, memory } +17: enabled = settings.telemetry?.perf?.enabled ?? false +18: memory = settings.telemetry?.perf?.memory ?? false +19: IF not enabled: RETURN { enabled: false, memory: false } // master gates memory +20: RETURN { enabled: true, memory } +21: END +22: // D2: the persisted master is telemetry.perf.enabled (not telemetry.perf). +23: // (spec §7.4 names the master `telemetry.perf`; the resolved persisted +24: // contract is the nested shape — recorded here because spec is not rewritten.) +22: +30: // --- streaming reader (cross-platform; works on Windows) --- +31: FUNCTION* streamPerfRecords(dir): +32: FOR EACH file in readdirSorted(dir): // perf-*.jsonl +33: FOR EACH r in readPerfLines(file): // pseudocode 01 lines 88-102 +34: yield r // null ⇒ caller counts skipped +35: END +36: END +37: END +38: // NO gzip. NO argument-limit breakage (streams one file at a time). +39: // NO abort-on-malformed-line (parsePerfRecord returns null; count it). +40: +50: // --- report (REQ-3167-9, D7): groups by version/commit within matched dims --- +51: FUNCTION buildReport(dir, baseline?): +52: counts = { total:0, skipped:0, truncated:0 } +53: groups = Map // groupKey = version|commit|dims +54: FOR EACH r in streamPerfRecords(dir): +55: IF r == null: counts.skipped++; CONTINUE +56: IF r is truncated-tail sentinel: counts.truncated++; CONTINUE +57: counts.total++ +58: groups.get(groupKeyOf(r)).push(measurementsOf(r)) +59: END +60: // D7 baseline: --baseline accepts an exact llxprt_version or git_sha. +61: // • no baseline → grouped matched-dimension p50/sample/self-health, NO delta +62: // • with baseline → each non-baseline group is compared ONLY against +63: // baseline rows sharing provider/model/render_mode/terminal-geometry +64: // buckets; unmatched groups reported as unmatched, NEVER pooled. +65: // matched-dimension comparison: p50 within (version,provider,model, +66: // render_mode, terminal size); contaminated via concurrent_instances>=2 +67: // (NOT contended — excluded). /perf = current-process snapshot; report = longitudinal. +68: RETURN { counts, groups, baselineDelta, selfHealth } +69: END +70: // D1 read-time join: token-usage/session rows are joined to a perf operation +71: // by deriving operation_id from each row's prompt_id (pseudocode 01 lines +72: // 104-115); multi-continuation rows join to one operation without child ids +73: // on the perf record. +74: +75: // self-health surfaced (NOT records_dropped — excluded): +76: // last write error, evictions, skipped/truncated counts. +68: +80: // --- /perf slash command (SlashCommand convention; subcommands) --- +81: perfCommand: SlashCommand = { +82: name: "perf", +83: subCommands: [ +84: // /perf → snapshot of THIS process (live MemoryRing + current op) +85: // /perf inspect → where data lives, what fields, sample counts +86: // /perf report → longitudinal buildReport() output +87: // /perf delete → remove all perf files (with live-writer safety) +88: ], +89: } +90: // registered in BuiltinCommandLoader alongside statsCommand/loggingCommand. +91: +88: // --- inspect (REQ-3167-8, D7): path/schema/privacy/record counts --- +89: FUNCTION perfInspect(): { dir, schemaVersion, privacy, fileCount, totalBytes, +90: operationCount, memorySampleCount } +91: +92: // --- delete (REQ-3167-8, D3): remove files respecting live claims --- +93: FUNCTION perfDelete(dir): +94: FOR EACH f in (perf-*.jsonl AND *.claim): +95: IF isLiveWriter(f, now): CONTINUE // pseudocode 06 lines 62-68 +96: IF isFreshClaim(f, now): CONTINUE // D3: respect live claims +97: unlink(f) // fail-open; count failures +98: END +99: END +``` + +**Anti-patterns (must NOT):** +- gzip streams (excluded §9). +- `contended` exclusion (use concurrent_instances, line 67). +- `records_dropped` in self-health (excluded; use skipped/truncated, line 76). +- Pool unmatched baseline groups (D7 — report them as unmatched). +- Require child ids on the perf record to join (D1 — derive at read time). +- Treat `telemetry.perf` as a boolean (D2 — nested `.enabled`/`.memory`). +- Default-on telemetry (line 17 default false). +- Collect memory without the perf master (line 19 gates). +- `gzcat | jq -s` one-liner (not cross-platform, breaks on arg limits, aborts + on malformed line — lines 31-39 stream and tolerate). diff --git a/project-plans/issue3167/analysis/pseudocode/09-overhead-harness.md b/project-plans/issue3167/analysis/pseudocode/09-overhead-harness.md new file mode 100644 index 0000000000..e2d2c90c53 --- /dev/null +++ b/project-plans/issue3167/analysis/pseudocode/09-overhead-harness.md @@ -0,0 +1,63 @@ +# Pseudocode 09 — End-to-end overhead harness (Bun, real integration) + +Plan ID: PLAN-20260808-PERFTREND +Applies to: `packages/cli` test (Bun/bun:test), exercising the REAL integration. + +**Settled (PLAN §"Measured facts" withdrawal):** no overhead claim ships until a +real end-to-end measurement exists, enabled and disabled, under streaming load, +reporting p50/p95/p99 event-loop impact — **without asserting unstable wall-clock +thresholds.** + +``` +10: // The harness exercises the ACTUAL integrated pipeline (NOT mocks): +11: // real PerfSink (tmpdir) + real stdout observer + real Ink onRender path +12: // + real operation lifecycle registry + a fixture streaming provider. +13: // +14: // It runs the SAME code paths a user runs, twice: perf ENABLED and DISABLED. +15: +20: FUNCTION runOverheadHarness(scenario): +21: // scenario drives N streaming turns through the integrated recorder with +22: // a fixture provider that emits a deterministic delta stream (no network). +23: samplesEnabled = measure(scenario, perfEnabled=true) +24: samplesDisabled = measure(scenario, perfEnabled=false) +25: RETURN { +26: enabled: percentiles(samplesEnabled), // p50/p95/p99 of per-op overhead +27: disabled: percentiles(samplesDisabled), +28: delta: compare(enabled, disabled), +29: } +30: END +31: +32: FUNCTION measure(scenario, perfEnabled): +33: set resolvePerf to {enabled: perfEnabled} +34: perOpOverheads = [] +35: FOR i in 0..N: +36: t0 = performance.now() +37: await scenario.runOneTurn() // real integrated recorder path +38: perOpOverheads.push(performance.now() - t0 - scenario.baselineCpuMs) +39: END +40: RETURN perOpOverheads +41: END +42: +43: // --- ASSERTION POLICY (critical) --- +44: // The harness REPORTS p50/p95/p99 and the enabled-vs-disabled delta. +45: // It does NOT assert a wall-clock threshold (those are CI-flaky and +46: // machine-dependent). Instead it asserts STABLE INVARIANTS: +47: ASSERT(perfDisabled ⇒ no perf file created) // default-off no files +48: ASSERT(perfEnabled ⇒ record count == operation count) // every op recorded +48: ASSERT(perfDisabled ⇒ overhead delta is within noise band)// not measurably costly +49: // The quantitative p50/p95/p99 are PRINTED as evidence, not gated. +50: END +51: +60: // --- observer-effect evidence: enabled path must not perturb disabled path --- +61: // The disabled path short-circuits at resolvePerf (line 16/19 of pseudocode 08): +62: // no sink construction, no observer install, no ring, no onRender wiring. +63: // The harness proves the disabled path produces ZERO side-effects (no files, +64: // no appended listeners) — the architectural guarantee, not a timing number. +``` + +**Anti-patterns (must NOT):** +- Mock the recorder/sink/observer (must exercise the real integration). +- Assert a fixed µs threshold (line 49 prints, never gates). +- Omit the disabled-path measurement (line 24 is the control). +- Run only the idle heap (the cost-relevance is under load — §7.3). +- Count wall time without subtracting the fixture baseline (line 38). diff --git a/project-plans/issue3167/execution-tracker.md b/project-plans/issue3167/execution-tracker.md new file mode 100644 index 0000000000..454dd31a5e --- /dev/null +++ b/project-plans/issue3167/execution-tracker.md @@ -0,0 +1,851 @@ +# Execution Tracker — Client-Side Performance Telemetry + +Plan ID: PLAN-20260808-PERFTREND +Issue: #3167 + +> **Decisions D1–D8 applied** across AC/domain/pseudocode/phase artifacts this pass +> (source-verified blockers resolved; preflight corrected). P03 remains COMPLETE. + +| Phase | ID | Status | Verified | Semantic? | Notes | +|---|---|---|---|---|---| +| 01 | P01 | DONE (planning) | yes | N/A | Preflight — seams verified; child-id AgentEvent claim FALSIFIED (D1); D1–D8 recorded | +| 02 | P02 | DONE (planning) | yes | N/A | Analysis + 9 pseudocode files + acceptance-criteria.md (D1–D8 applied) | +| 03 | P03 | DONE | yes | yes | IntervalUnion extracted (incremental O(1) durationMs) — UNAFFECTED by D1–D8 | +| 04 | P04 | COMPLETE | yes | yes | P04A+P04B: tightened schema boundaries + streaming reader API + PerfSink (serialized no-drop chain, exclusive 0600 day files, fail-open rate-limited diagnostics via fs port) + D5 benchmark — all 225 Bun behavioral tests green | +| 05 | P05 | COMPLETE | yes | yes | Stdout observer seam + Ink onRender wiring (observer fails fast — D8); 27 Bun behavioral tests green | +| 06 | P06 | COMPLETE | yes | yes | Operation lifecycle registry + identity (no child ids — D1; claims — D3); 38 Bun behavioral tests green. **Remediation:** D8 fail-fast finalisation (internal errors propagate, not swallowed); prep-rejection gaps (begin before prepareQueryForAgent/prepareTurnForQuery now finalise 'error' + preserve original rejection); queueWrite chain propagates internal rejections (fail-fast, no permanently non-rejecting shared chain). 41 Bun behavioral tests green | +| 07 | P07 | COMPLETE | yes | yes | Client phase measurement + record assembly (no capAndCount — D1). Default-off event dispatch (no timing when no observer), stale terminal cancellation evidence cleared on provider start. 26 contract + 32 behavior + 4 default-off tests green | +| 08 | P08 | COMPLETE | yes | yes | Retention (eventual bound, live-writer safe, claim files — D3/D5/D6); 44 Bun behavioral tests green; constants derived from D5 benchmark. **Remediation:** dispose() now propagates in-flight tick internal error after cleanup (try/finally); dual tick+cleanup internal errors aggregate via AggregateError; external errno remain fail-open. 47 Bun behavioral tests green | +| 09 | P09 | COMPLETE | yes | yes | Settings (opt-in, default-off, nested — D2); 64 Bun behavioral tests green; hierarchy fact-checked | +| 10 | P10 | COMPLETE | yes | yes | Memory trend (zero timers, ring, slopes); 31 Bun behavioral tests green | +| 11 | P11 | COMPLETE | yes | yes | Reader/consumer + report + /perf + inspect + delete (baseline — D7; join — D1); self-health (lastWriteErrorCode/evictionCount); injected live snapshot; shared artifact protection; 72 Bun behavioral tests green | +| 12 | P12 | COMPLETE | yes | yes | Integration wiring + overhead harness. Constructible InteractivePerfRuntime owner at CLI composition boundary; disabled returns null before any construction (AC-2 zero side effects). Owner owns PerfRetention+PerfSink+OperationLifecycleRegistry+optional MemoryTelemetryController+observers+snapshot capability. Startup installs observers BEFORE inkRenderOptions(). Ordered disposal with AggregateError. Threading through interactiveUI→AppWrapper→AppContainer→useAppInput→useAgentStream→useSubmitQuery (operationLifecycle) and useAppBootstrap→useMemoryMonitor (memoryController). Schema geometry corrected (terminal_cols/rows nonNegInt — unknown is zero). MemoryTelemetryController serialized write chain + drain(). useMemoryMonitor disabled path uses rss() not full memoryUsage(). OperationLifecycleRegistry getActiveOperationSnapshot(). BuiltinCommandLoader factory wiring (createPerfCommand with owned snapshot capability). Overhead harness prints p50/p95/p99/deltas (evidence, no wall-clock gate). **P12 Remediation:** PerfSink.dispose always runs writeChain drain + retention cleanup (try/catch per step, AggregateError); pre-start replacement via replacePreviousInstanceAndOwner() before buildAndStartPerfOwner (no observer conflict); shared production helper session/interactiveUiLifecycle.ts (cleanupInstanceAndOwner + rollbackInteractiveFailure) called by interactiveUI.tsx for pre-start replacement, registered global cleanup, AND post-render/setup-failure rollback — every cleanup step (clear/unmount/dispose; on rollback also owner.dispose, raw disableMouseEvents, mouse+restore listener removal, restoreTerminalProtocolsSync) runs independently even if a prior step throws, single Error or AggregateError, primary failure preserved first, no swallowing catches; render rollback now UNCONDITIONAL + non-swallowing (raw disableMouseEvents instead of swallowing mouseEventsExitHandler); setup-failure transactional catch added so a rendered instance/owner cannot leak if setupInstanceLifecycle/registerCleanup throws; startup rollback timer cleanup proven via counting scheduler (PerfScheduler seam); createIdentityProviderFromGetters takes immutable + mutable args (getter-based dynamic identity); buildAndStartPerfOwner typed to real TelemetrySettings (not unknown); disabled path test proves only getTelemetrySettings called; dynamic identity persistence test (provider/model/geometry mutate between operations); IS_REACT_ACT_ENVIRONMENT=true in all React test files (no act warnings); overhead harness uses REAL createInteractivePerfRuntime owner with owner.start() (genuine stdout/render/phase observer installation + claim + maintenance timer), deterministic disposal ordering (owner disposed BEFORE disabled workload; observers null, claim removed, timer clearCount >= 1 via CountingScheduler), disabled workload asserted via REAL on-disk artifact diff (no new JSONL/artifacts) not an empty local array, no operationLifecycle/observers in disabled, renderHook harnesses unmounted under act, accepted render mode (incremental) + `${process.platform}-${process.arch}` fixture, same deterministic fixture async stream + useSubmitQuery workload, prints p50/p95/p99/delta, no timing threshold; behavior tests call ACTUAL production helpers (replacePreviousInstanceAndOwner via tracked-state test seam __setTrackedInstanceAndOwnerForTesting, cleanupInstanceAndOwner, rollbackInteractiveFailure) — no mirrored/mock-theater code; slash runtime + BuiltinCommandLoader snapshot confirmed. 592 tests green across 34 files; CLI/telemetry/core typechecks clean; ESLint+Prettier clean; git diff --check clean. **P12 Focused Correctness Pass:** extracted shared session/interactiveUiLifecycle.ts helper; render rollback made unconditional/non-swallowing; setup-failure transactional leak fixed; behavior tests rewritten to exercise actual production helpers (no mirror/mock-theater); overhead harness rewritten to use real owner+owner.start with genuine observer install/dispose + real on-disk artifact diff; lifecycle (8) + overhead (1) + startInteractiveUI (9) tests green with clean stderr; CLI+telemetry typechecks clean; touched-file ESLint+Prettier clean; git diff --check clean. | +| 13 | P13 | COMPLETE | yes | yes | Definitive final-tree suite green (including 368/368 core, 561/561 providers, 365/365 agents, and 706/706 CLI files); final lint, typecheck before/after build, format, build, StepFun smoke, mechanical/scope guards, and real tmux `/perf`/`/perf inspect`/`/perf report` validation all green; evidence refreshed in `.completed/P13.md` | + +## Recommended next implementation phase +**P04 — Schema + PerfSink + tolerant reader + record-size benchmark.** P03 +(IntervalUnion) is COMPLETE. P04 lands the writer/reader contract and **first** +adds the Bun record-size benchmark (D5) whose output P08 uses to derive retention +constants. P04 depends on P03 only; P05/P09 may follow in parallel once the schema +lands, then the registry (P06) is the integration spine. + +## P04 progress — schema + derivation + reader + join + PerfSink + benchmark (COMPLETE) + +**Scope delivered (P04A only):** +- Single Zod schema source in `packages/telemetry/src/perf/perfRecords.ts`; + writer/reader/types all derive from it. `PERF_SCHEMA_VERSION=1`. +- `operation` + `memory_sample` discriminated record types; all operation + fields, identity/build/comparison fields, optional memory fields, and the + seven terminal statuses (incl. `superseded`). NO `prompt_ids`/`turn_ids`/ + true-count/cap fields (D1); NO `contended`, `records_dropped`, gzip, or size + rolling. Empty identity strings rejected at the schema boundary. +- `deriveOperationId`/`joinKeyFromPromptId` remove ONLY the exact terminal + `#continuation#` marker; an initial id is byte-identical. +- Tolerant streaming reader (`readPerfRecords`): streams without reading whole + files, tolerates malformed/truncated final lines with explicit counters + (parsed/malformed/futureVersion/unversioned/truncated/blank), ignores unknown + fields, skips+counts future versions without coercion, never throws on + malformed external JSONL, distinguishes malformed complete lines from a + truncated final line, counts unversioned records rather than fake-normalizing. +- Read-time join helper proven by behavioral tests: N continuation rows join to + one perf operation without copying child ids into the perf record (D1). +- Bun/bun:test behavioral tests only (real temp JSONL files; no mocks). +- Exported via `packages/telemetry/src/perf/index.ts` → package index + + `package.json` deep-import exports. + +**Deferred to P04B (NOW COMPLETE):** PerfSink (serialized no-drop promise chain, +D4), exclusive-create day files, fail-open + rate-limited diagnostics (D6/D8), +roundtrip/exclusive/failopen sink tests — all delivered in P04B. + +**P04B deliverables:** +- `PerfSink` in `packages/telemetry/src/perf/PerfSink.ts`: constructible, + non-singleton, does NOT inherit FileOutput. Serialized no-drop promise chain. + One exclusive-created 0600 file per run UUID per UTC record day + (`perf-YYYYMMDD-.jsonl`). Day from each record's `ts`; rolls on next + serialized record. Empty sink creates no file. Drain on dispose. No gzip, size + sub-roll, bounded queue, drop counter, retry threshold, or extra timer. +- Schema/programming/serialization errors fail fast (synchronous throw before + queueing). Only filesystem create/append/close errors fail-open (rate-limited). + Narrow package-private `PerfSinkFilesystem` port for deterministic + EACCES/EROFS/ENOSPC fault injection; default uses real `node:fs/promises`. +- Safe state transitions: failed exclusive open does not advance day/file state. + Concurrent `write()` calls preserve enqueue order and produce untorn JSONL + lines. A filesystem failure in one write does not poison later writes. Dispose + blocks further writes deterministically and drains all accepted writes. +- `FaultInjectingPerfFilesystem` exported for fault-injection tests (D6). + +**P04A corrections (delivered this pass):** +- A. Schema tightened to encode value boundaries: ISO 8601 `ts`; all durations + except `unclassified_elapsed_ms` finite + non-negative; `unclassified_elapsed_ms` + finite (may be negative); counts/tokens/index and terminal geometry are + non-negative integers (unknown geometry is zero); `concurrent_instances` is + an integer ≥ 1; bytes/memory/uptime/sample ages are finite + non-negative. + Unknown-field tolerance preserved. Boundary tests are in + `perfSchema.boundary.behavior.test.ts`. +- B. Genuinely streaming public reader API (`streamPerfRecords`): async generator + yielding `PerfStreamEntry` outcomes incrementally without accumulating the + file. Its `streamPerfFromReadable` test seam is package-private in the + non-exported `perfRecordsStream.ts` module. `readPerfRecords` delegates to the + streaming API as a bounded convenience collector. Large-file streaming and + incremental-yield evidence is in `perfReader.streaming.behavior.test.ts`. + +**D5 benchmark observed bytes** (`perfRecordSize.bench.ts`, actual v1 schema): +- `operation` record (WITH memory columns): **1220 bytes/line** +- `memory_sample` record: **242 bytes/line** +- combined per-operation pair: **1462 bytes** +- P08 derives retention constants (max-bytes/max-files/maintenance-interval/ + diagnostic-rate-limit) from these figures. + +## P05 progress — stdout observer seam + Ink onRender wiring (COMPLETE) + +**Scope delivered:** +- `packages/core/src/utils/stdio.ts`: `StdoutWriteObserver` interface + + optional `observer` param on `createInkStdio`. No observer ⇒ the Proxy + returns `writeToStdout` directly (same function identity/behaviour as today). + With an observer, a single `createObservedStdoutWrite` wrapper counts encoded + bytes (`Uint8Array.byteLength` / `Buffer.byteLength` with the supplied + encoding) and measures only the synchronous `writeToStdout` invocation + duration. Observer is called directly after the write returns — **no + try/catch** (D8: internal/programming errors fail fast). If the underlying + write throws synchronously, no observer sample is produced. Stderr remains + unobserved. No CLI import in core. +- `packages/cli/src/ui/inkRenderOptions.ts`: replaced the eager module-scope + `createInkStdio()` with a lazy cached interactive stdio seam + (`getInteractiveStdio` / `setInteractiveStdoutObserver`). Setting a different + observer invalidates the cache; the same value reuses the cached instance. + Zed's direct `createInkStdio()` call remains observer-free and uncounted. No + global `process.stdout` monkey patch. +- Ink `onRender` wiring (verified against installed + `@jrichman/ink@6.4.8`): `RenderOptions.onRender` is + `(metrics: { renderTime: number }) => void` — Ink **does** provide a real + render duration (`renderTime`, computed as `performance.now()` delta around + the render computation). The pseudocode's `renderDurationMs` field name is + corrected to `renderTime`. `InteractiveRenderObserver` + setter wired + conditionally (default-off: no observer ⇒ no onRender field in the returned + RenderOptions). Render passes stay distinct from stdout writes. +- Default-off: no observer setter call means no stdout/render observer + installed and no counter allocation. P09 owns persisted settings; no settings + wired yet. + +**Ink API correction (verified this pass):** the installed Ink package provides +`RenderMetrics = { renderTime: number }` (not `renderDurationMs`). The onRender +callback fires once per render pass (throttled by Ink's maxFps) with a real +duration. This is recorded in pseudocode `05-client-phases.md` line 22 and the +P05 phase file. Pseudocode `03-stdout-observer.md` is otherwise accurate (the +stdout seam signatures match the installed code). + +**Bun/bun:test behavioral evidence** (27 tests, 2 NEW files — no Vitest/Node +suites modified): +- `packages/core/src/utils/stdio.observer.behavior.test.ts` (16 tests): + multibyte UTF-8 byte counting with encoding, Uint8Array byteLength, write-call + count, finite/nonnegative duration, true/false backpressure passthrough + through the package-private `createObservedStdoutWrite` seam, both write + overloads, observer throw propagates (D8), underlying write throw ⇒ no sample, + absent observer identity unchanged, stderr uncounted, Zed path + characterization. +- `packages/cli/src/ui/inkRenderOptions.observer.behavior.test.ts` (11 tests): + lazy cache reuse, different-observer invalidation, same-observer reuse, + null-when-null no-op, default-off identity, onRender default-off, onRender + renderTime forwarding, onRender clear, render passes distinct from write + calls, existing options preserved. + +Artifacts outside `project-plans/issue3167/`: `packages/core/src/utils/stdio.ts`, +`packages/cli/src/ui/inkRenderOptions.ts`, and the two new test files. `.llxprt` +untouched. Authoritative design records NOT rewritten. + +## Pseudocode → phase map (line ranges) +| Pseudocode | Lines | Phase | +|---|---|---| +| 01-schema-and-reader | 10-66 (schema+derivation) | P04, P06 | +| 01-schema-and-reader | 67-102 (reader) + 104-115 (read-time join, D1) | P04, P11 | +| 02-perfsink-and-interval-union | 10-35 (IntervalUnion) | P03 | +| 02-perfsink-and-interval-union | 50-108 (PerfSink, D4) | P04 | +| 03-stdout-observer | 10-58 | P05 | +| 04-operation-lifecycle | 10-66 (D1: no child ids; D3: claims at finalise) | P06 | +| 05-client-phases | 10-74 (D1: no capAndCount) | P07 | +| 06-retention | 10-72 (D3 claim files; D5 overshoot) | P08 | +| 07-memory-trend | 10-93 | P10 | +| 08-consumer-and-perf-command | 10-24 (settings, D2) | P09 | +| 08-consumer-and-perf-command | 30-99 (reader/perf, D1 join + D7 baseline) | P11 | +| 09-overhead-harness | 10-64 | P12 | + +## Implementation status (this task) +P04 implementation COMPLETE (schema + derivation + tolerant reader + streaming +reader API + read-time join + PerfSink + D5 benchmark). Production code and Bun +behavioral tests land in `packages/telemetry/src/perf/`. 225 tests across 7 +files green. Artifacts outside `project-plans/issue3167/` are limited to the +`packages/telemetry/src/perf/` module, its package index/exports, and this +tracker. `.llxprt` untouched. Authoritative design records (`specification.md`, +`PLAN.md`, `decision.html`, `design.html`) NOT rewritten. P06–P13 remain TODO. + +**This pass (D1–D8 decision task):** source fact-checking found blockers in the +original artifacts and **falsified** the claim that child continuation ids arrive +in the CLI via `AgentEvent`. The resolved implementation contract is recorded as +decisions D1–D8 across `acceptance-criteria.md`, `analysis/domain-model.md`, the +nine pseudocode files, the thirteen phase files, and this tracker, while +preserving P03 as COMPLETE. Where a resolved decision diverges from a +`specification.md` detail (notably D1 — child-id arrays removed, and D2 — nested +`telemetry.perf.enabled`/`.memory` rather than a boolean `telemetry.perf`), the +divergence is recorded in the companion artifacts, not by editing the +authoritative spec. Contradictions between PLAN.md REQ text and spec §9 remain +resolved in favour of spec §9. + +## P09 progress — settings (opt-in, default-off, nested — D2) (COMPLETE) + +**Scope delivered:** +- `packages/core/src/config/configTypes.ts`: added `PerfTelemetrySettings` + interface (`{ enabled?: boolean; memory?: boolean }`); added `perf?: + PerfTelemetrySettings` to `TelemetrySettings`. +- `packages/core/src/config/configConstructor.ts`: exported + `resolveTelemetrySettings` (was private); added `resolvePerfSettings(settings): + { enabled: boolean; memory: boolean }` with master-gates-memory and +default-false semantics; `resolveTelemetrySettings` defensively clones perf +(`withClonedPerf`) on every ingress and egress so `Config.getTelemetrySettings()` +shallow copy + cannot leak nested mutable state. +- `packages/core/src/index.ts`: exports `resolvePerfSettings` + + `PerfTelemetrySettings` type for downstream phases (P06/P10/P12). +- `packages/core/src/config/config.ts`: **unchanged** (0 lines — the cloned perf + in the stored `telemetrySettings` prevents mutation through the shallow copy + returned by `getTelemetrySettings()`). +- `packages/cli/src/config/configBuilder.ts`: `buildTelemetryConfig` passes + `perf: telemetrySettings?.perf` through to Config. +- `packages/cli/src/config/settingsSchema.ts`: `TelemetrySettings` $def gains + `perf` property (type: object, additionalProperties: false, properties: + enabled/memory booleans with descriptions). +- `schemas/settings.schema.json`: regenerated `TelemetrySettings` $def with perf. +- `docs/telemetry-privacy.md`: added "Client Performance Telemetry" section + (default-off, local-only, master-gates-memory, settings JSON example). + +**Hierarchy fact-check (corrected):** the original plan's "CLI flag > env > +workspace > user > default" claim conflates persisted settings merge with +CLI/env mapping. Source fact-checking found: +- Persisted settings merge (`mergeSettings()`): shallow spread at the telemetry + level via `mergeObjectSection` — a higher-precedence layer's `perf` REPLACES + the lower-precedence `perf` entirely (not a deep merge of `perf.enabled` / + `perf.memory` across layers). Precedence: schema defaults < system defaults < + user < workspace (trusted) < system. +- CLI/env: yargs exposes ONLY flat flags (`--telemetry`, + `--telemetry-log-prompts`, `--telemetry-outfile`). There are **no CLI flags + or env vars for `telemetry.perf.*`** and none were added (the issue spec does + not require them). Perf is configured only via persisted settings files. +- `resolveTelemetrySettings` (core) does NOT implement a settings-layer + hierarchy — it applies per-field defaults and defensively clones perf +(`withClonedPerf`) on ingress and egress. + +**Bun/bun:test behavioral evidence** (44 tests, 4 NEW files — no Vitest/Node +suites modified): +- `packages/core/src/config/perfSettings.behavior.test.ts` (16 tests): + default-off, enabled-only, memory-gated-off, both-on, false-overrides, input + immutability (3), nested-return copy isolation (2), return type safety. +- `packages/core/src/config/telemetrySettingsCopy.behavior.test.ts` (6 tests): + perf is a copy not caller reference, resolved perf is a mutable isolated copy + (isolation by cloning, not freezing), input mutation isolation, caller not + mutated, undefined perf, field preservation. +- `packages/cli/src/config/perfSettingsMerge.behavior.test.ts` (8 tests): real + `mergeSettings` behavior — absent/user-only/workspace-replaces-user/ + both-set-wins/telemetry-scalar-coexist/untrusted-ignored/system-wins/ + system-defaults-overridden. +- `packages/cli/src/config/perfSettingsValidation.behavior.test.ts` (14 tests): + real Zod validation — 6 accepted shapes (object with enabled/memory, only + enabled, only memory, empty, both false, no perf), 8 rejected shapes (boolean + true/false, non-boolean enabled/memory, unknown properties, string/number/ + array). + +**Existing relevant tests verified unmodified:** `settings-validation.test.ts` +(69 tests) and `settingsSchema.previewFeatures.test.ts` (1 test) pass under Bun. + +Artifacts outside `project-plans/issue3167/`: `packages/core/src/config/ +configTypes.ts`, `configConstructor.ts`, `src/index.ts`; `packages/cli/src/ +config/configBuilder.ts`, `settingsSchema.ts`; `schemas/settings.schema.json`; +`docs/telemetry-privacy.md`; 4 new test files. `config.ts` **unchanged**. +`.llxprt` untouched. Authoritative design records NOT rewritten. + +## P08 progress — retention + claim lifecycle (eventual bound, live-writer safe — D3/D5/D6) (COMPLETE) + +**Scope delivered:** +- `packages/telemetry/src/perf/retention.ts`: constructible `PerfRetention` owner + (non-singleton). Owns exactly one coarse maintenance interval that touches + this run's UUID claim file AND performs oldest-first retention. Timer is + `unref`'d so it does not hold the CLI process open. No drift timer, no memory + timer. +- **Constants (D5)** derived from the P04 benchmark (1220-byte operation, + 242-byte memory_sample, 1462-byte combined pair): + - `PERF_MAX_BYTES = 64 MiB` (67,108,864) — ~45,902 operation pairs at 1462 + bytes/pair. + - `PERF_MAX_FILES = 128` — one file per writer per UTC day = 128 days at + single-writer volume. Claim files (0 bytes) count toward file count. + - `PERF_MAINTENANCE_INTERVAL_MS = 60,000` (60 s) — the owned coarse interval + and the live-writer protection window. + - `PERF_CLAIM_LEASE_MS = 180,000` (3 × interval) — a crashed run's claim + becomes stale within 3 minutes (bounded crash overshoot — D3). + - `PERF_DIAG_RATE_LIMIT_MS = 60,000` (60 s) — at most one diagnostic per + window for retention filesystem errors. + - **Cap binding at representative single-writer volume:** crossover is + MAX_BYTES / MAX_FILES = 524,288 bytes/file ≈ 359 operation pairs/day (memory + on). Below ~359 pairs/day, the **file cap binds** (128 days of data). Above + ~359 pairs/day, the **byte cap binds** (64 MiB reached before 128 days). At + typical interactive use (~50–200 ops/day), the file cap is the binding + constraint. +- **Claim lifecycle (D3):** claim created exclusively (`wx`, 0600) at `start()`; + touched every interval by `tick()`; removed on clean `dispose()`. A crash (no + dispose) leaves a stale claim until the next sweep. `countNonStaleClaims(now)` + for P06 derives `concurrent_instances` from non-stale claims (lease-window + semantics with bounded crash overshoot). +- **Retention (AC-7):** scans only owned artifacts (`perf-YYYYMMDD-*.jsonl` + + `*.claim`). Claims counted toward count/bytes but never JSONL-parsed. Evicts + oldest-first until BOTH caps satisfied. Protects a perf file only if its + filename day is today UTC AND mtime within maintenance interval. Protects every + non-stale claim. Stale claims eligible. Future mtimes remain protected until + eligibility. Decrement accounting ONLY on successful unlink. Stable + deterministic tie-break by name. Re-scan on later interval gives eventual + convergence. +- **Error policy (D8):** internal/programming errors fail fast. Only genuine + filesystem persistence/maintenance errors (create/touch/stat/readdir/unlink) + fail open and are rate-limited. +- **Filesystem port (D6):** narrow package-private `PerfRetentionFilesystem` + port + `FaultInjectingRetentionFilesystem` for deterministic + EACCES/EROFS/ENOSPC fault injection — never real-disk fill or chmod. + `PerfScheduler` / `PerfTimerHandle` package-private scheduler seam for + deterministic interval firing in tests. +- **PerfSink wiring:** optional `retention?: PerfRetention` on `PerfSinkOptions`; + `PerfSink.start()` starts the retention; roll boundary triggers + `maybeMaintain`; `dispose()` drains writes, then stops maintenance and removes + the claim. Backward-compatible: PerfSink without retention works without + `start()`. `FileOutput` unchanged. +- **Barrel cleanup:** removed `FaultInjectingPerfFilesystem` from the public + perf barrel (tests deep-import it from `PerfSink.js`); added retention + constants + `PerfRetention` / `PerfRetentionOptions` to the barrel. Kept + package-private fault/scheduler types out of the public barrel. +- **P05 comment correction:** fixed the inaccurate comment in + `packages/cli/src/ui/inkRenderOptions.ts` that claimed the render observer + "reads the current observer at invocation time" — it actually captures a local + closure at options-construction time, so a later clear does NOT take effect + without rebuilding the options object. + +**Bun/bun:test behavioral evidence** (44 tests across 3 NEW files — no Vitest/Node +suites modified): +- `retention.behavior.test.ts` (33 tests): constants (D5), claim lifecycle + (create/touch/dispose/crash-stale), countNonStaleClaims, live-writer safety, + claim handling (stale/fresh/future), oldest-first cap convergence + tie-break, + failed-unlink accounting intact + rate-limited diagnostics (D6), one coarse + interval touches claim + sweeps without restart, maybeMaintain rate-limiting, + future-mtime protection, claims never parsed as JSONL. +- `retention.capSelection.behavior.test.ts` (2 tests): file cap binds at + representative volume; byte cap binds at high volume. +- `perfSink.retention.behavior.test.ts` (9 tests): start creates only claim, + start-then-write, roll-boundary maintenance, disposal drains + stops + + removes claim, backward compatibility without retention, concurrent overshoot + convergence, countNonStaleClaims. + +All 269 perf tests green (225 P04/P05 + 44 P08). Typecheck, ESLint, Prettier +clean. Benchmark confirms constants. `git diff --check` clean. + +Artifacts outside `project-plans/issue3167/`: `packages/telemetry/src/perf/ +retention.ts` (new), `PerfSink.ts` (modified), `index.ts` (modified), 3 new test +files; `packages/cli/src/ui/inkRenderOptions.ts` (comment-only fix). `.llxprt` +untouched. Authoritative design records NOT rewritten. + +## P06 progress — operation lifecycle registry + identity (D1 no child ids, D3 claims) (COMPLETE) + +**Scope delivered:** +- `packages/cli/src/ui/hooks/agentStream/operationLifecycle.ts`: + `OperationLifecycleRegistry` — constructible CLI-owned registry keyed by + AbortSignal (not a global singleton). Disabled mode is the runtime not + constructing it (AC-2). +- `begin(signal, promptId)`: derives `operation_id` via `deriveOperationId` + (`promptId.split('#continuation#')[0]` — D1 binding correction), snapshots + immutable identity/build/dimensions through a narrow `OperationIdentityProvider`, + initializes monotonic per-session index, and creates a mutable per-operation + measurement state for P07. Returns a typed `OperationHandle`. No + prompt_ids/turn_ids collected (D1). A superseded sweep finalises every prior + still-active signal as `superseded` exactly once before admitting the new op. +- `finalise(signal, status)`: exactly-once async finalise. Atomically claims/ + removes the pending op (mark `finalised` WeakSet + delete from active map) + before awaiting external work. Derives `concurrent_instances` from + `PerfRetention.countNonStaleClaims(now)` (D3 lease semantics), clamped to a + schema-valid minimum 1 if filesystem fail-open yields zero. Builds one + schema-valid v1 operation record and writes through PerfSink. Duplicate/late + finalise no-ops. All seven statuses including `superseded`. +- `drain()`: awaits the serialized lifecycle chain so the runtime/tests can + deterministically flush pending writes before sink dispose. +- Identity: `OperationIdentitySnapshot` contract for session/runtime/project/ + build/provider/model/terminal/render fields. P12 supplies the provider; + tests use a fixture. No new CLI flags/env vars. Tokens begin at zero and + update through the typed P07 measurement handle. Timestamp uses wall-clock + ISO; elapsed/uptime uses monotonic clock. Memory columns omitted in P06. +- Record assembly: honest residual (`unclassified_elapsed_ms = elapsed − + directly-measured phases`) with zero/default P06 measurements equals elapsed. + Provider/tool sums and unions report zero. No P07 phase math. + +**Integration into real useSubmitQuery turn path:** +- `useSubmitQuery.ts`: optional `operationLifecycle?: OperationLifecycleRegistry` + dep (supplied by P12). `begin` after prompt ID resolution (after `initTurn`) + and before `prepareQueryForAgent`/send. `finalise` on normal completion + (`completed`), error (`error`), and pre-send-abort (`cancelled_before_send`) + paths. Fire-and-forget with error surfacing via `debugLogger.error` (AC-8: + no throw escapes to the operation path). The superseded sweep is triggered by + the new turn's `begin` — the stale turn's guarded `finally` (isCurrentTurn + false) never runs but cannot lose its record. Each consumed turn has its own + operation. No user-visible behavior change. + +**Bun/bun:test behavioral evidence** (38 tests across 2 NEW files — no Vitest/Node +suites modified): +- `operationLifecycle.behavior.test.ts` (32 tests): D1 split rule (5 — initial, + continuation #1/#2, non-terminal marker, CLI-fallback), no child arrays, + seven terminal statuses (7), duplicate finalise (3), superseded sweep (3), + concurrent_instances/D3 claims (3), session index monotonic, measurement + handle, record assembly (5 — identity/residual/subtracted-phases/wall-vs- + monotonic/memory-omitted), error policy (2 — schema fail-fast, filesystem + fail-open), each turn distinct operation. +- `useSubmitQuery.lifecycle.test.tsx` (6 tests): completed, error, pre-send- + abort, superseded (real useSubmitQuery control flow with deferred runStream + + terminal event displacement + new turn begin sweep), exactly-once, disabled + (no records when operationLifecycle absent). + +All 290 telemetry perf tests, 46 lifecycle+useSubmitQuery CLI tests, CLI/ +telemetry/core typechecks, ESLint, Prettier clean. `git diff --check` clean. +`git diff packages/agents` empty. + +Artifacts outside `project-plans/issue3167/`: `packages/cli/src/ui/hooks/ +agentStream/operationLifecycle.ts` (new), `operationLifecycle.behavior.test.ts` +(new), `__tests__/useSubmitQuery.lifecycle.test.tsx` (new), `useSubmitQuery.ts` +(modified — optional lifecycle dep + begin/finalise wiring). `.llxprt` +untouched. Authoritative design records NOT rewritten. P07 phase classification +deferred; P10 memory deferred; P12 runtime construction deferred. + +## P06/P08/D8 Remediation — focused fail-fast correction pass + +**Scope:** Corrected lifecycle defects in P08 dispose, P06 preparation gaps, D8 +finalisation, and queueWrite chain semantics. Strict Bun TDD (RED → GREEN); +no broadening into P07 provider/tool phase integration. + +**Corrections delivered:** + +1. **P08 disposal (retention.ts):** `PerfRetention.dispose()` now captures the + in-flight tick's internal (non-errno) error, ALWAYS proceeds with claim + cleanup (try/finally pattern), then rethrows the tick error. If claim cleanup + also fails internally, both errors are surfaced via `AggregateError` (project + convention). External errno failures from tick or cleanup remain fail-open / + rate-limited via `emitDiagnostic`. (3 new tests in + `retention.lifecycle.behavior.test.ts` D-LC-4 block.) + +2. **P06 preparation gaps (useSubmitQuery.ts):** `OperationLifecycleRegistry.begin()` + happens before `prepareQueryForAgent()` and `prepareTurnForQuery()`. Their + rejections now finalise the op exactly once as `'error'` via + `finalisePrepRejection` (which preserves the original rejection to the caller; + if finalise also fails internally, an `AggregateError` carries both). (2 new + tests in `useSubmitQuery.lifecycle.test.tsx`.) + +3. **Item 3 classification:** Verified every `!shouldProceed || queryToSend === null` + producer by reading `prepareQueryForAgent` (queryPreparer.ts) source. All + producers are non-send paths: abort signal, empty query, slash/shell command + consumed, @-command error. `cancelled_before_send` is the only semantically + grounded existing status for "started but did not send to the model" (AC-4). + No non-cancellation path reaches this branch with a more-specific status + available. `prepareTurnForQuery` returns void and is called AFTER the check. + No change needed; classification confirmed accurate. + +4. **D8 finalisation (useSubmitQuery.ts):** Replaced fire-and-forget + `finaliseOperation` (which `.catch(debugLogger.error)`-swallowed every + rejection) with a version that returns the promise for awaiting. All three + finalisation paths (completed / error / cancelled_before_send) are now + awaited in `runSubmitQueryCore`. Internal instrumentation errors propagate + fail-fast; external errno errors resolve in PerfSink/retention (fail-open). + The error path handles the original provider error for the user FIRST via + `handleProviderError`, then throws the instrumentation error — the + instrumentation error is NOT routed through user-facing provider-error + handling and does NOT replace the original operation status. (2 new tests in + `useSubmitQuery.lifecycle.test.tsx`.) + +5. **queueWrite / drain (operationLifecycle.ts):** `queueWrite` now chains via + `this.lifecycleChain.then(attempt)` (no recovery handler), so the shared + chain propagates internal rejections. An internal rejection from a write + whose individual promise was not awaited (notably the superseded sweep from + `begin`) is surfaced via `drain()` rather than hidden by a permanently + non-rejecting chain. Serialization is preserved (writes still happen in + order via the chain). External fs errors resolve in the sink. (3 new tests + in `operationLifecycle.behavior.test.ts` P06-D8 block.) + +6. **P07 granular statuses:** `cancelled_during_api/tool/approval` deferred to + P07 as planned. No existing statuses regressed. No `packages/agents` changes. + +**Verification (all GREEN):** +- P06 tests: `operationLifecycle.behavior.test.ts` (35 tests) + + `useSubmitQuery.lifecycle.test.tsx` (10 tests) = 45 tests. +- Affected existing useSubmitQuery tests: `activationFailure.test.tsx` (3) + + `mcpDiscovery.test.tsx` (5) = 8 tests. +- All telemetry perf tests: 293 tests. +- Telemetry + CLI typechecks (`tsc --noEmit`): clean. +- ESLint + Prettier on all 6 touched files: clean. +- `git diff --check`: clean. `packages/agents` / `.llxprt` diffs: empty. +- No settings files changed. + +**Files changed (this remediation):** +- `packages/telemetry/src/perf/retention.ts` — dispose() fail-fast fix. +- `packages/telemetry/src/perf/retention.lifecycle.behavior.test.ts` — 3 new + D-LC-4 tests. +- `packages/cli/src/ui/hooks/agentStream/operationLifecycle.ts` — queueWrite + chain fail-fast fix. +- `packages/cli/src/ui/hooks/agentStream/operationLifecycle.behavior.test.ts` — + 3 new P06-D8 tests + InternalErrorFilesystem. +- `packages/cli/src/ui/hooks/agentStream/useSubmitQuery.ts` — finaliseOperation + returns promise; runSubmitQueryCore restructured; helpers extracted + (finalisePrepRejection, handleProviderError, finaliseStreamError). +- `packages/cli/src/ui/hooks/agentStream/__tests__/useSubmitQuery.lifecycle.test.tsx` + — 4 new tests (prep rejection, turn prep rejection, D8 completed, D8 error). + +**P07-deferred risk:** granular `cancelled_during_api/tool/approval` status +classification is deferred to P07 as planned. The registry supports all seven +statuses (AC-4); the useSubmitQuery wiring currently only emits +`completed`, `error`, `cancelled_before_send`, and `superseded`. P07 will add +the during-api/tool/approval status transitions based on real phase boundaries. + +## P07 progress — client phase measurement + stale-evidence/default-off corrections (COMPLETE) + +**Scope delivered (this final correction pass):** + +1. **Default-off event dispatch (`useAgentEventStream.ts`):** the event loop + previously called `performance.now()` before/after EVERY AgentEvent even when + `onAgentEventObserved` was undefined (perf disabled). Refactored so the + absent-observer branch performs the existing handler dispatch/catch behavior + with NO timing calls and NO sample allocation; the present-observer branch + measures synchronous dispatch and invokes the observer OUTSIDE the generic + catch (D8: a perf-callback throw rejects the stream / fail-fast). A + package-private monotonic-clock seam (`__setMonotonicClockForTesting`, + deep-imported — NOT in the agentStream barrel) lets tests prove zero timing + work on the default-off path. + +2. **Stale terminal cancellation evidence (`operationLifecycle.ts`):** after a + tool-status `cancelled` terminal, a new provider attempt start for the same + operation proves the operation continued. `onProviderAttemptStart` now clears + retained tool/approval cancellation evidence so a later independent API abort + classifies `cancelled_during_api`. Provider-aborted (api) terminal evidence + is PRESERVED (overlap precedence + provider-aborted honesty); current active + tool/approval state is NOT cleared. `retainCancellationEvidence` precedence + (approval > tool > api) is untouched. + +3. **`P07` no longer deferred:** granular `cancelled_during_api/tool/approval` + transitions on real phase boundaries are now wired through the registry's + live phase tracking (AC-4). + +**Bun/bun:test behavioral evidence:** +- `operationLifecycle.p07.contract.behavior.test.ts` (26 tests): provider + correlation (5), live cancellation phases (9), tool interval honesty (3), + observer ownership (3), stale terminal cancellation evidence (5 NEW + + corrected — tool/approval cleared on provider start, active state preserved, + api evidence preserved, overlap precedence preserved). +- `operationLifecycle.p07.behavior.test.ts` (32 tests): direct client phases, + provider/tool interval metrics, honest residual, granular cancellation + classification, D1 continuation, observer fail-fast, superseded queue, + default-off, approval_wait_ms. +- `useAgentEventStream.defaultoff.p07.bun.tsx` (4 NEW tests): absent observer + performs NO monotonic-clock calls; absent observer continues after an ordinary + handler error; present observer measures dispatch; present observer throw + rejects the stream (fail-fast). + +**Verification (all GREEN):** 97 tests across the 4 lifecycle/default-off files; +`useAgentEventStream.bun.tsx` + `loopIntegration` + `useSubmitQuery.lifecycle` +unaffected. Telemetry perf + provider perf tests green (see below). Affected +package typechecks (CLI, telemetry, providers) clean. ESLint + Prettier on +touched files clean. `git diff --check` clean. + +**Out of scope (NOT claimed complete):** P12 integration wiring (runtime +construction of the registry + identity provider + overhead harness) remains +TODO. P10 memory trend and P11 reader/consumer remain TODO. + +## P10 progress — memory trend (zero new timers, ring, two slopes) (COMPLETE) + +**Scope delivered:** +- `packages/cli/src/ui/hooks/memoryTrend/memoryRing.ts`: `MemoryRing` — + fixed-capacity overwrite ring (capacity 180 entries = 3 hours at 60 s + cadence, bounded for a CLI process). Oldest→newest snapshot, defensive + copy (no mutable internal alias). No dynamic/unbounded array growth. +- `packages/cli/src/ui/hooks/useMemoryMonitor.ts` (extended): DEFECT 1 fixed — + the warn-once latch is separated from the sampling loop; the interval no + longer `clearInterval`s itself after warning. Zero new timers (only the + existing 60 s interval is extended). When a `memoryController` is present, + each tick calls `process.memoryUsage()` once and hands the same full sample + to the controller. When absent, warn-only behavior is retained and no + telemetry ring/write work occurs. Package-private + `MemoryMonitorPorts`/`__setMemoryMonitorPortsForTesting` seam for + deterministic timer/memory behavior. Cleanup and default-off preserved. +- `packages/cli/src/ui/hooks/memoryTrend/memoryTelemetry.ts`: + `MemoryTelemetryController` — constructible (no singleton), shares the + existing PerfSink. Maintains the bounded ring, writes schema-valid + `memory_sample` records with wall ISO ts, monotonic `uptime_ms`, and + `ms_since_last_operation`. Pre-first-operation `ms_since_last_operation` + = uptime since process start (honest, not fabricated). `markOperationEnd` + + `sampleOperationEndMemory` implement the `OperationMemorySampler` interface + for the lifecycle registry. `snapshot()` exposes ring contents for P11. +- `packages/cli/src/ui/hooks/agentStream/operationLifecycle.ts` (modified): + `OperationLifecycleRegistry` gets an optional `memorySampler` (present only + when memory telemetry enabled). At exactly-once finalisation, + `captureOperationEndMemory()` marks operation-end and samples + `process.memoryUsage()` once to include `rss_bytes`/`heap_used_bytes`/ + `external_bytes`/`array_buffers_bytes`. Disabled/absent omits all four + fields (never zeros). P07 elapsed/finalize semantics and existing default + behavior preserved. +- `packages/cli/src/ui/hooks/memoryTrend/memorySlope.ts`: read-time slopes + (never persisted). `derivePerOperationMemorySlope` — least-squares of each + memory column on `session_operation_index`. `derivePerMinuteMemorySlope` — + least-squares on `uptime_ms` scaled to bytes/min. Requires ≥2 usable points + and nonzero x variance; otherwise null (never NaN/Infinity). Negative + slopes preserved (not clamped). Per-record-series functions (P11 invokes + per run/file; separate files NOT combined in P10). +- `packages/cli/src/ui/hooks/memoryTrend/index.ts`: barrel exporting all P10 + types/functions for downstream P11/P12. + +**Bun/bun:test behavioral evidence** (31 tests across 5 NEW files — no Vitest/Node +suites modified): +- `memoryRing.behavior.test.ts` (8 tests): empty ring, oldest→newest order, + overwrite after capacity, multi-wrap, defensive copy (mutating snapshot does + not affect ring), independent snapshot arrays, default capacity exposed, + default-capacity overflow holds exactly capacity. +- `memoryTelemetry.behavior.test.ts` (8 tests): schema-valid memory_sample + with correct values/uptime, pre-first-operation idle = uptime (honest), + post-operation idle = uptime − last op end, ring snapshot oldest→newest, + sampleOperationEndMemory returns four columns, implements + OperationMemorySampler interface, idempotent columns, no slope key in + persisted records. +- `useMemoryMonitor.behavior.test.ts` (6 tests): exactly one interval on + mount, clears on unmount, warning fires once but interval continues (DEFECT 1 + fix), no warning below threshold, memory-off retains warn-only, memory-on + records tick samples to ring. +- `memorySlope.behavior.test.ts` (13 tests): per-operation positive slope + across all four metrics, negative slope preserved, <2 points null, empty + null, zero x-variance null, ignores records without memory columns, exactly + 2 points yields slope; per-minute positive slope, negative slope preserved, + <2 points null, empty null, zero x-variance null, exactly 2 points yields + slope. +- `operationLifecycle.p10.memory.behavior.test.ts` (4 tests): memory ON + includes all four columns, memory OFF omits all four (absent not zero), + markOperationEnd called at finalisation, no slope key in persisted record. + +**Verification (all GREEN):** 132 tests across 8 CLI files (P06/P07/P10 + +memoryTrend); 297 telemetry perf tests; 64 P09 settings tests. CLI + telemetry +typechecks clean. ESLint + Prettier on all touched files clean. +`git diff --check` clean. `packages/agents` / `.llxprt` / `specification.md` / +`PLAN.md` diffs: empty. No dependencies/workflows/lint config modified. + +Artifacts outside `project-plans/issue3167/`: `packages/cli/src/ui/hooks/ +memoryTrend/` (new module — 7 files), `packages/cli/src/ui/hooks/useMemoryMonitor.ts` +(modified), `packages/cli/src/ui/hooks/agentStream/operationLifecycle.ts` +(modified), `operationLifecycle.p10.memory.behavior.test.ts` (new). +`.llxprt` untouched. Authoritative design records NOT rewritten. +P11 reader/consumer remains TODO. + +## P11 progress — reader/consumer + report + /perf + inspect + delete (COMPLETE) + +**Scope delivered:** + +1. **Sorted per-file tolerant consumer** (`perfConsumer.ts`): + `streamPerfDirectory` / `consumePerfDirectory` — async generators that read + sorted `perf-*.jsonl` files one at a time (no gzip, no shell pipeline, no + argument-limit breakage). Each entry carries source-file name + run-UUID + identity so memory slopes are per-run/file and never pooled across process + uptimes. Missing directory = empty dataset (fail open). Genuine filesystem + errors propagate. Does NOT parse claim files. Bounded `consumePerfDirectory` + accumulates entries + aggregate counts (parsed/malformed/futureVersion/ + unversioned/truncated/blank/files/bytes). + +2. **D7 dimension-matched report + no-baseline semantics** (`perfReport.ts`): + `buildReport` / `assembleReport` — groups operations by build identity + (`llxprt_version` + `git_sha`) within exact comparison dimensions (provider, + model, render_mode, terminal_cols, terminal_rows). Computes sample count, + contaminated count (`concurrent_instances >= 2`, NOT contended), p50 for all + recorded timing/counter/token metrics, terminal status counts, and per-file + memory slopes from P10 functions. Without `--baseline`: grouped p50 / sample / + self-health, NO delta. With `--baseline `: matched-dimension + deltas vs baseline rows only; unmatched groups reported as unmatched, NEVER + pooled. Percent delta avoids division by zero (null when baseline p50 is 0). + Self-health surfaces skipped/truncated/lastWriteErrorCode/evictionCount (NOT + records_dropped — excluded). + +3. **D1 read-time join** (`joinTokenRowsByOperation`): token-usage/session rows + carrying a `prompt_id` are joined by deriving the operation_id via + `joinKeyFromPromptId` (splits on `#continuation#`). N continuation rows join + to one operation without persisted child ids. + +4. **Inspect** (`perfInspect.ts`): surfaces directory path, schema version, + privacy statement (local-only/default-off/no-upload/memory-separately-opt-in), + owned JSONL file count/bytes, operation/memory-sample record counts, tolerant + skipped breakdown, and claim count. `formatInspect` produces stable output. + +5. **Live-writer-safe delete** (`perfDelete.ts`): removes owned perf JSONL and + stale claim artifacts. Protects the current UTC-day perf file with recent + mtime (active writer) and any perf JSONL whose run UUID has a non-stale claim + (lease). Reuses shared `isLiveWriterFile` / `isNonStaleClaim` / `extractRunUuid` + from `perfArtifacts.ts`. Never deletes unrelated files. External fs failures + fail open and are counted. Internal invalid options (NaN/negative timing) fail + fast. Injected `PerfDeleteFilesystem` port for deterministic fault injection. + Delete's claim→JSONL protection is deliberately broader than automatic + retention (see item 6). + +6. **Shared artifact protection helpers** (`perfArtifacts.ts`): single source of + truth for `isPerfJsonl`, `isClaimFile`, `isOwnedArtifact`, `parseDayKeyFromName`, + `extractRunUuid`, `utcDayKey`, `isLiveWriterFile`, `isNonStaleClaim`. Retention + and delete share these primitives but apply deliberately different JSONL + policies: automatic retention protects a JSONL only as a live writer (today's + UTC day-key + mtime within the maintenance window) so a 24×7 process converges + to the eventual caps by evicting its own old-day files; explicit delete + additionally protects any JSONL whose run holds a fresh claim + (`isPerfJsonlProtected`) to avoid unlinking a file another active process may + still be appending. Fresh claims themselves remain protected in retention + (non-stale claims survive and count toward caps) but never shield that run's + older JSONL. + +7. **Self-health** (P04/P08 source): `PerfSink.lastWriteErrorCode` (string | null) + and `PerfRetention.evictionCount` (number). No `records_dropped` counter. + 8 Bun tests prove these surface correctly and that records_dropped is absent. + +8. **Injected live snapshot command** (`perfCommand.ts`): `createPerfCommand` + accepts an optional `PerfSnapshotCapability` and `perfDir` override. Bare + `/perf` shows current-process snapshot (live MemoryRing + active operation) + when capability is present; says "not active" honestly when absent. No + unowned global singleton — `perfCommand` export has null capability; P12 + wires the production capability via `createPerfCommand({ snapshotCapability })`. + Subcommands: `/perf inspect`, `/perf report [--baseline ]`, + `/perf delete`. Registered in `BuiltinCommandLoader`. `MessageActionReturn` + result type throughout (no wrong slash command result types). + +9. **Stable formatter/arg handling**: `formatReport`, `formatInspect`, + `formatDeleteResult` produce deterministic output. `parseReportArgs` handles + `--baseline ` (exact version or sha), rejects missing values and + unexpected args with useful error messages. + +10. **P10 single wall-clock sample correction**: `MemoryTelemetryController. + recordTickSample` captures `wallNow()` exactly once so the ring timestamp + and the record `ts` describe the same sample. `useMemoryMonitor` calls + `process.memoryUsage()` once per tick and passes the same full sample to + the controller. + +11. **P09 planning wording correction**: execution-tracker P09 entry corrected + from "deep-copies + freezes perf (`Object.freeze`)" to "defensively clones + perf (`withClonedPerf`) on every ingress and egress" — matching the actual + implementation (isolation by cloning, not freezing). + +12. **Exports**: all P11 public API re-exported from + `packages/telemetry/src/perf/index.ts` → `packages/telemetry/index.ts`. + Consumer, report, inspect, delete, artifact helpers, types. + +**Bun/bun:test behavioral evidence** (72 tests across 6 NEW files — no Vitest/Node +suites modified): +- `perfConsumer.behavior.test.ts` (12 tests): sorted parsing, source/run-UUID + identity, streaming order, missing-dir empty dataset, future-version skip+count, + unversioned skip+count, malformed skip+count, truncated count, blank count, + no claim parsing, multi-file aggregation, lazy streaming. +- `perfReport.behavior.test.ts` (18 tests): dimension grouping, never-pool, + p50, contamination, terminal status, no-baseline-no-delta, baseline-by-version, + baseline-by-sha, unmatched-explicit, baseline-not-found, empty-dir, percent + div-zero, per-file memory slopes, formatter stability, formatter self-health, + D1 join (2), mixed multi-version fileset. +- `perfInspect.behavior.test.ts` (7 tests): dir/schema/privacy, file/byte counts, + operation/memory counts, skipped breakdown, claim count, missing-dir zero, + formatter fields. +- `perfDelete.behavior.test.ts` (11 tests): stale delete, active-writer protect, + claim protect, non-stale/future claim protect, stale-claim delete, unrelated + untouched, missing-dir no-op, fs-failure fail-open, NaN fail-fast, negative + interval fail-fast, same-run stale+fresh. +- `perfSelfHealth.behavior.test.ts` (8 tests): lastWriteErrorCode null/error/ + persistent, no records_dropped; evictionCount zero/increment/no-increment-on- + failure, no records_dropped. +- `perfCommand.behavior.test.ts` (16 tests): inspect output + empty-dir, report + output + no-baseline + baseline version/sha + missing value + malformed + + unexpected arg, delete stale + active-writer, no-snapshot unavailable, + snapshot with capability, null-snapshot, unknown-subcommand error, loader + registration. + +**Verification (all GREEN):** +- P11 telemetry tests: 56 tests (consumer + report + inspect + delete + self-health). +- P11 CLI tests: 16 tests (perfCommand). +- All telemetry perf tests: 353 tests. +- Affected P06/P07/P10 tests: 97 + 36 = 133 tests. +- Affected P05/P09 tests: 58 + 22 = 80 tests. +- Provider perf tests: 11 tests. +- CLI + telemetry typechecks (`tsc --noEmit`): clean. +- ESLint + Prettier on all touched files: clean. +- `git diff --check`: clean. No `.llxprt` / `packages/agents` / `specification.md` + / `PLAN.md` / dependency / workflow / lint-config changes. + +Artifacts outside `project-plans/issue3167/`: `packages/telemetry/src/perf/ +perfConsumer.ts`, `perfReport.ts`, `perfInspect.ts`, `perfDelete.ts`, +`perfArtifacts.ts`, `index.ts` (new/modified); `packages/cli/src/ui/commands/ +perfCommand.ts` (new); `packages/cli/src/services/BuiltinCommandLoader.ts` +(modified — perfCommand registration); 6 new test files. `.llxprt` untouched. +Authoritative design records NOT rewritten. + +--- + +## Remediation Evidence (Findings A/B/C) + +**Finding A — Synchronous immutable terminal snapshot:** +- `operationLifecycle.ts`: `finalise` now atomically claims + synchronously + freezes one immutable `FrozenTerminalSnapshot` (wall ts, terminal monotonic, + elapsed/uptime, identity, status, ALL measurement counters/tokens, interval + durations, approval-wait closure, client_finalize_ms measured against the + synchronous finalize boundary, honest residual, optional operation-end memory) + BEFORE queueing. The queued async work (`persistSnapshot`) does ONLY external + `countNonStaleClaims` + `sink.write` from the frozen copy — never a mutable + PendingOp/measurement reference. Superseded sweep uses the same path. +- `retainedCancellationEvidence` changed from strong `Map` to `WeakMap` — + evidence survives active-map removal until classification (behaviorally + proven). +- Missing tool boundaries: count/sum recorded WITHOUT interval synthesis; + CLI status transitions do NOT synthesize tool intervals. +- Tests: `operationLifecycle.snapshot.behavior.test.ts` — 6 behavioral tests + (mutation-after-finalise cannot change record; delayed countNonStaleClaims + cannot alter terminal snapshot; coherent client_finalize/elapsed/residual + clocks; superseded snapshots frozen; cancellation evidence after active + removal without strong retention; missing tool boundaries no synthesized + intervals). All pass. + +**Finding B — Fully transactional interactive startup:** +- `interactiveUI.tsx`: every fallible stage after perf owner starts runs as ONE + transaction via `commitInteractiveStartup` with injectable + `InteractiveStartupPorts` (renderOptions, buildUiRuntime, buildSlashRuntime, + debugAppend, setupTerminal, isMouseEnabled, render, registerSync, + setupLifecycle). Single try/catch with `StartupTransactionState` — no nested + rollback. On failure: primary error preserved first, tracked refs atomically + cleared (exactly-once), owner disposed, instance cleared/unmounted, staged + mouse disabled + listener removed, terminal protocols restored + listener + removed. `mouseStaged` computed BEFORE `setupTerminal` so pre-mouse failures + do NOT falsely disable unstaged mouse. Cleanup errors aggregate via + `rollbackInteractiveFailure`. +- Tests: `interactiveUI.startup.transaction.behavior.test.ts` — 9 behavioral + tests (render-options/ui-runtime/slash-runtime/debug/terminal-setup/render/ + setup failures all roll back; primary-error ordering; exactly-once cleanup). + All pass. + +**Finding C — Production report wiring:** +- `interactivePerfRuntime.ts`: `createSnapshotCapability` now implements + `getSelfHealth()` returning `{ lastWriteErrorCode: sink.lastWriteErrorCode, + evictionCount: retention.evictionCount }` — known null/0, not unavailable. +- `perfCommand.ts`: `PerfSnapshotCapability` extended with `getSelfHealth()`; + `PerfOperations.report` extended to accept `selfHealth` + `tokenUsageDir`; + `PerfCommandOptions` extended with `tokenUsageDir`; `createReportSubCommand` + passes self-health from capability (undefined when no active runtime → + "unavailable") + token-usage directory to telemetry `buildReport`. +- `BuiltinCommandLoader.ts`: wires `tokenUsageDir: join(config.getProjectTempDir(), + 'token-usage')` (guarded with optional chaining so null config stays undefined). +- `perfReport.ts` (telemetry): production `buildReport` now streams token files + and aggregates by derived operation ID (O(operation IDs) memory) instead of + retaining all token rows. `assembleReport` accepts `aggregatedTokens` map. + Stale self-health doc corrected (not defaulted to null/0 when unavailable). +- Tests: `perfCommand.wiring.behavior.test.ts` — 7 behavioral tests (exact + params; inactive health = unavailable; active clean = null/0; active + errors/evictions propagate; real continuation token join in production + command; no tokenUsageDir keeps persisted totals; no capability passes + undefined self-health). All pass. + +**Verification (all GREEN):** +- Telemetry perf tests: 435 tests. +- operationLifecycle tests: 107 tests (39 + 62 + 6 new). +- Finding B startup tests: 9 tests. +- perfCommand tests: 32 tests (25 existing + 7 new). +- interactivePerfRuntime + buildPerfOwner tests: 29 tests. +- Related CLI behavior tests (inkRenderOptions, perfSettings): 33 tests. +- Telemetry + CLI typechecks (`tsc --noEmit`): clean. +- ESLint on all touched files: clean (no eslint-disable/suppression). +- Prettier on all touched files: clean. +- `git diff --check`: clean. diff --git a/project-plans/issue3167/plan/00-overview.md b/project-plans/issue3167/plan/00-overview.md new file mode 100644 index 0000000000..bb359b352d --- /dev/null +++ b/project-plans/issue3167/plan/00-overview.md @@ -0,0 +1,88 @@ +# Plan Overview — Client-Side Performance Telemetry + +Plan ID: PLAN-20260808-PERFTREND +Issue: #3167 +Milestone: 0.11.0 +Phases: 01 → 13 (sequential, no skips) + +> Binding design record: `specification.md` (§1–§9). `PLAN.md` requirements are +> reconciled here against spec §9; where broader, the spec's reduced delivery +> wins. Acceptance: `../acceptance-criteria.md`. Analysis: +> `../analysis/domain-model.md`, `../analysis/pseudocode/0[1-9]-*.md`. +> Execution status: `../execution-tracker.md`. + +## Phase sequence (mandatory execution order) + +| Phase | Title | Pseudocode | ACs | Package(s) | +|---|---|---|---|---| +| 01 | Preflight verification (findings) | — | — | all (read-only) | +| 02 | Analysis & pseudocode (finalise) | 01–09 | — | — | +| 03 | IntervalUnion extraction (incremental duration) | 02 lines 10-35 | AC-5 | telemetry | +| 04 | Schema + PerfSink + tolerant reader + record-size benchmark | 01 lines 10-115; 02 lines 50-108 | AC-1, AC-2, AC-3(join), AC-8 | telemetry | +| 05 | Stdout observer seam + Ink onRender wiring | 03 lines 10-58; 05 lines 20-26 | AC-6 | core, cli | +| 06 | Operation lifecycle registry + identity | 04 lines 10-66; 01 lines 60-66 + 104-115 | AC-3, AC-4 | cli | +| 07 | Client phase measurement + record assembly | 05 lines 10-74 | AC-5 | cli | +| 08 | Retention (eventual bound, live-writer safe, claim files) | 06 lines 10-72 | AC-7 | telemetry | +| 09 | Settings (opt-in, default-off, nested — D2) | 08 lines 10-24 | AC-2 | core config | +| 10 | Memory trend (zero new timers, ring, slopes) | 07 lines 10-93 | AC-10, AC-11 | cli | +| 11 | Reader/consumer + report + /perf + inspect + delete | 08 lines 30-99 | AC-9 | telemetry, cli | +| 12 | Integration wiring + overhead harness | 09 lines 10-64 | AC-12 | cli | +| 13 | Final verification (whole-suite + manual smoke) | — | all | all | + +> **Resolved decisions (D1–D8)** are applied across AC/domain/pseudocode/phase +> artifacts; see `acceptance-criteria.md`. P03 remains COMPLETE. P04 first adds a +> Bun record-size benchmark (D5) whose output P08 uses to derive retention +> constants. + +## TDD cycle within each implementation phase (03–12) + +Each implementation phase follows integration-first TDD: +1. **Stub** — minimal skeleton that compiles (throws `NotYetImplemented` or typed + empty); no reverse tests; update existing files, never parallel versions. +2. **Integration TDD** — write the integration behavioral test FIRST (real files, + real pipeline) that the stub fails naturally. +3. **Impl** — implement to pseudocode line ranges; make the integration test pass. +4. **Verify** — run phase tests, typecheck, lint, deferred-impl detection; + no `eslint-disable`/`@ts-*`/severity downgrades (fix design instead). + +## Hard constraints (carry into every phase) + +- No dependency/workflow/lint/complexity/source-size/quality-tool changes beyond + what the settled spec requires; no settings unrelated to perf. +- No `eslint-disable`, TS suppression directives, severity downgrades, complexity + threshold increases, or ignore exclusions — fix the underlying design. +- All new tests are Bun / `bun:test`, behavioral, integration-first, proving real + outputs/state/files (no mock theater). No new JS/Vitest/Node tests. +- `packages/agents` is untouched (no telemetry edge; operation_id derived; no + child-id arrays on the record — D1). +- Fail-fast in-process (incl. the stdout observer — D8); defensive parsing ONLY + for external JSONL/filesystem. +- Retention = eventual bound + live-writer safety + claim-file accounting (D3), + not instantaneous no-loss cap; explicitly permits active-day/claim overshoot (D5). +- Persistent perf telemetry opt-in/default-off (`telemetry.perf.enabled` — D2, + not a boolean), inspectable, deletable. +- PerfSink does NOT inherit FileOutput's bounded/drop queue (D4); serialized + no-drop promise chain; no gzip; no size sub-rolling. +- fs-failure tests use a package-private port / failing file handle, never + real-disk fill or chmod (D6). +- Report `--baseline` is an exact version/sha; unmatched groups never pooled (D7). +- Memory sampling adds no timer (uses existing 60 s monitor); memory-disabled ⇒ + fields omitted, not zeroed. + +## Sequential execution order (practical) + +``` +01 (preflight) → 02 (analysis) → 03 (IntervalUnion) + → 04 (schema+sink+reader) [foundation; depends on 03] + → 05 (stdout observer/onRender) [core seam; parallel-safe with 04] + → 09 (settings) [gates 06/10/11; do before wiring] + → 06 (lifecycle/identity) [depends on 04, 09] + → 07 (client phases) [depends on 05, 06] + → 08 (retention) [depends on 04] + → 10 (memory trend) [depends on 06, 09] + → 11 (reader/consumer/perf) [depends on 04, 09] + → 12 (integration + overhead) [depends on 06,07,10,11] + → 13 (final verification) +``` +03, 05, 09 may be developed in parallel once 04's schema lands; the registry +(06) is the integration spine everything hangs off. diff --git a/project-plans/issue3167/plan/01-preflight-verification.md b/project-plans/issue3167/plan/01-preflight-verification.md new file mode 100644 index 0000000000..6ed3d71aa1 --- /dev/null +++ b/project-plans/issue3167/plan/01-preflight-verification.md @@ -0,0 +1,255 @@ +# Phase 01: Preflight Verification (blocking) + +Plan ID: PLAN-20260808-PERFTREND.P01 +Status: **COMPLETE** (findings below). No implementation until every checkbox is +green; all are. + +> **Preflight correction (this pass).** Source fact-checking against the live tree +> found blockers in the original artifacts and **falsified** the claim that child +> continuation ids arrive in the CLI via `AgentEvent` (§3). These blockers are +> resolved here and in the companion artifacts by decisions **D1–D8** (see +> `acceptance-criteria.md`): D1 child-id arrays removed + join at read time; D2 +> nested settings; D3 claim-file concurrency accounting; D4 PerfSink does not +> inherit FileOutput's bounded/drop queue; D5 retention constants derived from a +> Bun record-size benchmark; D6 fs-failure testing via a package-private port; +> D7 report `--baseline`; D8 stdout observer fails fast. P03 (IntervalUnion) +> remains COMPLETE and is unaffected. + +This phase verifies every Phase-0.5 assumption named in PLAN.md Phase 0.5 and the +task directive, against the actual source tree. Evidence is file:line. + +--- + +## 1. Ink observer injection seam — RESOLVED (the real blocker) + +**Finding:** `packages/cli/src/ui/inkRenderOptions.ts:24` builds +`const sharedStdio = createInkStdio();` at **module scope** (import time), before +any config/settings object exists → no seam to inject an observer into today. +`createInkStdio()` in `packages/core/src/utils/stdio.ts:118` returns Proxies +delegating `write` → `writeToStdout` with **no observer parameter**. + +Zed builds its OWN instance separately +(`packages/cli/src/zed-integration/runZedIntegration.ts:105-112`), so a global +patch would double-count Zed. + +**Ink onRender:** CLOSED (verified earlier). `node_modules/ink/build/render.d.ts:44` +declares `onRender?: (metrics: RenderMetrics) => void`; `ink.js:74-76` throttles +per actual render pass. `ink_render_ms` is a pure accumulate (Ink computes the +duration). + +**Decision (locks the design):** +- `createInkStdio(observer?)` gains an **optional** `StdoutWriteObserver` param + (pseudocode 03 lines 10-36). Absence ⇒ identical current behaviour. +- The Proxy `write` trap measures encoded bytes + sync duration, delegates to + `writeToStdout`, preserves overload/encoding/callback/backpressure, and calls + the observer **directly with no try/catch** (D8: internal observer/programming + errors fail fast; only filesystem writer failures fail open as external I/O). +- `inkRenderOptions.ts` replaces the module-scope constant with a **lazy cache** + (`getInteractiveStdio()`, pseudocode 03 lines 41-55) so a late, + settings-gated `setInteractiveStdoutObserver()` invalidates the cache and the + next render carries the observer. +- Zed keeps calling `createInkStdio()` with no observer ⇒ uncounted. +- A single observer accumulates globally; the operation recorder snapshots the + delta per operation (no per-operation Proxy churn needed). + +- [x] Seam absent → RESOLVED by optional-observer + lazy cache (interactive only). + +## 2. Operation lifecycle seams — VERIFIED + +`packages/cli/src/ui/hooks/agentStream/useSubmitQuery.ts`: +- Acquire `~:627-631`: `activeTurnRef.current = true`; installs the turn's + `AbortController` at `:630`. +- Release `~:650-659`: inside `finally`, guarded by `isCurrentTurn(current, turnSignal)` + (`:813-814` compares `abortControllerRef.current?.signal === signal`). +- `isCurrentTurn` is false for a superseded turn → its `finally` release never + runs → **finalisation needs its own registry + sweep** (confirmed load-bearing). +- Cancellation: `useAgentStreamLifecycle.ts` `useCancellation` (`:219-288`) sets + `turnCancelled`, aborts the controller, cancels tool calls. +- No-send path: `prepareTurnForQuery` (`queryPreparer.ts:82`) gates on the turn's + own abort signal; pre-send failure flows to `handleSubmissionError`. +- Queued submissions: `useDrainSubmission` + `useScheduleNext` (`:430-530`). + +- [x] Acquire/release stable; superseded unreachable via release (needs sweep). +- [x] All terminal-status paths reachable (completed/error/4×cancelled/superseded). +- [x] CLI holds initial prompt id before `runStream` (`:787`). + +## 3. Prompt/turn identity + prefix invariant — VERIFIED; child-id claim CORRECTED + +`packages/agents/src/core/agenticLoop/AgenticLoop.ts`: +- `:238-244`: `generateInitialPromptId() = ${sessionId}#agentic-loop#${uuid}`; + `generateContinuationPromptId(initial) = ${initial}#continuation#${n}`. +- `run()` (`:378-410`): `initialPromptId = promptId ?? generateInitialPromptId()`; + continuations recompute `currentPromptId = generateContinuationPromptId(initialPromptId)`. +- CLI fallback (no caller promptId): `resolvedPromptId = sessionId + '########' + count` + (EIGHT hashes — never contains `#continuation#`). +- Existing test: `packages/agents/src/core/agenticLoop/__tests__/agenticLoop.prompt-id.test.ts`. + +⇒ `operation_id = promptId.split('#continuation#')[0]` is sound in **all** paths. + +**BLOCKER FOUND (D1).** An earlier revision of this preflight claimed: "Continuation +ids reach the CLI via per-turn stream events (`useStreamEventHandlers`, `streamUtils` +carry `prompt_id`)." Source fact-checking **falsified** this: `AgentEvent` +(`packages/agents/src/api/event-types.ts`) defines ~20 variants (text, thinking, +tool-call, tool-result, tool-confirmation, tool-status, usage, model-info, notice, +compression, context-warning, retry, citation, loop-detected, idle-timeout, +invalid-stream, hook-blocked, error, done) — **none carries a per-continuation +`prompt_id`**. The `prompt_id` the CLI supplies to `agent.stream()` / +`useStreamEventHandlers` is the **outer** id only; the child continuation ids stay +inside `packages/agents`. So the CLI **cannot** observe child prompt/turn ids, and +the original plan to collect + cap them on the perf record is unimplementable +without an excluded agents→telemetry edge. + +**Resolved by D1:** the perf record carries **no** `prompt_ids`/`turn_ids` +arrays/true-count; `operation_id` is the sole join key, and the report derives it +from token-usage/session `prompt_id` metadata at read time (token-usage records DO +carry per-send `prompt_id` — `tokenUsageRecords.ts:63`). Zero plumbing; +`packages/agents` untouched. + +- [x] Prefix invariant holds; derivation safe; existing test to extend. +- [x] FALSE child-id claim removed; D1 join-at-read-time recorded. + +## 4. Settings hierarchy / default / privacy — VERIFIED; shape CORRECTED (D2) + +- `TelemetrySettings` interface: `packages/core/src/config/configTypes.ts:117` + (`enabled`, `logConversations` default false). +- Resolved in `packages/core/src/config/configConstructor.ts` `resolveTelemetrySettings` + (hierarchy: CLI flags > env > workspace `.llxprt/settings.json` > user > defaults). +- No zod schema for telemetry settings (interface + manual resolution) — new keys + follow the same interface pattern; `perf?: { enabled?: boolean; memory?: boolean }`. +- `docs/telemetry-privacy.md`: persistent telemetry opt-in, disabled by default, + records carry session/project/provider/model identity ⇒ mandatory. + +**D2 (corrected):** the persisted shape is **nested** `telemetry.perf.enabled` +(master) and `telemetry.perf.memory`, both default **false**; `memory` requires +`enabled`. `telemetry.perf` is **not** itself a boolean (spec §7.4 names the +master `telemetry.perf`; the resolved persisted contract is the nested shape — +recorded here because spec is not rewritten). + +- [x] Hierarchy understood; new keys follow it; default-off honoured. +- [x] Nested settings shape confirmed (D2); `telemetry.perf` is not a boolean. + +## 5. FileOutput reuse + retention constraints — VERIFIED + +`packages/telemetry/src/debug/FileOutput.ts`: +- Singleton (`private static instance`), `getInstance`/`disposeInstance`. +- `maxFileSize = 10*1024*1024`, `maxQueueSize = 1000`, `batchSize = 50`, + `flushInterval = 1000`, serialized-write guard `isWriting`, `dispose()` drain. +- `fs.stat` per flush (in `checkFileRotation`); unbounded `console.error` on + failure; `grep -c 'unlink|rm'` == **0** (never deletes anything). +- Existing test `FileOutput.test.ts` is **mock-theater** (`vi.mock('fs')`) — the + new perf tests must NOT repeat this; they use real files (or a package-private + fs port for fault injection — D6). + +⇒ **D4 (corrected):** PerfSink does **not** inherit/extend `FileOutput` and does +not carry over its bounded/drop queue, batch+interval flush, or singleton. Narrow +file/path/append primitives are extracted/reused where practical while +`FileOutput`'s public singleton/debug behaviour is preserved. PerfSink uses a +serialized no-drop promise chain, one exclusive-create day file per run UUID, UTC +roll on next record, no gzip, no size sub-rolling. (An earlier "extend it + fix 4 +defects" framing is withdrawn — that would inherit the bounded/drop queue this +design must not have.) + +- [x] Reusable narrow primitives identified; FileOutput's public behaviour preserved. +- [x] PerfSink does NOT inherit FileOutput's bounded/drop queue (D4). + +## 6. IntervalUnion extraction — VERIFIED + +`packages/telemetry/src/telemetry/sessionMetricsAggregator.ts`: +- `IntervalUnion` is a **private** class; `add()` → `recomputeDuration()` walks + every interval on each insert → O(n²) over a 24/7 session. +- Exports: `ApiAttemptRecord`, `ModelBreakdown`, `SessionMetricsSnapshot`, + `SessionMetricsAggregator` (the class itself, not IntervalUnion). + +⇒ Extract to exported `intervalUnion.ts`; maintain `cachedDurationMs` +incrementally; refactor `SessionMetricsAggregator` to import it. + +- [x] Private + quadratic confirmed; extraction plan defined. + +## 7. Memory-monitor timer/ring seam — VERIFIED + +`packages/cli/src/ui/hooks/useMemoryMonitor.ts`: +- `MEMORY_CHECK_INTERVAL_MS = 60 * 1000`; unconditional interval; calls + `process.memoryUsage().rss`; **`clearInterval(intervalId)` inside the warning + branch** → self-terminates after warning once (the defect). +- `Footer.tsx` `ResponsiveMemoryDisplay` runs a 2 s interval **gated on + `showMemoryUsage` and on being mounted** → NOT a viable host. + +⇒ Extend the 60 s interval; separate warn-once latch from sampling loop; add a +fixed-capacity overwrite ring for the live `/perf` view. Zero new timers. + +- [x] Host interval exists; self-termination is the defect to fix; ring to add. + +## 8. Command routing for /perf + inspect/delete/report — VERIFIED + +- Commands aggregated in `packages/cli/src/services/BuiltinCommandLoader.ts:141-200` + (array incl. `statsCommand`, `loggingCommand`, …). +- `SlashCommand` type + subcommand pattern (`packages/cli/src/ui/commands/types.ts`, + `statsCommand.ts`, `loggingCommand.ts`). +- `/perf` with subcommands (default/inspect/report/delete) follows the pattern. + +- [x] Registration point + convention confirmed. + +## 9. Version/git/project/runtime identity — VERIFIED + +- `getCliVersion()` async+cached: `packages/cli/src/utils/version.ts`. +- `getGitCommitInfo()` sync+cached: `packages/cli/src/utils/gitCommitInfo.ts`. +- runtime: `process.versions`/`Bun.version`; platform: `process.platform`+`process.arch`. +- runtime_id: agent runtime state `runtime.getRuntimeId()`; session_id: + `config.getSessionId()`; parent/subagent via subagent orchestrator keys. +- project_hash: SHA-256 of project root (cwd) — computed; path stays global. + +- [x] All identity fields reachable from cli without new edges. + +## 10. Concurrent-instance calculation — DESIGNED (D3) + +`concurrent_instances` replaces the [EXCLUDED] `contended` drift probe at zero +timer cost. **D3 (corrected):** a per-run claim file in the global perf dir is +created on perf enable, touched by the single owned coarse maintenance interval, +and removed on clean dispose. At operation finalization the count of **non-stale** +claims (mtime within the lease window) is `concurrent_instances`. The name is kept +but the value has **lease-window semantics with bounded crash overshoot** (a +crashed run leaves a stale claim until the next sweep). This reuses one owned +maintenance timer; there is **no drift-probe timer and no additional memory +timer**. Claim files are included in retention artifact accounting but are never +parsed as JSONL records. (An earlier "count perf files by mtime, minus self" +framing is withdrawn — it conflates JSONL files with run liveness.) + +- [x] Zero-timer claim-file design defined (D3). + +## 11. End-to-end overhead harness design — VERIFIED FEASIBLE + +Bun/`bun:test` harness exercises the REAL integrated pipeline (real PerfSink in +tmpdir + real stdout observer + real onRender path + real lifecycle registry + +fixture streaming provider). Runs perf-enabled and perf-disabled; reports +p50/p95/p99; asserts stable invariants only (no wall-clock gate). No mocks of the +recorder/sink/observer (would be mock theater). + +- [x] Harness design feasible with Bun; no new deps required. + +## 12. Dependency / type / test-infra checks + +- `zod` present (used by `tokenUsageRecords.ts`) — schema-first mandated by RULES. +- `bun:test` is the test framework (`FileOutput.test.ts` imports from `bun:test`). +- All target packages export `.js` ESM (no CommonJS). + +- [x] zod available; bun:test is the framework; ESM throughout. + +## Blocking issues found + +**Found and RESOLVED (D1–D8).** Source fact-checking falsified the original +"child ids arrive via AgentEvent" claim (§3) and surfaced five further design +corrections (D2 settings, D3 concurrent_instances, D4 PerfSink inheritance, D6 +fs-failure testing, D8 stdout observer). All are resolved in the companion +artifacts (acceptance-criteria, domain-model, pseudocode, phase files) and do not +block implementation: every seam is verified, the Ink observer blocker is resolved +by the optional-observer + lazy-cache decision (§1), and the schema contract is +settled. P03 (IntervalUnion) remains COMPLETE. Proceed to Phase 02 → P04. + +## Verification gate + +- [x] All dependencies verified (no new deps; zod + bun:test present). +- [x] All types/edges match expectations (telemetry < core < agents < cli; no agents→telemetry). +- [x] All call paths possible (interactive-only observer; lifecycle acquire/release). +- [x] Test infrastructure ready (bun:test; existing tests to extend, not duplicate). +- [x] Ink observer injection mechanism DECIDED and documented. +- [x] Blockers D1–D8 resolved and applied across companion artifacts. diff --git a/project-plans/issue3167/plan/02-analysis-pseudocode.md b/project-plans/issue3167/plan/02-analysis-pseudocode.md new file mode 100644 index 0000000000..ff3caca83e --- /dev/null +++ b/project-plans/issue3167/plan/02-analysis-pseudocode.md @@ -0,0 +1,33 @@ +# Phase 02: Analysis & Pseudocode + +Plan ID: PLAN-20260808-PERFTREND.P02 +Prerequisites: P01 (preflight) complete. + +Artifacts already produced (finalised in this phase, not rewritten): +- `analysis/domain-model.md` — entities, layering, invariants, state transitions, failure model. +- `analysis/pseudocode/01-schema-and-reader.md` — schema + derivation + tolerant reader. +- `analysis/pseudocode/02-perfsink-and-interval-union.md` — PerfSink + extracted IntervalUnion. +- `analysis/pseudocode/03-stdout-observer.md` — core seam + cli lazy install. +- `analysis/pseudocode/04-operation-lifecycle.md` — registry + identity collection. +- `analysis/pseudocode/05-client-phases.md` — direct measurement + record assembly. +- `analysis/pseudocode/06-retention.md` — eventual bound + live-writer safety. +- `analysis/pseudocode/07-memory-trend.md` — two slopes, zero timers, ring. +- `analysis/pseudocode/08-consumer-and-perf-command.md` — reader/report/perf/settings. +- `analysis/pseudocode/09-overhead-harness.md` — real integration overhead harness. +- `acceptance-criteria.md` — AC-1…AC-12 finite criteria. + +## Verification (this phase) + +- [x] Every REQ from spec §1–§9 mapped to a pseudocode line range (see overview table). +- [x] Every AC has ≥1 pseudocode reference and ≥1 evidence tag. +- [x] Contradictions vs PLAN.md resolved (operation_id derived; no gzip; no contended; + no records_dropped; no retry-threshold; compression deferred). +- [x] **Source-verified blockers resolved (D1–D8)** and applied across the + pseudocode: D1 no child-id arrays + join at read time; D2 nested settings; D3 + claim-file concurrency accounting; D4 PerfSink does not inherit FileOutput's + bounded/drop queue; D5 retention constants from a P04 benchmark; D6 fs-failure + testing via a package-private port; D7 report `--baseline`; D8 stdout observer + fails fast. +- [x] Pseudocode line-numbered; no actual TS implementation written. + +Proceed to P03. diff --git a/project-plans/issue3167/plan/03-interval-union.md b/project-plans/issue3167/plan/03-interval-union.md new file mode 100644 index 0000000000..5d3e62e646 --- /dev/null +++ b/project-plans/issue3167/plan/03-interval-union.md @@ -0,0 +1,73 @@ +# Phase 03: IntervalUnion extraction (incremental duration) + +Plan ID: PLAN-20260808-PERFTREND.P03 +Prerequisites: P02. +Package: `telemetry`. @pseudocode: `02-perfsink-and-interval-union.md` lines 10-35. + +## Why first +Foundation: both `SessionMetricsAggregator` (provider/tool unions) and the perf +recorder (provider/tool/agent-activity unions) need it. Extracting first unblocks +04 and 07 with the quadratic bug fixed. + +## Stub +- Create `packages/telemetry/src/telemetry/intervalUnion.ts` exporting `IntervalUnion` + (methods throw `new Error('NotYetImplemented')` or return typed empty: + `durationMs()=>0`, `count()=>0`, `add()=>{}`, `union()=>new IntervalUnion()`). +- No reverse tests. + +## Integration TDD (Bun, real behaviour) +- `intervalUnion.behavior.test.ts`: + - EVIDENCE-AC5a: `add` of disjoint intervals ⇒ `durationMs` = sum, `count` grows. + - Overlapping/adjacent intervals merge; duration does **not** double-count overlap. + - Nested interval fully inside another ⇒ no duration change. + - After N inserts, `durationMs()` is O(1) and equals a brute-force recompute + (proves incremental correctness, not the algorithm). + - `union(a,b)` merges two sets correctly. +- Refactor `SessionMetricsAggregator` to import `IntervalUnion`; existing session + metrics tests stay green (proves the extraction preserves semantics). + +## Impl (pseudocode 02 lines 10-35) +- `cachedDurationMs` maintained incrementally: on merge, subtract removed spans, + add merged span. No full `recomputeDuration()` walk per `add`. +- Implemented in `packages/telemetry/src/telemetry/intervalUnion.ts`. `add()` + binary-searches the insert position, computes the merge range `[from, to)`, + and adjusts `cachedDurationMs` by `mergedSpan - sum(removedSpans)` only — no + re-walk. `durationMs()` returns `cachedDurationMs` (O(1)). `add()`/`union()` + are the public API; internals (`intervals`) are not exposed for tests. + +## Verify +- [x] `bun test` for telemetry package green; existing sessionMetricsAggregator tests green. +- [x] typecheck/lint clean; no eslint-disable / suppression. +- [x] No duplicate class; existing file UPDATED. + +## Behavioral evidence (post-implementation) +- Test file: `packages/telemetry/src/telemetry/intervalUnion.behavior.test.ts` + (Bun / `bun:test`, 21 tests, 61 expect() calls, all pass). +- EVIDENCE-AC5a: disjoint `add` sums durations and grows `count()`. +- Overlapping intervals merge (overlap counted once); adjacent/touching + intervals merge (`[0,10)+[10,20)` => 20ms, count 1); nested interval adds + zero duration. +- Incremental correctness: `durationMs()` equals an independent brute-force + recompute after every insert across mixed, 250-disjoint, and out-of-order + overlapping sequences. +- `union(a,b)` merges two sets and does not mutate operands. +- Degenerate/zero-length/negative/non-finite intervals ignored; `clear()` + resets; `latestEnd` tracked. +- `SessionMetricsAggregator` refactored to import the extracted class; its + existing 77 tests (`sessionMetricsAggregator.test.ts` + + `sessionMetricsAggregator.advanced.test.ts`) stay green, proving extraction + preserved interval semantics. +- Combined run: `bun test sessionMetricsAggregator*.test.ts + intervalUnion.behavior.test.ts` => 98 pass, 0 fail. + +### Commands run and results +- `bun test src/telemetry/intervalUnion.behavior.test.ts` (RED first: module + not found while class was private) => then 21 pass, 0 fail. +- `bun test src/telemetry/sessionMetricsAggregator.test.ts + src/telemetry/sessionMetricsAggregator.advanced.test.ts + src/telemetry/intervalUnion.behavior.test.ts` => 98 pass, 0 fail. +- `tsc --noEmit` (packages/telemetry) => EXIT=0. +- `eslint` over the 4 touched source files => EXIT=0 (no eslint-disable / + suppression directives added). +- `prettier --check` over touched files + package.json => all match. +- `git diff --check` => clean (no whitespace errors). diff --git a/project-plans/issue3167/plan/04-schema-perfsink-reader.md b/project-plans/issue3167/plan/04-schema-perfsink-reader.md new file mode 100644 index 0000000000..15209781f5 --- /dev/null +++ b/project-plans/issue3167/plan/04-schema-perfsink-reader.md @@ -0,0 +1,76 @@ +# Phase 04: Schema + PerfSink + tolerant reader + record-size benchmark + +Plan ID: PLAN-20260808-PERFTREND.P04 +Prerequisites: P03. +Package: `telemetry`. @pseudocode: `01-schema-and-reader.md` lines 10-115, +`02-perfsink-and-interval-union.md` lines 50-108. + +> **Decisions applied (D1/D4/D5/D6):** the v1 perf record carries **no** +> `prompt_ids`/`turn_ids`/true-count (D1). PerfSink does **not** inherit +> FileOutput's bounded/drop queue (D4). P04 first adds a **Bun record-size +> benchmark** for the actual schema (D5); P08 derives retention constants from it. +> fs-failure tests use a **package-private filesystem port / failing file handle**, +> never real-disk fill or chmod (D6). + +## Stub +- `packages/telemetry/src/perf/perfRecords.ts`: Zod schemas (envelope, identity, + build, dimensions, operation, memory_sample) + `PERF_SCHEMA_VERSION=1` + + `deriveOperationId` + `joinKeyFromPromptId` (throws NotYetImplemented) + + `parsePerfRecord` (returns null). **No** prompt_ids/turn_ids fields. +- `packages/telemetry/src/perf/PerfSink.ts`: constructible class; `write`/`dispose` + no-op/throw. Serialized no-drop promise chain; does NOT inherit FileOutput. + +## Step 0 — Bun record-size benchmark (D5, FIRST) +- `perfRecordSize.bench.ts` (Bun): serialize a representative `operation` record + (incl. memory columns) and a `memory_sample` record; report the byte size of a + single JSONL line. Output is the input P08 uses to derive max-bytes/max-files/ + maintenance-interval/diagnostic-rate-limit. No placeholders here. + +## Integration TDD (Bun, REAL files — no vi.mock(fs)) +- `perfSink.roundtrip.behavior.test.ts` (EVIDENCE-AC1): + - Real PerfSink to a tmpdir; write N terminal-operation records; read the file + with the real reader; assert each round-trips to identical field values and + that **no** `prompt_ids`/`turn_ids` fields exist (D1). + - Midnight UTC day-key roll: a record whose ts crosses midnight rolls to a new + file on the next write; both files parse. + - Empty operation set ⇒ no file created. +- `perfReader.tolerant.behavior.test.ts` (EVIDENCE-AC9 partial): + - Unknown fields ignored (no version bump needed). + - A line with `schema_version` > known ⇒ `parsePerfRecord` returns null + (skip+count), never throws, never coerces. + - Truncated final line (no newline / partial JSON) ⇒ counted, reader continues. +- `perfReader.join.behavior.test.ts` (EVIDENCE-AC3 read-time join, D1): + - One perf `operation` record + N token-usage rows (one per continuation, each + with its own `prompt_id`); the reader/report derives `operation_id` from each + token row and joins all N to the single perf operation — multi-continuation + rows join to one operation without child ids on the perf record. +- `perfSink.failopen.behavior.test.ts` (EVIDENCE-AC8, D6): + - Inject EACCES/EROFS/ENOSPC via a package-private filesystem port / failing + file handle (NOT chmod / NOT real-disk fill); assert write does not throw into + the caller; diagnostics emitted at most once per rate-limit window. +- `perfSink.exclusive.behavior.test.ts` (EVIDENCE-AC1): + - Two PerfSinks with distinct runUuids create distinct day files via exclusive + `wx`; concurrent appends produce no torn lines. + +## Impl (pseudocode) +- Schema: single Zod declaration (no child-id arrays — D1); writer & reader + `z.infer` from it. +- PerfSink: does NOT inherit FileOutput; serialized no-drop promise chain (own + back-pressure ⇒ no drop counter); stat-once + in-memory byte counter; one + exclusive-create day file per run UUID (`perf--.jsonl`); UTC + roll on next record; no gzip, no size sub-rolling; drain on dispose (removes + claim — D3); rate-limited diagnostics (filesystem errors only). +- Reader: discriminated union on `record_type`; tolerant normalise-to-v0 for + unversioned; skip+count above known version; truncation tolerance; read-time + join via `joinKeyFromPromptId` (D1). + +## Verify +- [ ] Step-0 record-size benchmark committed (D5); its output recorded for P08. +- [ ] AC-1, AC-2(writer-half), AC-3(read-time join), AC-8 evidenced. +- [ ] No `vi.mock('fs')` anywhere in new tests; real files + package-private port. +- [ ] typecheck/lint clean; FileOutput reused via narrow primitives, not inherited. + +## Note on REQ-3167-7 reduction (spec §9 / D4) +There is **no** `records_dropped` field and **no** retry-threshold self-disable. +PerfSink does not carry FileOutput's bounded queue. Fail-open + rate-limited +diagnostics for filesystem errors only; internal errors fail fast (D8). diff --git a/project-plans/issue3167/plan/05-stdout-observer-onrender.md b/project-plans/issue3167/plan/05-stdout-observer-onrender.md new file mode 100644 index 0000000000..2ed191abac --- /dev/null +++ b/project-plans/issue3167/plan/05-stdout-observer-onrender.md @@ -0,0 +1,44 @@ +# Phase 05: Stdout observer seam + Ink onRender wiring + +Plan ID: PLAN-20260808-PERFTREND.P05 +Prerequisites: P02 (parallel-safe with P04). +Packages: `core` (seam), `cli` (install). @pseudocode: `03-stdout-observer.md` +lines 10-58, `05-client-phases.md` lines 20-26. + +## Stub +- `core/utils/stdio.ts`: add `StdoutWriteObserver` interface + optional `observer` + param to `createInkStdio`; absent ⇒ current behaviour unchanged. +- `cli/ui/inkRenderOptions.ts`: replace module-scope `sharedStdio` with + `getInteractiveStdio()` lazy cache + `setInteractiveStdoutObserver()`. + +## Integration TDD (Bun, real behaviour) +- `createInkStdio.observer.behavior.test.ts` (EVIDENCE-AC6): + - With an observer, a `Uint8Array` write increments bytes by + `Uint8Array.byteLength` (NOT string length); a multi-byte UTF-8 string counts + encoded bytes. + - `stdout_write_calls` increments once per write; the wrapper returns the real + `writeToStdout` boolean (backpressure) and invokes the callback. + - **D8:** an internal observer that throws **propagates** (fail fast — no + try/catch swallows it). Filesystem writer failures remain fail-open (AC-8). + - Without an observer, behaviour is byte-identical to today (no counting side-effect). +- `inkRenderOptions.observer.behavior.test.ts` (EVIDENCE-AC6): + - `setInteractiveStdoutObserver(obs)` then `getInteractiveStdio()` yields a + Proxy carrying `obs`; calling it again returns the SAME cached instance. + - A second `setInteractiveStdoutObserver` invalidates the cache (next build + carries the new observer). + - Ink `onRender` accumulate: render metrics add `renderTime` (verified + against installed @jrichman/ink@6.4.8: `RenderMetrics = { renderTime: number }`) + and increment a render-pass counter (distinct from write calls) — assert + render_count ≠ write_calls on a coalesced frame. + +## Impl (pseudocode 03 lines 10-58, D8) +- Optional observer; measure encoded bytes + sync invocation duration only; + delegate to `writeToStdout`; the observer is called directly with **no** + try/catch (internal errors fail fast). +- Zed's `createInkStdio()` call passes no observer ⇒ uncounted. + +## Verify +- [x] AC-6 evidenced; Zed path explicitly uncounted. +- [x] Observer errors fail fast (D8); no swallowed internal exception. +- [x] core has no cli import; cli installs the observer it owns. +- [x] typecheck/lint clean; existing stdio tests green. diff --git a/project-plans/issue3167/plan/06-operation-lifecycle-identity.md b/project-plans/issue3167/plan/06-operation-lifecycle-identity.md new file mode 100644 index 0000000000..3d76769bb7 --- /dev/null +++ b/project-plans/issue3167/plan/06-operation-lifecycle-identity.md @@ -0,0 +1,50 @@ +# Phase 06: Operation lifecycle registry + identity + +Plan ID: PLAN-20260808-PERFTREND.P06 +Prerequisites: P04, P09. +Package: `cli`. @pseudocode: `04-operation-lifecycle.md` lines 10-66, +`01-schema-and-reader.md` lines 60-66 + 104-115. + +> **Decisions applied (D1/D3):** the registry collects **no** child prompt/turn +> ids (D1 — they do not arrive via `AgentEvent`); `operation_id` is the sole join +> key. `concurrent_instances` is derived from non-stale claim files at +> finalization (D3). + +## Stub +- `OperationLifecycleRegistry`: `begin`/`finalise` throw NotYetImplemented or + return typed handles. (No `observePromptId` — child ids are not collected.) + +## Integration TDD (Bun, real behaviour) +- `operationLifecycle.behavior.test.ts` (EVIDENCE-AC3, AC4): + - EVIDENCE-AC3: a real continuation stream through the recorder; for initial id + `S#agentic-loop#U` and continuations `…#continuation#1/2`, assert + `deriveOperationId(c) === S#agentic-loop#U` for every observed child id; assert + the produced record carries **no** `prompt_ids`/`turn_ids` fields (D1); assert + `concurrent_instances` reflects non-stale claims (D3). + - EVIDENCE-AC3 read-time join: N continuation token-usage rows each derive to the + same `operation_id` and join to the single perf operation. + - EVIDENCE-AC4: drive each terminal path (completed, error, + cancelled_before_send, cancelled_during_api/tool/approval) through the + integrated lifecycle (fixture provider + real AbortControllers); assert one + record per path with the correct `status`. + - **Superseded**: a newer turn replaces `abortControllerRef.current`; assert the + displaced op is finalised as `superseded` exactly once despite + `isCurrentTurn==false`. + - Exactly-once: double-finalise is a no-op. +- `promptIdPrefixInvariant.behavior.test.ts` (EVIDENCE-AC3): + - Assert `operation_id = promptId.split('#continuation#')[0]` for initial, + continuation, and CLI-fallback (8-hash) ids. Extend, do not duplicate, the + existing `agenticLoop.prompt-id.test.ts`. + +## Impl (pseudocode 04 lines 10-66) +- Registry keyed by turn signal; `begin` derives operationId (pseudocode 01 lines + 60-66), snapshots identity; `finalise` derives `concurrent_instances` from + non-stale claims (pseudocode 06), builds+writes the record exactly once. +- Superseded sweep: on a new `begin` that detects an already-displaced signal, + finalise the displaced op as `superseded`. + +## Verify +- [ ] AC-3, AC-4 evidenced; superseded explicitly covered. +- [ ] `packages/agents` untouched (no telemetry import added there). +- [ ] No mint+propagation; operation_id derived; no child-id arrays on the record. +- [ ] typecheck/lint clean. diff --git a/project-plans/issue3167/plan/07-client-phases.md b/project-plans/issue3167/plan/07-client-phases.md new file mode 100644 index 0000000000..2c7a7aeb80 --- /dev/null +++ b/project-plans/issue3167/plan/07-client-phases.md @@ -0,0 +1,38 @@ +# Phase 07: Client phase measurement + record assembly + +Plan ID: PLAN-20260808-PERFTREND.P07 +Prerequisites: P05, P06. +Package: `cli`. @pseudocode: `05-client-phases.md` lines 10-79. + +## Stub +- Recorder measurement fields + `buildRecord` throw/empty. + +## Integration TDD (Bun, real behaviour) +- `clientPhases.behavior.test.ts` (EVIDENCE-AC5): + - Real streaming turn (fixture provider, deterministic deltas): assert every + client phase (`client_prepare_ms`, `stream_handler_ms`, `ink_render_ms`, + `stdout_write_sync_ms`, `client_finalize_ms`) is ≥ 0 and directly measured. + - Assert NO phase is computed as `elapsed − provider − tool` (inspect the record + shape: phases are independent fields; the subtraction never appears). + - Overlapping provider+tool: `provider_union_ms` ≤ `provider_attempt_sum_ms` + permitted; `agent_activity_union_ms` = provider∪tool union; record does not + claim they sum to elapsed. + - `unclassified_elapsed_ms` reported honestly; on a turn with a synthetic gap it + is positive and non-clamped (assert ≥ the injected gap, never < 0 unconditionally + — it MAY be 0 only when there is genuinely no residual). +- `inkRenderVsWrite.behavior.test.ts` (EVIDENCE-AC6, partial): + - Coalesced/throttled frame ⇒ `ink_render_count` ≠ `stdout_write_calls`. + +## Impl (pseudocode 05 lines 10-74) +- `client_prepare_ms`/`client_finalize_ms` via `performance.now()` deltas; + `stream_handler_ms` Σ sync delta CPU; `ink_render_ms`/`ink_render_count` from + onRender; stdout fields from the observer; provider/tool sums+unions via + IntervalUnion; `unclassified_elapsed_ms` honest residual (provider/tool NOT + subtracted); `concurrent_instances` passed in from finalisation (D3); memory + columns omitted when disabled (delegated to P10). **D1:** no `capAndCount` / + prompt/turn-id arrays — `operation_id` is the sole join key. + +## Verify +- [ ] AC-5 evidenced; AC-6 (render/write) evidenced. +- [ ] No subtraction-based phase; unclassified never clamped. +- [ ] typecheck/lint clean. diff --git a/project-plans/issue3167/plan/08-retention.md b/project-plans/issue3167/plan/08-retention.md new file mode 100644 index 0000000000..95072cb165 --- /dev/null +++ b/project-plans/issue3167/plan/08-retention.md @@ -0,0 +1,73 @@ +# Phase 08: Retention (eventual bound, live-writer safe, claim files) + +Plan ID: PLAN-20260808-PERFTREND.P08 +Prerequisites: P04. +Package: `telemetry`. @pseudocode: `06-retention.md` lines 10-72. + +> **Decisions applied (D3/D5/D6):** claim-file concurrency accounting (D3) reuses +> the one owned maintenance timer to touch claims and reap stale ones. +> **D5:** before implementing caps, P08 derives and documents **concrete** +> max-bytes/max-files/maintenance-interval/diagnostic-rate-limit defaults from the +> P04 Bun record-size benchmark and operational evidence — **no placeholders at +> implementation time**. The eventual bound explicitly permits active-day and +> claim overshoot. **D6:** unlink-failure tests use a package-private fs port, not +> chmod. + +## Stub +- `packages/telemetry/src/perf/retention.ts`: `maybeMaintain`/`maintain`/ + `isLiveWriter`/`createClaim`/`touchClaim`/`countNonStaleClaims` throw/empty. + +## Constants derivation (D5 — FIRST in this phase) +- From the P04 record-size benchmark + operational evidence, derive and record: + `MAX_BYTES`, `MAX_FILES`, `MAINTENANCE_INTERVAL_MS`, `CLAIM_LEASE_MS`, + `DIAG_RATE_LIMIT_MS`. Document that the eventual bound permits active-day and + claim overshoot. No placeholders remain after this step. + +## Integration TDD (Bun, REAL files — no vi.mock(fs)) +- `retention.behavior.test.ts` (EVIDENCE-AC7): + - **Live-writer safety**: a file with today's day-key and mtime within the + maintenance window survives a sweep that evicts older files (assert present + after `maintain`). + - **24×7 convergence**: one continuously claimed run's old-day JSONL remains + evictable while its current/recent JSONL and fresh claim survive; assert both + the byte and artifact caps converge after `maintain`. + - **24×7 trigger**: `maintain` runs on the coarse interval without restart; + a roll boundary also triggers it; the same interval touches the claim file. + - **Failed unlink (D6)**: inject the failure via a package-private fs port; + accounting NOT decremented (total unchanged); older file still listed for next + sweep; diagnostics rate-limited. + - **Concurrency overshoot**: many writers appending between scan and delete ⇒ + documented overshoot, NOT an assertion of zero loss. Assert that after enough + evictions the total eventually falls under the cap. + - **Claim accounting (D3)**: `.claim` files count toward total bytes/files but + are never parsed as JSONL; a fresh claim is not reaped; a stale claim is. + - **Clock step**: materially-future mtime delays eligibility (benign). +- `retention.capSelection.behavior.test.ts`: + - Under observed volume, assert which of (count cap, byte cap) binds. + +## Impl (pseudocode 06 lines 10-72) +- Claim-file lifecycle: create on enable, touch by the maintenance interval, + remove on clean dispose; count non-stale claims for `concurrent_instances`. +- Evict oldest-first until BOTH caps satisfied; skip live-writer files; decrement + accounting ONLY on unlink success; rate-limit diagnostics; run on roll boundary + + coarse interval. Claims counted in accounting but not parsed. A fresh claim + protects the claim artifact, not that run's old-day JSONL; explicit delete's + broader claim-to-JSONL protection is intentionally separate. + +## Verify +- [x] Constants derived from the P04 benchmark (D5); no placeholders. +- [x] AC-7 evidenced; no instantaneous-cap assertion; live-writer never deleted. +- [x] Claim files counted but not parsed as JSONL (D3). +- [x] No `vi.mock('fs')`; real files + package-private port for fault injection (D6). +- [x] typecheck/lint clean. + +Post-implementation convergence evidence: the telemetry perf suite passes 456 +Bun tests, including a deterministic fixture where one continuously claimed +owner begins over both caps, loses five historical day files, and retains only +its current/recent live file plus its fresh claim. + +## Note on REQ-3167-6 +Guarantee is eventual-with-overshoot + live-writer safety (§6), explicitly NOT an +instantaneous no-loss cap, and explicitly permits active-day and claim overshoot. +`rotateReports()` shape reused; its weaker guarantees (in-process-only protection, +decrement-on-failure) NOT copied. diff --git a/project-plans/issue3167/plan/09-settings.md b/project-plans/issue3167/plan/09-settings.md new file mode 100644 index 0000000000..133d88fa9f --- /dev/null +++ b/project-plans/issue3167/plan/09-settings.md @@ -0,0 +1,126 @@ +# Phase 09: Settings (opt-in, default-off, nested shape — D2) + +Plan ID: PLAN-20260808-PERFTREND.P09 +Prerequisites: P02 (do before wiring 06/10/11). +Package: `core` (config) + `cli` (settings schema/configBuilder). @pseudocode: +`08-consumer-and-perf-command.md` lines 10-24. + +> **Decision D2:** the persisted shape is **nested** `telemetry.perf.enabled` +> (master) and `telemetry.perf.memory`, both default **false**; memory requires +> enabled. `telemetry.perf` is **not** itself a boolean. (Spec §7.4 names the +> master `telemetry.perf`; the resolved persisted contract is the nested shape — +> recorded here because spec is not rewritten.) + +## Why before wiring +The lifecycle/observer/memory/command phases all gate on `resolvePerf`. Landing +the setting + resolver first gives them a real (non-mock) switch to read. + +## Stub +- `TelemetrySettings` gains `perf?: PerfTelemetrySettings` where + `PerfTelemetrySettings = { enabled?: boolean; memory?: boolean }`. +- `resolvePerfSettings` helper returns `{ enabled, memory }` with master-gates- + memory and default-false semantics. +- `resolveTelemetrySettings` deep-copies + freezes the perf sub-object so + `Config.getTelemetrySettings()` cannot leak nested mutable state. + +## Fact-check: hierarchy (corrected from original) + +The original plan claimed hierarchy "CLI flag > env > workspace +`.llxprt/settings.json` > user > default (follow existing +`resolveTelemetrySettings` precedence)." Source fact-checking found this +conflates two distinct mechanisms: + +1. **Persisted settings merge** (CLI `mergeSettings()` in `settingsMerge.ts`): + layers are merged via `mergeObjectSection('telemetry', ...)` which does a + **shallow spread** — a higher-precedence layer's `perf` object REPLACES the + lower-precedence `perf` entirely (not a deep merge of `perf.enabled` / + `perf.memory` across layers). Precedence: schema defaults < system defaults < + user < workspace (trusted only) < system. + +2. **CLI/env mapping**: yargs exposes ONLY flat flags (`--telemetry`, + `--telemetry-log-prompts`, `--telemetry-outfile`). There are **no CLI flags + or env vars for `telemetry.perf.*`** (the issue spec does not require them). + `buildTelemetryConfig()` overlays these flat CLI flags on merged settings and + passes `perf: telemetrySettings?.perf` through to Config unchanged. + +3. **`resolveTelemetrySettings`** (core `configConstructor.ts`): does NOT + implement a settings-layer hierarchy — it applies per-field defaults and + deep-copies + freezes the perf sub-object. The hierarchy is in `mergeSettings` + (CLI) + `buildTelemetryConfig` (CLI). + +**No dedicated CLI perf flags or environment variables were added** (the existing +generic yargs pipeline does not make nested `telemetry.perf` addressable via +flags/env). Perf is configured only via persisted settings files. + +## Implementation delivered + +### Core (`packages/core/src/config/`) +- **`configTypes.ts`**: added `PerfTelemetrySettings` interface + (`{ enabled?: boolean; memory?: boolean }`); added `perf?: PerfTelemetrySettings` + to `TelemetrySettings`. +- **`configConstructor.ts`**: exported `resolveTelemetrySettings` (was private); + added `resolvePerfSettings(settings): { enabled: boolean; memory: boolean }` + with master-gates-memory; `resolveTelemetrySettings` now clones perf + (`{ ...perf }`) on every ingress and egress via `withClonedPerf` so the + shallow copy returned by `getTelemetrySettings()` cannot reach the source. + The resolved perf is a mutable isolated copy — isolation is by cloning, not + freezing. +- **`config.ts`**: unchanged (0 lines added). `getTelemetrySettings()` returns + `{ ...this.telemetrySettings }` as before — the cloned perf isolates mutation. +- **`src/index.ts`**: exports `resolvePerfSettings` and `PerfTelemetrySettings` + for downstream phases. + +### CLI (`packages/cli/src/config/`) +- **`configBuilder.ts`**: `buildTelemetryConfig` passes `perf: + telemetrySettings?.perf` through to Config. +- **`settingsSchema.ts`**: `SETTINGS_SCHEMA_DEFINITIONS.TelemetrySettings` gains + `perf` property (type: object, additionalProperties: false, properties: + enabled/memory booleans). +- **`schemas/settings.schema.json`**: regenerated `TelemetrySettings` $def with + perf property. + +### Docs +- **`docs/telemetry-privacy.md`**: added "Client Performance Telemetry" section + documenting `telemetry.perf.enabled` / `telemetry.perf.memory`, default-off, + local-only, master-gates-memory. No commands documented (P11 will add + inspect/delete/report). + +## Integration TDD (Bun, real behaviour) +- `perfSettings.behavior.test.ts` (16 tests — EVIDENCE-AC2): + - Default (no setting) ⇒ `{ enabled: false, memory: false }`. + - Master on, memory omitted ⇒ `{ enabled: true, memory: false }`. + - Master off, memory on ⇒ `{ enabled: false, memory: false }`. + - Both on ⇒ `{ enabled: true, memory: true }`. + - False overrides. + - Input immutability (3 tests). + - Nested-return copy isolation (2 tests). + - Return type safety. +- `telemetrySettingsCopy.behavior.test.ts` (6 tests — EVIDENCE-AC2): + - Perf is a copy, not caller reference. + - Resolved perf is a mutable isolated copy — isolation is by cloning, not freezing. + - Mutations to input after resolution do not affect resolved copy. + - Does not mutate caller settings object. + - Undefined perf resolves to undefined. + - Preserves perf fields through resolution. +- `perfSettingsMerge.behavior.test.ts` (8 tests — real `mergeSettings`): + - Absent in all layers ⇒ no perf key. + - User-only perf flows through. + - Workspace replaces user perf (shallow merge at telemetry level). + - Both layers set enabled — workspace wins. + - Telemetry scalar fields coexist with perf across layers. + - Untrusted workspace ignored. + - System layer wins over user and workspace. + - System defaults overridden by user. +- `perfSettingsValidation.behavior.test.ts` (14 tests — real Zod validation): + - Accepts perf as object with enabled/memory (6 accepted shapes). + - Rejects perf as boolean true/false (D2: not a boolean). + - Rejects non-boolean enabled/memory. + - Rejects unknown properties (additionalProperties: false). + - Rejects string/number/array. + +## Verify +- [x] AC-2 (settings half) evidenced; default-off; nested shape (D2). +- [x] No unrelated settings changed (config.ts: 0 lines changed). +- [x] typecheck/lint/prettier clean. +- [x] No CLI flags or env vars invented for perf. +- [x] Hierarchy fact-checked and corrected. diff --git a/project-plans/issue3167/plan/10-memory-trend.md b/project-plans/issue3167/plan/10-memory-trend.md new file mode 100644 index 0000000000..da771d6371 --- /dev/null +++ b/project-plans/issue3167/plan/10-memory-trend.md @@ -0,0 +1,39 @@ +# Phase 10: Memory trend (zero new timers, ring, two slopes) + +Plan ID: PLAN-20260808-PERFTREND.P10 +Prerequisites: P06, P09. +Package: `cli`. @pseudocode: `07-memory-trend.md` lines 10-93. + +## Stub +- `MemoryRing` (fixed-capacity overwrite) + extended `useMemoryMonitor` sampling + hook (throw/empty). + +## Integration TDD (Bun, real behaviour) +- `memoryRing.behavior.test.ts` (EVIDENCE-AC11): + - Push M > CAPACITY samples; assert ring length == CAPACITY and the oldest were + overwritten (snapshot oldest→newest). +- `useMemoryMonitor.behavior.test.ts` (EVIDENCE-AC11): + - After a warning fires, the interval is STILL active (warn-once latch separated + from the sampling loop — the self-`clearInterval` defect is fixed). + - No NEW timer is created (assert only the existing 60 s interval exists). +- `memoryTrend.behavior.test.ts` (EVIDENCE-AC10): + - Memory **off** ⇒ operation records OMIT memory columns (field absent, not 0) + and NO `memory_sample` rows are written; monitor reverts to warn-only. + - Memory **on** ⇒ operation records carry the four columns; `memory_sample` rows + carry `uptime_ms` + `ms_since_last_operation`. + - Reader derives TWO slopes: per-operation on `session_operation_index`; + per-minute on `uptime_ms` using sample rows; idle samples + (`ms_since_last_operation` large) expose the #3114 leak signature. + - Assert slopes are DERIVED at read time (no stored slope field on any record). + +## Impl (pseudocode 07 lines 10-93) +- Extend the 60 s interval (no new timer); separate latch from sampling; push to + fixed-capacity ring; emit `memory_sample` when perf+memory on; memory columns + ride the operation record (omit when off); slopes derived at read time. + +## Verify +- [ ] AC-10, AC-11 evidenced. +- [ ] Zero new timers (Footer 2 s interval NOT used). +- [ ] Memory-disabled ⇒ fields absent (not zero). +- [ ] Ring fixed-capacity; monitor continues after warning. +- [ ] typecheck/lint clean. diff --git a/project-plans/issue3167/plan/11-reader-consumer-perf-command.md b/project-plans/issue3167/plan/11-reader-consumer-perf-command.md new file mode 100644 index 0000000000..347c4ef2e4 --- /dev/null +++ b/project-plans/issue3167/plan/11-reader-consumer-perf-command.md @@ -0,0 +1,54 @@ +# Phase 11: Reader/consumer + report + /perf + inspect + delete + +Plan ID: PLAN-20260808-PERFTREND.P11 +Prerequisites: P04, P09. +Packages: `telemetry` (reader/report), `cli` (command). @pseudocode: +`08-consumer-and-perf-command.md` lines 30-99. + +> **Decisions applied (D1/D7):** the report joins multi-continuation token/session +> rows to a perf operation by deriving `operation_id` at read time (D1). +> `--baseline` accepts an exact `llxprt_version` or `git_sha`; without it the +> report prints grouped matched-dimension p50/sample/self-health and no delta, and +> with it unmatched groups are reported as unmatched, never pooled (D7). `/perf` is +> a current-process snapshot; the report is longitudinal. `inspect` shows +> path/schema/privacy/record counts; `delete` respects live claims (D3). + +## Stub +- `perfReader`/`buildReport`/`perfInspect`/`perfDelete` throw/empty; `/perf` + SlashCommand skeleton registered in `BuiltinCommandLoader`. + +## Integration TDD (Bun, REAL files) +- `report.behavior.test.ts` (EVIDENCE-AC9): + - Real multi-version fileset (v0 unversioned, v1 known, v999 unknown, one + malformed line, one truncated final line): known records parse; unknown-version + skipped+counted; malformed skipped+counted; truncated tail counted; report + includes counts and self-health (skipped/truncated/last write error/evictions). + - **D7 baseline:** without `--baseline` → grouped matched-dimension + p50/sample/self-health, no delta; with `--baseline ` → + matched-dimension delta vs baseline rows only (provider/model/render-mode/ + terminal-geometry buckets); unmatched groups reported as unmatched, never pooled. + - Groups by version/commit within matched dimensions; contamination via + `concurrent_instances >= 2` (NOT contended). + - **D1 read-time join:** N continuation token-usage rows derive the same + `operation_id` and join to the single perf operation. + - Cross-platform path handling (no shell pipeline). +- `perfCommand.behavior.test.ts` (EVIDENCE-AC9): + - `/perf inspect` ⇒ dir, schema version, privacy, file count, total bytes, + operation/memory-sample record counts. + - `/perf report` ⇒ longitudinal report output. + - `/perf delete` ⇒ removes JSONL files AND stale claims WITH live-writer safety + (today + recent mtime / fresh claims survive); failures counted, fail-open. + - `/perf` (no subcommand) ⇒ snapshot of THIS process (live MemoryRing + current op). + +## Impl (pseudocode 08 lines 30-99) +- Streaming reader (one file at a time; no gzip; tolerant); report groups + + matched-dimension p50 + optional exact baseline (D7) + read-time join (D1); + inspect/delete follow the SlashCommand convention; delete respects live claims + (reuses `isLiveWriter` / fresh-claim check — pseudocode 06). + +## Verify +- [ ] AC-9 evidenced; no gzip; no contended; no records_dropped. +- [ ] Baseline semantics (D7); unmatched groups never pooled. +- [ ] delete respects live claims (D3). +- [ ] Command registered in BuiltinCommandLoader; subcommands work. +- [ ] typecheck/lint clean; no new JS/Vitest tests. diff --git a/project-plans/issue3167/plan/12-integration-overhead-harness.md b/project-plans/issue3167/plan/12-integration-overhead-harness.md new file mode 100644 index 0000000000..de3ec2b923 --- /dev/null +++ b/project-plans/issue3167/plan/12-integration-overhead-harness.md @@ -0,0 +1,37 @@ +# Phase 12: Integration wiring + overhead harness + +Plan ID: PLAN-20260808-PERFTREND.P12 +Prerequisites: P06, P07, P10, P11. +Package: `cli`. @pseudocode: `09-overhead-harness.md` lines 10-64. + +## Goal +Wire the recorder into the real CLI operation path end-to-end (settings → observer +install → lifecycle → record → sink → retention), and prove the observer effect +with a real Bun harness. + +## Integration TDD (Bun, REAL integration — NO mocks of recorder/sink/observer) +- `overheadHarness.behavior.test.ts` (EVIDENCE-AC12): + - Runs a streaming-load scenario through the REAL integrated pipeline twice: + perf ENABLED and DISABLED. + - PRINTS p50/p95/p99 per-op overhead for both + delta (evidence, not a gate). + - ASSERTS stable invariants only: + - disabled ⇒ no perf file created; no observer installed; no ring allocated. + - enabled ⇒ record count == operation count. + - disabled path produces zero side-effects (architectural guarantee). + - Does NOT assert a wall-clock µs threshold. +- `endToEnd.behavior.test.ts` (EVIDENCE-AC1, AC2, AC4 — integration spine): + - Full default-off run ⇒ no files, no claim file. + - Full enabled run ⇒ records on disk joinable to identity at read time (D1 — + no child ids on the record); superseded recorded; `concurrent_instances` + reflects claim files (D3); PerfSink uses the no-drop serialized chain (D4). + +## Impl (pseudocode 09 lines 10-64) +- Wire `resolvePerf` → observer install (`setInteractiveStdoutObserver`) → + registry construction → onRender wiring → sink + retention. Disabled path + short-circuits before any construction. + +## Verify +- [ ] AC-12 evidenced; no wall-clock assertion; real integration. +- [ ] No mock theater (real PerfSink/observer/lifecycle). +- [ ] Disabled path has zero side-effects. +- [ ] typecheck/lint clean. diff --git a/project-plans/issue3167/plan/13-final-verification.md b/project-plans/issue3167/plan/13-final-verification.md new file mode 100644 index 0000000000..fd833e7c3d --- /dev/null +++ b/project-plans/issue3167/plan/13-final-verification.md @@ -0,0 +1,41 @@ +# Phase 13: Final verification + +Plan ID: PLAN-20260808-PERFTREND.P13 +Prerequisites: P03–P12 all complete. + +## Whole-suite verification +- `npm run test`, `npm run lint`, `npm run typecheck`, `npm run format`, + `npm run build` — all green. +- `bun scripts/start.ts --profile-load stepfun-37 "write me a haiku and nothing else"` + — smoke run. +- Deferred-implementation detection across changed packages (no TODO/HACK/STUB/ + placeholder/empty-returns in implemented code). +- No `eslint-disable` / `@ts-ignore` / `@ts-expect-error` / severity downgrades / + complexity threshold increases / ignore exclusions introduced. +- `packages/agents` diff is empty for this feature (no telemetry edge added). +- No dependency/workflow/quality-tool/setting-unrelated-to-perf changes. + +## AC coverage sign-off +- AC-1 single-schema round trip ✔ (P04) +- AC-2 default-off no files ✔ (P09, P12) +- AC-3 identity joins + continuation grouping ✔ (P06) +- AC-4 every terminal status incl. superseded ✔ (P06) +- AC-5 direct phase measurement + overlap ✔ (P07) +- AC-6 Ink render/write distinction + stdout correctness ✔ (P05) +- AC-7 retention under concurrency/24×7/clock/live/unlink ✔ (P08) +- AC-8 fail-open + rate-limited diagnostics under EACCES/EROFS/ENOSPC ✔ (P04) +- AC-9 cross-platform consumer + inspect/delete + /perf ✔ (P11) +- AC-10 memory omitted when off; two slopes when on ✔ (P10) +- AC-11 fixed-capacity ring + continued monitor ✔ (P10) +- AC-12 observer-effect harness p50/p95/p99 no wall-clock gate ✔ (P12) + +## Out-of-scope confirmation +- gzip / size sub-rolling: deferred (not implemented). +- contended / records_dropped / retry-threshold: excluded (not implemented). +- `prompt_ids`/`turn_ids` arrays + true-count/cap on the perf record: excluded + (D1 — child ids do not arrive via AgentEvent; join at read time). +- PerfSink inheriting FileOutput's bounded/drop queue: excluded (D4). +- agents→telemetry edge / operation_id propagation: excluded. + +## Completion marker +Write `project-plans/issue3167/.completed/P13.md` with the verification output. diff --git a/schemas/settings.schema.json b/schemas/settings.schema.json index 37c8b2f451..d464a97cf8 100644 --- a/schemas/settings.schema.json +++ b/schemas/settings.schema.json @@ -1689,6 +1689,21 @@ "outfile": { "type": "string", "description": "File path for writing telemetry output." + }, + "perf": { + "type": "object", + "description": "Client-side performance telemetry (local-only, default off). When enabled, timing and resource data is written to local perf files. memory requires enabled to be true.", + "additionalProperties": false, + "properties": { + "enabled": { + "type": "boolean", + "description": "Master switch for local client performance telemetry. Default false." + }, + "memory": { + "type": "boolean", + "description": "Include memory trend data in perf records. Effective only when enabled is true. Default false." + } + } } } },